Page MenuHomePhabricator
Create Task
Maniphest T391083

Prepare our custom installer and the base layer for Trixie
Closed, ResolvedPublic
Actions

Description

We'll probably start with alpha2

Trixie is now fully released, on 2025-08-09.

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 4 2025, 9:56 AM
MoritzMuehlenhoff triaged this task as Medium priority.Apr 4 2025, 9:56 AM
gerritbot added a comment.Apr 4 2025, 10:27 AM

Change #1134187 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] uwsgi: trixie support

https://gerrit.wikimedia.org/r/1134187

gerritbot added a project: Patch-For-Review.Apr 4 2025, 10:27 AM

Change #1134188 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] wmflib: postgresql_version add trixie

https://gerrit.wikimedia.org/r/1134188

gerritbot added a comment.Apr 4 2025, 10:27 AM

Change #1134189 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] ruby: move to .exist?

https://gerrit.wikimedia.org/r/1134189

gerritbot added a comment.Apr 4 2025, 10:41 AM

Change #1134188 merged by Filippo Giunchedi:

[operations/puppet@production] wmflib: postgresql_version add trixie

https://gerrit.wikimedia.org/r/1134188

gerritbot added a comment.Apr 4 2025, 10:45 AM

Change #1134187 merged by Filippo Giunchedi:

[operations/puppet@production] uwsgi: trixie support

https://gerrit.wikimedia.org/r/1134187

gerritbot added a comment.Apr 7 2025, 11:08 AM

Change #1134189 merged by Filippo Giunchedi:

[operations/puppet@production] ruby: move to .exist?

https://gerrit.wikimedia.org/r/1134189

Maintenance_bot removed a project: Patch-For-Review.Apr 7 2025, 11:30 AM
gerritbot added a comment.Apr 24 2025, 9:44 AM

Change #1138694 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add d-i config for Trixie

https://gerrit.wikimedia.org/r/1138694

gerritbot added a project: Patch-For-Review.Apr 24 2025, 9:44 AM
gerritbot added a comment.Apr 24 2025, 9:58 AM

Change #1138697 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add pxelinux config for Trixie

https://gerrit.wikimedia.org/r/1138697

gerritbot added a comment.Apr 24 2025, 9:59 AM

Change #1138698 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/cookbooks@master] Add trixie to the list of supported OSes

https://gerrit.wikimedia.org/r/1138698

gerritbot added a comment.Apr 24 2025, 12:20 PM

Change #1138694 merged by Muehlenhoff:

[operations/puppet@production] Add d-i config for Trixie

https://gerrit.wikimedia.org/r/1138694

gerritbot added a comment.Apr 24 2025, 12:56 PM

Change #1138697 merged by Muehlenhoff:

[operations/puppet@production] Add pxelinux config for Trixie

https://gerrit.wikimedia.org/r/1138697

gerritbot added a comment.Apr 25 2025, 6:30 AM

Change #1138698 merged by Muehlenhoff:

[operations/cookbooks@master] Add trixie to the list of supported OSes

https://gerrit.wikimedia.org/r/1138698

Maintenance_bot removed a project: Patch-For-Review.Apr 25 2025, 7:30 AM
ops-monitoring-bot added a comment.Apr 25 2025, 11:31 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host sretest1001.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Apr 25 2025, 1:08 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host sretest1001.eqiad.wmnet with OS trixie executed with errors:

  • sretest1001 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202504251153_jmm_583340_sretest1001.out, asking the operator what to do
    • First Puppet run failed and the operator aborted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console sretest1001.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
gerritbot added a comment.Apr 25 2025, 1:38 PM

Change #1139037 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add trixie to pbuilder setup

https://gerrit.wikimedia.org/r/1139037

gerritbot added a project: Patch-For-Review.Apr 25 2025, 1:38 PM
taavi subscribed.Apr 25 2025, 1:40 PM
gerritbot added a comment.Apr 28 2025, 6:50 AM

Change #1139037 merged by Muehlenhoff:

[operations/puppet@production] Add trixie to pbuilder setup

https://gerrit.wikimedia.org/r/1139037

Maintenance_bot removed a project: Patch-For-Review.Apr 28 2025, 7:30 AM
Stashbot added a comment.Apr 29 2025, 7:23 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-29T07:23:27Z] <moritzm> imported debdeploy 0.0.99.14-1+deb13u1 to apt.wikimedia.org/main for trixie-wikimedia T391083

Stashbot added a comment.Apr 29 2025, 7:50 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-29T07:50:08Z] <moritzm> copied wmf-certificates 1~20230906-1 from bookworm-wikimedia to trixie-wikimedia T391083

Stashbot added a comment.Apr 29 2025, 7:53 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-29T07:53:06Z] <moritzm> copied cadvisor 0.44.0+ds1-1~wmf1 from bookworm-wikimedia to trixie-wikimedia T391083

gerritbot added a comment.May 5 2025, 6:27 AM

Change #1141585 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Install linux-sysctl-defaults on trixie

https://gerrit.wikimedia.org/r/1141585

gerritbot added a project: Patch-For-Review.May 5 2025, 6:27 AM
gerritbot added a comment.May 5 2025, 12:32 PM

Change #1141585 merged by Muehlenhoff:

[operations/puppet@production] Install linux-sysctl-defaults on trixie

https://gerrit.wikimedia.org/r/1141585

Maintenance_bot removed a project: Patch-For-Review.May 5 2025, 1:31 PM
gerritbot added a comment.May 9 2025, 5:35 AM

Change #1143703 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Stop installing prometheus-node-exporter on Trixie

https://gerrit.wikimedia.org/r/1143703

gerritbot added a project: Patch-For-Review.May 9 2025, 5:35 AM
Stashbot added a comment.May 9 2025, 9:50 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-09T09:50:42Z] <moritzm> imported debmonitor-client 0.4.0-3+deb13u1 for trixie-wikimedia T391083

Volans subscribed.May 12 2025, 6:54 AM

So the error for the debmonitor client is due by the fact that in /etc/os-release there is no line with VERSION_ID yet, so the client reports unknown as OS and a result the server errors out with:

400 OS name 'unknown' is not valid: {'name': ['The OS name needs to follow the following pattern: ^(Debian( \\d\\d)?|Ubuntu( \\d\\d\\.\\d\\d)?)$']}

Once that reports a proper OS it should be created automatically in the DB, if not we can always add it manually.

MoritzMuehlenhoff added a comment.May 13 2025, 6:24 AM
In T391083#10810288, @Volans wrote:

So the error for the debmonitor client is due by the fact that in /etc/os-release there is no line with VERSION_ID yet, so the client reports unknown as OS and a result the server errors out with:

400 OS name 'unknown' is not valid: {'name': ['The OS name needs to follow the following pattern: ^(Debian( \\d\\d)?|Ubuntu( \\d\\d\\.\\d\\d)?)$']}

Once that reports a proper OS it should be created automatically in the DB, if not we can always add it manually.

This is now fixed in Debian unstable and should progress to testing soon:
https://tracker.debian.org/news/1645404/accepted-base-files-138-source-into-unstable/

I manually installed that version on sretest1001 and that made debmonitor-client run fine: https://debmonitor.wikimedia.org/hosts/sretest1001.eqiad.wmnet

Stashbot added a comment.May 13 2025, 7:54 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-13T07:54:00Z] <moritzm> imported python-wmflib 1.3.1+deb13u1 to trixie-wikimedia T391083

gerritbot added a comment.May 13 2025, 7:56 AM

Change #1143703 merged by Muehlenhoff:

[operations/puppet@production] Stop installing prometheus-ethtool-exporter on Trixie

https://gerrit.wikimedia.org/r/1143703

Stashbot added a comment.May 13 2025, 8:04 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-13T08:04:07Z] <moritzm> imported python-wmflib 1.3.1+deb13u1 to trixie-wikimedia T391083

gerritbot added a comment.May 13 2025, 8:10 AM

Change #1145097 had a related patch set uploaded (by Volans; author: Volans):

[operations/software/pywmflib@master] Add support for Python 3.13 and Debian Trixie

https://gerrit.wikimedia.org/r/1145097

Stashbot added a comment.May 13 2025, 8:12 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-13T08:12:03Z] <moritzm> copied prometheus-rsyslog-exporter 1.0.0+git20221110-1 from bookworm-wikimedia to trixie-wikimedia T391083

gerritbot added a comment.May 13 2025, 8:28 AM

Change #1145100 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Stop installing dstat on Trixie

https://gerrit.wikimedia.org/r/1145100

gerritbot added a comment.May 13 2025, 8:42 AM

Change #1145097 merged by jenkins-bot:

[operations/software/pywmflib@master] Add support for Python 3.13 and Debian Trixie

https://gerrit.wikimedia.org/r/1145097

gerritbot added a comment.May 13 2025, 9:20 AM

Change #1145100 merged by Muehlenhoff:

[operations/puppet@production] Stop installing dstat on Trixie

https://gerrit.wikimedia.org/r/1145100

Maintenance_bot removed a project: Patch-For-Review.May 13 2025, 9:31 AM
ops-monitoring-bot added a comment.May 13 2025, 9:35 AM

Test comment

Stashbot added a comment.May 13 2025, 9:38 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-13T09:38:02Z] <moritzm> imported confd 0.16.0-1+deb13u0 to trixie-wikimedia T391083

MoritzMuehlenhoff added a comment.May 13 2025, 9:58 AM

There's some augeas-related output spam on Puppet runs, already reported as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1098696

gerritbot added a comment.May 13 2025, 12:42 PM

Change #1145187 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] debdeploy: add support for Debian Trixie

https://gerrit.wikimedia.org/r/1145187

gerritbot added a project: Patch-For-Review.May 13 2025, 12:42 PM
gerritbot added a comment.May 13 2025, 12:53 PM

Change #1145187 merged by Volans:

[operations/puppet@production] debdeploy: add support for Debian Trixie

https://gerrit.wikimedia.org/r/1145187

Maintenance_bot removed a project: Patch-For-Review.May 13 2025, 1:31 PM
gerritbot added a comment.May 14 2025, 9:26 AM

Change #1145833 had a related patch set uploaded (by Volans; author: Volans):

[operations/software/debmonitor-client@master] Add support for trixie

https://gerrit.wikimedia.org/r/1145833

gerritbot added a project: Patch-For-Review.May 14 2025, 9:26 AM
gerritbot added a comment.May 19 2025, 9:50 AM

Change #1145833 merged by jenkins-bot:

[operations/software/debmonitor-client@master] Add support for trixie

https://gerrit.wikimedia.org/r/1145833

Maintenance_bot removed a project: Patch-For-Review.May 19 2025, 10:31 AM
jhathaway subscribed.May 19 2025, 2:21 PM
Stashbot added a comment.May 23 2025, 10:31 AM

Mentioned in SAL (#wikimedia-operations) [2025-05-23T10:31:02Z] <moritzm> importing ferm 2.5.1-4+wmf13u1 T391083

gerritbot added a comment.May 30 2025, 9:09 AM

Change #1152246 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] standard_packages: Handle dnsutils/bind9-dnsutils correctly across all supported OSes

https://gerrit.wikimedia.org/r/1152246

gerritbot added a project: Patch-For-Review.May 30 2025, 9:09 AM
Jdforrester-WMF added a parent task: T393173: Publish Wikimedia trixie base Docker image.Jun 5 2025, 8:45 PM
Jdforrester-WMF subscribed.
gerritbot added a comment.Jun 18 2025, 12:08 PM

Change #1152246 merged by Muehlenhoff:

[operations/puppet@production] standard_packages: Handle dnsutils/bind9-dnsutils correctly across all OSes

https://gerrit.wikimedia.org/r/1152246

Maintenance_bot removed a project: Patch-For-Review.Jun 18 2025, 12:31 PM
ops-monitoring-bot added a comment.Jul 11 2025, 7:36 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1003 for host sretest1003.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jul 11 2025, 9:04 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1003 for host sretest1003.eqiad.wmnet with OS trixie executed with errors:

  • sretest1003 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507110802_jmm_1240797_sretest1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console sretest1003.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
Stashbot added a comment.Jul 11 2025, 9:25 AM

Mentioned in SAL (#wikimedia-operations) [2025-07-11T09:25:15Z] <moritzm> imported perccli for trixie-wikimedia T391083

ops-monitoring-bot added a comment.Jul 11 2025, 9:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1003 for host sretest1003.eqiad.wmnet with OS trixie

gerritbot added a comment.Jul 11 2025, 10:25 AM

Change #1168145 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] late-command: Check whether qemu_fw_cfg.ko is present

https://gerrit.wikimedia.org/r/1168145

gerritbot added a project: Patch-For-Review.Jul 11 2025, 10:25 AM
ops-monitoring-bot added a comment.Jul 11 2025, 10:26 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1003 for host sretest1003.eqiad.wmnet with OS trixie executed with errors:

  • sretest1003 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console sretest1003.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
gerritbot added a comment.Jul 11 2025, 12:02 PM

Change #1168145 merged by Muehlenhoff:

[operations/puppet@production] late-command: Check whether qemu_fw_cfg.ko is present

https://gerrit.wikimedia.org/r/1168145

ops-monitoring-bot added a comment.Jul 11 2025, 12:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin1003 for host sretest1003.eqiad.wmnet with OS trixie

Maintenance_bot removed a project: Patch-For-Review.Jul 11 2025, 12:32 PM
ops-monitoring-bot added a comment.Jul 11 2025, 12:52 PM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin1003 for host sretest1003.eqiad.wmnet with OS trixie completed:

  • sretest1003 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507111238_jmm_1269185_sretest1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
MoritzMuehlenhoff added a comment.Jul 11 2025, 12:57 PM

Installations with Trixie are now possible, which directly install the backport of Puppet 7, all known issues affecting the Puppet base classes have been fixed. I'm keeping this bug open to rebase the installer image to newer release candidates, but Trixie can now be used for pilot installations,

MoritzMuehlenhoff closed subtask T392790: Use a forward port of Puppet 7 on Trixie hosts as Resolved.Jul 15 2025, 8:47 AM
gerritbot added a comment.Aug 11 2025, 9:57 AM

Change #1177335 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] debian: Add trixie as a valid codename

https://gerrit.wikimedia.org/r/1177335

gerritbot added a project: Patch-For-Review.Aug 11 2025, 9:57 AM
gerritbot added a comment.Aug 11 2025, 11:01 AM

Change #1177335 merged by Majavah:

[operations/puppet@production] debian: Add trixie as a valid codename

https://gerrit.wikimedia.org/r/1177335

Maintenance_bot removed a project: Patch-For-Review.Aug 11 2025, 11:31 AM
Jdforrester-WMF updated the task description. (Show Details)Aug 11 2025, 3:06 PM
gerritbot added a comment.Aug 13 2025, 7:05 PM

Change #1178604 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] wmflib: add 8.4 as a valid PHP version string, for trixie support

https://gerrit.wikimedia.org/r/1178604

gerritbot added a project: Patch-For-Review.Aug 13 2025, 7:06 PM
gerritbot added a comment.Aug 13 2025, 7:16 PM

Change #1178604 merged by Dzahn:

[operations/puppet@production] wmflib: add 8.4 as a valid PHP version string, for trixie support

https://gerrit.wikimedia.org/r/1178604

Maintenance_bot removed a project: Patch-For-Review.Aug 13 2025, 7:31 PM
MoritzMuehlenhoff closed this task as Resolved.Aug 22 2025, 7:10 AM
MoritzMuehlenhoff claimed this task.

Trixie had the initial stable release on Aug 9 and the installer and base system works fine. Closing this task, all further role adaptions can happen via separate, new bugs.

Stashbot added a comment.Wed, Oct 8, 11:09 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-08T11:09:47Z] <moritzm> imported megacli into thirdparty/hwraid (upstream repo doesn't cover trixie yet, copied over from bookworm) T391083

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits