Page MenuHomePhabricator

LSobanski (Lukasz Sobanski)
Woo$

Today

  • No visible events.

Tomorrow

  • No visible events.

Friday

  • No visible events.

User Details

User Since
Aug 31 2020, 5:40 PM (268 w, 1 d)
Availability
Available
LDAP User
LSobanski
MediaWiki User
LSobanski (WMF) [ Global Accounts ]

Recent Activity

Today

LSobanski created T407946: Alert in need of triage: PeeringBGPDown (instance cr3-eqsin:9804).
Wed, Oct 22, 7:30 AM · netops, Infrastructure-Foundations, sre-alert-triage
LSobanski created T407945: Alert in need of triage: PeeringBGPDown (instance cr2-drmrs:9804).
Wed, Oct 22, 7:30 AM · netops, Infrastructure-Foundations, sre-alert-triage

Yesterday

LSobanski added a project to T407557: OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm: Release-Engineering-Team.

Summarizing what was said above, there are two parts to this request:

  • Host-deployed OpenSSH version, which should be resolved with the upgrade to Bookworm and above
  • Gerrit mina-ssh version, which should be resolved with the upgrade of Gerrit to 3.11 (we're at 3.10 now) - FYI @hashar
Tue, Oct 21, 8:33 AM · Release-Engineering-Team, Infrastructure-Foundations, collaboration-services, GitLab
LSobanski created T407833: Alert in need of triage: PeeringBGPDown (instance cr3-eqsin:9804).
Tue, Oct 21, 7:27 AM · netops, Infrastructure-Foundations, sre-alert-triage

Mon, Oct 20

LSobanski moved T407745: phabricator.wmcloud.org account verification request: jhsoby from Incoming to Backlog on the collaboration-services board.
Mon, Oct 20, 3:44 PM · collaboration-services, VPS-project-Phabricator
LSobanski moved T407458: phabricator.wmcloud.org account verification request: Nehtechnine from Incoming to Backlog on the collaboration-services board.
Mon, Oct 20, 3:44 PM · collaboration-services, VPS-project-Phabricator
LSobanski moved T407616: phabricator.wmcloud.org account verification request: tiisu from Incoming to Backlog on the collaboration-services board.
Mon, Oct 20, 3:44 PM · collaboration-services, VPS-project-Phabricator
LSobanski moved T407671: PuppetFailure - Puppet failure on zuul2001:9100 from Incoming to Work in Progress on the collaboration-services board.
Mon, Oct 20, 3:42 PM · collaboration-services
LSobanski triaged T407671: PuppetFailure - Puppet failure on zuul2001:9100 as Medium priority.
Mon, Oct 20, 3:42 PM · collaboration-services
LSobanski triaged T407616: phabricator.wmcloud.org account verification request: tiisu as Low priority.
Mon, Oct 20, 3:42 PM · collaboration-services, VPS-project-Phabricator
LSobanski triaged T407458: phabricator.wmcloud.org account verification request: Nehtechnine as Low priority.
Mon, Oct 20, 3:42 PM · collaboration-services, VPS-project-Phabricator
LSobanski triaged T407745: phabricator.wmcloud.org account verification request: jhsoby as Low priority.
Mon, Oct 20, 3:42 PM · collaboration-services, VPS-project-Phabricator
LSobanski assigned T407745: phabricator.wmcloud.org account verification request: jhsoby to Dzahn.
Mon, Oct 20, 3:37 PM · collaboration-services, VPS-project-Phabricator
LSobanski assigned T407616: phabricator.wmcloud.org account verification request: tiisu to Dzahn.
Mon, Oct 20, 3:37 PM · collaboration-services, VPS-project-Phabricator
LSobanski moved T406495: Allow Bitu to link Phabricator account from Incoming to Consultation on the collaboration-services board.
Mon, Oct 20, 3:35 PM · Patch-For-Review, collaboration-services, VPS-project-Phabricator, Infrastructure-Foundations, Bitu
LSobanski moved T406824: Evaluate generic backup tooling for object storage buckets from Incoming to Consultation on the collaboration-services board.
Mon, Oct 20, 3:34 PM · Data-Persistence-Backup, collaboration-services, SRE-swift-storage, Ceph
LSobanski assigned T407458: phabricator.wmcloud.org account verification request: Nehtechnine to Dzahn.
Mon, Oct 20, 3:33 PM · collaboration-services, VPS-project-Phabricator
LSobanski triaged T407513: Key packages missing from trixie-wikimedia as Medium priority.
Mon, Oct 20, 2:59 PM · Patch-For-Review, Infrastructure-Foundations, SRE-swift-storage, SRE
LSobanski triaged T407491: Optimize slow maps queries as Medium priority.
Mon, Oct 20, 2:51 PM · Infrastructure-Foundations, Maps, Content-Transform-Team
LSobanski edited projects for T407667: ProbeDown - wdqs1015:443, added: Data-Platform-SRE; removed collaboration-services.
Mon, Oct 20, 7:26 AM · Data-Platform-SRE (2025.10.17 - 2025.11.07)
LSobanski assigned T407671: PuppetFailure - Puppet failure on zuul2001:9100 to Dzahn.
Mon, Oct 20, 7:26 AM · collaboration-services

Fri, Oct 17

LSobanski renamed T407618: Mobileapps requiring a manual restart after a dependency outage from [BUG] to Mobileapps requiring a manual restart after a dependency outage.
Fri, Oct 17, 11:50 AM · Content-Transform-Team, Wikipedia-Android-App-Backlog, Wikipedia-iOS-App-Backlog
LSobanski created T407618: Mobileapps requiring a manual restart after a dependency outage.
Fri, Oct 17, 11:50 AM · Content-Transform-Team, Wikipedia-Android-App-Backlog, Wikipedia-iOS-App-Backlog
LSobanski added a project to T407557: OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm: Infrastructure-Foundations.
Fri, Oct 17, 11:21 AM · Release-Engineering-Team, Infrastructure-Foundations, collaboration-services, GitLab
LSobanski added a comment to T407615: Alert in need of triage: ProbeDown (instance proxoid:4260).

The alert is also firing for codfw.

Fri, Oct 17, 10:52 AM · serviceops, sre-alert-triage
LSobanski created T407615: Alert in need of triage: ProbeDown (instance proxoid:4260).
Fri, Oct 17, 10:52 AM · serviceops, sre-alert-triage

Thu, Oct 16

LSobanski created T407484: Alert in need of triage: PuppetConstantChange (instance prometheus2007:9100).
Thu, Oct 16, 11:25 AM · SRE Observability (FY2025/2026-Q2), sre-alert-triage
LSobanski added a comment to T400969: Alert in need of triage: KubernetesWorkerUnschedulable .

Just a heads up that the alert fired again, can it be silenced for another month?

Thu, Oct 16, 11:24 AM · serviceops, sre-alert-triage
LSobanski moved T407303: SystemdUnitFailed - zuul-web.service on zuul1001:9100 from Incoming to Work in Progress on the collaboration-services board.
Thu, Oct 16, 11:20 AM · collaboration-services
LSobanski moved T406762: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space from Incoming to Work in Progress on the collaboration-services board.
Thu, Oct 16, 11:18 AM · Patch-For-Review, collaboration-services, Gerrit
LSobanski lowered the priority of T406762: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space from High to Medium.
Thu, Oct 16, 11:18 AM · Patch-For-Review, collaboration-services, Gerrit

Tue, Oct 14

LSobanski added a subtask for T387833: Gerrit failover process: Unknown Object (Task).
Tue, Oct 14, 11:58 AM · Patch-For-Review, collaboration-services

Mon, Oct 13

LSobanski edited projects for T406634: Set up a working, usable dbt installation on stat boxes, added: Data-Platform-SRE; removed SRE.

Presumably this is correct, please revert if not.

Mon, Oct 13, 12:51 PM · OKR-Work, Data-Platform-SRE (2025.10.17 - 2025.11.07), Data-Engineering (Q2 FY25/26 October 1st - December 31th)

Tue, Oct 7

LSobanski reassigned T405945: eqiad row C/D Infrastructure Foundations host migrations from LSobanski to cmooney.

@RobH here's a summary of what needs to happen with the hosts, @cmooney will be coordinating the specifics:

Tue, Oct 7, 3:53 PM · Infrastructure-Foundations, SRE, DC-Ops, ops-eqiad
LSobanski closed T309027: Poweredge R730xd, R740xd, R740xd2 SSDs not visible to OS as SSDs as Resolved.

I'll re-close it then and we can get back to the topic when needed.

Tue, Oct 7, 12:16 PM · DC-Ops, Infrastructure-Foundations
LSobanski added a comment to T406141: Disable LVS paging for WDQS.

Now that the other endpoints were added, is there anything else that needs to happen before the patch is deployed?

Tue, Oct 7, 10:38 AM · Essential-Work, Data-Platform-SRE (2025.09.26 - 2025.10.17), Traffic
LSobanski moved T405940: eqiad row C/D Collaboration Services host migrations from Incoming to Work in Progress on the collaboration-services board.
Tue, Oct 7, 9:54 AM · collaboration-services, SRE, DC-Ops, ops-eqiad

Mon, Oct 6

LSobanski moved T403946: Split repository backups (gerrit and gitlab) into its own dedicated storage daemons from Incoming to Consultation on the collaboration-services board.
Mon, Oct 6, 3:50 PM · collaboration-services, GitLab, bacula, Data-Persistence-Backup
LSobanski moved T406333: gerrit: config tweaks from Incoming to Backlog on the collaboration-services board.
Mon, Oct 6, 3:43 PM · Gerrit, collaboration-services
LSobanski triaged T406333: gerrit: config tweaks as Low priority.
Mon, Oct 6, 3:43 PM · Gerrit, collaboration-services
LSobanski moved T406334: Gerrit switchover between secondary instances from Incoming to Backlog on the collaboration-services board.
Mon, Oct 6, 3:42 PM · Gerrit, collaboration-services
LSobanski triaged T406334: Gerrit switchover between secondary instances as Medium priority.
Mon, Oct 6, 3:42 PM · Gerrit, collaboration-services
LSobanski added a comment to T406334: Gerrit switchover between secondary instances.

In scope: decide and document whether we keep the second replica around permanently.

Mon, Oct 6, 3:42 PM · Gerrit, collaboration-services
LSobanski closed T406482: SystemdUnitFailed (jenkins.service on contint1002), a subtask of T387833: Gerrit failover process, as Resolved.
Mon, Oct 6, 3:40 PM · Patch-For-Review, collaboration-services
LSobanski closed T406482: SystemdUnitFailed (jenkins.service on contint1002) as Resolved.

Side effect of T387833: Gerrit failover process

Mon, Oct 6, 3:40 PM · collaboration-services
LSobanski added a comment to T406495: Allow Bitu to link Phabricator account.

There is https://idp.wmcloud.org, which is used for other projects (e.g. GitLab).

Mon, Oct 6, 3:37 PM · Patch-For-Review, collaboration-services, VPS-project-Phabricator, Infrastructure-Foundations, Bitu
LSobanski added a project to T406495: Allow Bitu to link Phabricator account: collaboration-services.
Mon, Oct 6, 3:26 PM · Patch-For-Review, collaboration-services, VPS-project-Phabricator, Infrastructure-Foundations, Bitu
LSobanski added a comment to T309027: Poweredge R730xd, R740xd, R740xd2 SSDs not visible to OS as SSDs.

@MatthewVernon is this still a problem that needs solving. I realize you closed it earlier but double checking.

Mon, Oct 6, 2:49 PM · DC-Ops, Infrastructure-Foundations
LSobanski closed T114446: move human users out of UID range for system accounts as Declined.

This one falls into "too much effort to fix and while technically breaking a rule doesn't cause issues that are worth fixing it"

Mon, Oct 6, 2:48 PM · Infrastructure-Foundations, SRE

Thu, Oct 2

LSobanski closed T402889: Puppet CA certificate Puppet CA: mailman-puppetmaster.mailman.eqiad.wmflabs expired as Resolved.

The certificate renewal was solved by @Dzahn, I'll follow up on the alerting component separately.

Thu, Oct 2, 4:06 PM · SRE Observability, collaboration-services
LSobanski moved T406200: sre.k8s.wipe-cluser JSONDecodeError in kubectl_version() from Incoming to K8s on the collaboration-services board.
Thu, Oct 2, 1:23 PM · Patch-For-Review, collaboration-services, Kubernetes, Prod-Kubernetes, serviceops
LSobanski removed a project from T406213: charlie wiped cluster redeployment use-case: collaboration-services.
Thu, Oct 2, 1:23 PM · Kubernetes, Prod-Kubernetes, serviceops
LSobanski removed a project from T406212: charlie wiped cluster redeployment use-case: collaboration-services.
Thu, Oct 2, 1:23 PM · Patch-For-Review, Kubernetes, Prod-Kubernetes, serviceops
LSobanski removed a project from T406201: kube-scheduler failed to start during sre.k8s.wipe-cluster: collaboration-services.
Thu, Oct 2, 1:23 PM · Kubernetes, Prod-Kubernetes, serviceops

Wed, Oct 1

LSobanski updated the task description for T406141: Disable LVS paging for WDQS.
Wed, Oct 1, 3:38 PM · Essential-Work, Data-Platform-SRE (2025.09.26 - 2025.10.17), Traffic
LSobanski created T406141: Disable LVS paging for WDQS.
Wed, Oct 1, 3:36 PM · Essential-Work, Data-Platform-SRE (2025.09.26 - 2025.10.17), Traffic
LSobanski moved T406034: Znuny LTS 6.5.18 from Incoming to Backlog on the collaboration-services board.
Wed, Oct 1, 10:00 AM · Znuny, collaboration-services
LSobanski triaged T406034: Znuny LTS 6.5.18 as Medium priority.
Wed, Oct 1, 10:00 AM · Znuny, collaboration-services
LSobanski closed T406062: Puppet failure on releases1003:9100 as Resolved.
Wed, Oct 1, 10:00 AM · collaboration-services

Tue, Sep 30

LSobanski moved T378028: Replace Exim on VRTS servers with Postfix from Work in Progress to Backlog on the collaboration-services board.
Tue, Sep 30, 1:51 PM · Patch-For-Review, collaboration-services, vrts, Znuny, Infrastructure-Foundations, Mail, SRE

Mon, Sep 29

LSobanski assigned T405119: Set up zuul web on zuul1001/zuul2001 to Dzahn.
Mon, Sep 29, 4:05 PM · collaboration-services, Essential-Work, Continuous-Integration-Infrastructure (Zuul upgrade)
LSobanski moved T402554: Redirect https://etherpad.wikimedia.org/p/ to https://etherpad.wikimedia.org from Incoming to Backlog on the collaboration-services board.
Mon, Sep 29, 3:50 PM · collaboration-services, Wikimedia-Etherpad
LSobanski triaged T402554: Redirect https://etherpad.wikimedia.org/p/ to https://etherpad.wikimedia.org as Medium priority.
Mon, Sep 29, 3:50 PM · collaboration-services, Wikimedia-Etherpad
LSobanski moved T405596: Disable IO for diffusion repositories from Incoming to Consultation on the collaboration-services board.
Mon, Sep 29, 3:49 PM · Release-Engineering-Team (Priority Backlog 📥), Patch-For-Review, collaboration-services, Diffusion, Phabricator
LSobanski moved T405120: Test new zuul test VMs from Incoming to Consultation on the collaboration-services board.
Mon, Sep 29, 3:49 PM · collaboration-services, Essential-Work, Continuous-Integration-Infrastructure (Zuul upgrade)
LSobanski moved T405118: Set up zuul scheduler on zuul1001 from Work in Progress to Backlog on the collaboration-services board.
Mon, Sep 29, 3:49 PM · collaboration-services, Essential-Work, Continuous-Integration-Infrastructure (Zuul upgrade)
LSobanski assigned T405118: Set up zuul scheduler on zuul1001 to Dzahn.
Mon, Sep 29, 3:49 PM · collaboration-services, Essential-Work, Continuous-Integration-Infrastructure (Zuul upgrade)
LSobanski moved T405118: Set up zuul scheduler on zuul1001 from Incoming to Work in Progress on the collaboration-services board.
Mon, Sep 29, 3:48 PM · collaboration-services, Essential-Work, Continuous-Integration-Infrastructure (Zuul upgrade)
LSobanski triaged T405118: Set up zuul scheduler on zuul1001 as High priority.
Mon, Sep 29, 3:48 PM · collaboration-services, Essential-Work, Continuous-Integration-Infrastructure (Zuul upgrade)
LSobanski raised the priority of T405120: Test new zuul test VMs from High to Needs Triage.
Mon, Sep 29, 3:48 PM · collaboration-services, Essential-Work, Continuous-Integration-Infrastructure (Zuul upgrade)
LSobanski triaged T405120: Test new zuul test VMs as High priority.
Mon, Sep 29, 3:48 PM · collaboration-services, Essential-Work, Continuous-Integration-Infrastructure (Zuul upgrade)
LSobanski moved T405703: Update wikikube eqiad to kubernetes 1.31 from Incoming to K8s on the collaboration-services board.
Mon, Sep 29, 3:47 PM · Discovery-Search (2025.09.26 - 2025.10.17), Data-Platform-SRE (2025.09.26 - 2025.10.17), Patch-For-Review, collaboration-services, Kubernetes, Prod-Kubernetes, serviceops

Fri, Sep 26

LSobanski added a comment to T405706: CI error on operations/cookbooks.

Adding @ltoscano as this is likely to be related to a Dell firmware change.

Fri, Sep 26, 12:45 PM · Infrastructure-Foundations, SRE-tools

Thu, Sep 25

LSobanski placed T402889: Puppet CA certificate Puppet CA: mailman-puppetmaster.mailman.eqiad.wmflabs expired up for grabs.
Thu, Sep 25, 1:30 PM · SRE Observability, collaboration-services
LSobanski added a comment to T402889: Puppet CA certificate Puppet CA: mailman-puppetmaster.mailman.eqiad.wmflabs expired.

None of the current project members outside of Collab have strong opinions on the project configuration so we're OK to make changes as we see fit.

Thu, Sep 25, 1:29 PM · SRE Observability, collaboration-services

Sep 22 2025

LSobanski added a comment to T404630: proposal: allow analytics-admins to also trigger puppet runs.

Approved in the I/F meeting.

Sep 22 2025, 2:45 PM · Data-Platform-SRE (2025.09.05 - 2025.09.26), Data-Engineering-Radar, Data-Engineering, Infrastructure-Foundations
LSobanski created T405217: Alert in need of triage: Dell PowerEdge or Supermicro Broadcom RAID Controller (instance an-worker1187).
Sep 22 2025, 9:31 AM · Data-Platform-SRE (2025.10.17 - 2025.11.07), Essential-Work, sre-alert-triage

Sep 18 2025

LSobanski moved T403663: Upgrade Envoy to v1.29.12 from K8s to Work in Progress on the collaboration-services board.
Sep 18 2025, 4:31 PM · Patch-For-Review, collaboration-services, SRE, serviceops, envoy
LSobanski moved T403663: Upgrade Envoy to v1.29.12 from Incoming to K8s on the collaboration-services board.
Sep 18 2025, 4:30 PM · Patch-For-Review, collaboration-services, SRE, serviceops, envoy
LSobanski created T404946: Alert in need of triage: SwitchCoreInterfaceDown (instance ssw1-f1-codfw:9804).
Sep 18 2025, 8:19 AM · Infrastructure-Foundations, netops, sre-alert-triage

Sep 16 2025

LSobanski created Data-Persistence-Design-Review.
Sep 16 2025, 1:40 PM

Sep 15 2025

LSobanski moved T390948: Cleanup collaboration-services WMCS hiera config from Work in Progress to Backlog on the collaboration-services board.
Sep 15 2025, 3:35 PM · collaboration-services
LSobanski triaged T390948: Cleanup collaboration-services WMCS hiera config as Medium priority.
Sep 15 2025, 3:35 PM · collaboration-services
LSobanski triaged T404111: Znuny LTS 6.5.16 as Medium priority.
Sep 15 2025, 3:29 PM · vrts, collaboration-services, Znuny
LSobanski moved T404111: Znuny LTS 6.5.16 from Incoming to Work in Progress on the collaboration-services board.
Sep 15 2025, 3:29 PM · vrts, collaboration-services, Znuny
LSobanski placed T327771: Test the Znuny DEB package up for grabs.

Based on the above, packaging the files distributed by Znuny ourselves may be the way forward here.

Sep 15 2025, 3:27 PM · collaboration-services
LSobanski moved T402889: Puppet CA certificate Puppet CA: mailman-puppetmaster.mailman.eqiad.wmflabs expired from Incoming to Work in Progress on the collaboration-services board.
Sep 15 2025, 3:24 PM · SRE Observability, collaboration-services
LSobanski claimed T402889: Puppet CA certificate Puppet CA: mailman-puppetmaster.mailman.eqiad.wmflabs expired.
Sep 15 2025, 3:24 PM · SRE Observability, collaboration-services
LSobanski assigned T404437: Allow docker-report to fetch MediaWiki restricted images from the registry to elukey.
Sep 15 2025, 2:35 PM · User-Elukey, Infrastructure-Foundations
LSobanski triaged T404355: secure-cookbook doesn't allow for --dry-run as Medium priority.
Sep 15 2025, 2:31 PM · Infrastructure-Foundations, SRE-tools
LSobanski moved T404478: SystemdUnitFailed - zuul-executor from Incoming to Work in Progress on the collaboration-services board.
Sep 15 2025, 11:33 AM · collaboration-services

Sep 9 2025

LSobanski added a project to T404011: docker-registry will show different last updated time as you refresh the page...: serviceops.
Sep 9 2025, 8:54 AM · serviceops, Release-Engineering-Team, SRE
LSobanski added a project to T404010: docker-registry "Last updated at" time should specify TZ: serviceops.
Sep 9 2025, 8:54 AM · serviceops, Release-Engineering-Team, SRE
LSobanski added a project to T404008: docker-registry "Last updated at" text hiding under scrollbar: serviceops.
Sep 9 2025, 8:54 AM · serviceops, Release-Engineering-Team, SRE
LSobanski edited projects for T404040: ProbeDown, added: Data-Platform-SRE; removed collaboration-services.
Sep 9 2025, 8:03 AM · Data-Platform-SRE (2025.10.17 - 2025.11.07), Essential-Work

Sep 8 2025

LSobanski added a comment to T402554: Redirect https://etherpad.wikimedia.org/p/ to https://etherpad.wikimedia.org.

If I'm reading both correctly the request in the task is the opposite of the one in the wiki discussion. Looking at https://meta.wikimedia.org/wiki/Interwiki_map/list, the request in the task is incorrect.

Sep 8 2025, 3:56 PM · collaboration-services, Wikimedia-Etherpad
LSobanski added a comment to T402889: Puppet CA certificate Puppet CA: mailman-puppetmaster.mailman.eqiad.wmflabs expired.

It's not clear whether we actually need the separate puppetmaster, looking into that.

Sep 8 2025, 3:45 PM · SRE Observability, collaboration-services
LSobanski moved T403847: Deploy zuul executor on executor VM from Incoming to Work in Progress on the collaboration-services board.
Sep 8 2025, 3:38 PM · Release-Engineering-Team (Priority Backlog 📥), collaboration-services, Continuous-Integration-Infrastructure (Zuul upgrade)
LSobanski changed the status of T403847: Deploy zuul executor on executor VM from Open to In Progress.
Sep 8 2025, 3:38 PM · Release-Engineering-Team (Priority Backlog 📥), collaboration-services, Continuous-Integration-Infrastructure (Zuul upgrade)
LSobanski changed the status of T403847: Deploy zuul executor on executor VM, a subtask of T395938: puppetize setup of new zuul VMs, from Open to In Progress.
Sep 8 2025, 3:38 PM · Patch-For-Review, collaboration-services, Continuous-Integration-Infrastructure (Zuul upgrade)