Page MenuHomePhabricator
Create Task
Maniphest T387833

Gerrit failover process
Open, In Progress, HighPublic
Actions

Description

A part of the effort to standardize failover procedures for Collab services.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
gerrit: unmask service & disable backup temporarilyoperations/puppetproduction+5 -2
gerrit: stop stopping gerrit.serviceoperations/cookbooksmaster+0 -8
gerrit: stop puppet across all instancesoperations/cookbooksmaster+6 -4
gerrit: rsync and chown fixesoperations/cookbooksmaster+12 -7
gerrit: disable gerrit service to enable backupsoperations/puppetproduction+2 -2
gerrit: ask the operator to merge puppet earlieroperations/cookbooksmaster+5 -3
gerrit: typo fix in post_sync_validationoperations/cookbooksmaster+1 -1
gerrit: re-enable backups on gerrit2003operations/puppetproduction+0 -2
Revert^4 "gerrit: switchover from gerrit1003 to gerrit2003"operations/dnsmaster+8 -8
Revert^4 "gerrit: Switchover gerrit1003 → gerrit2003"operations/puppetproduction+70 -33
gerrit: add dry run rsyncoperations/cookbooksmaster+12 -6
gerrit: local backup on source server onlyoperations/cookbooksmaster+4 -4
gerrit: remove localbackup logic from failoveroperations/cookbooksmaster+7 -43
gerrit: add a local backup cookbookoperations/cookbooksmaster+144 -0
Revert^2 "gerrit: switchover from gerrit1003 to gerrit2003"operations/dnsmaster+8 -8
Revert^2 "gerrit: Switchover gerrit1003 → gerrit2003"operations/puppetproduction+70 -33
gerrit: Switchover gerrit1003 → gerrit2003operations/puppetproduction+70 -33
Add a banner for a Gerrit switch over maintenanceoperations/software/gerritdeploy/wmf/stable-3.10+114 -0
Disable component rather than motd pluginoperations/software/gerritdeploy/wmf/stable-3.10+4 -4
Disable motd banner: maintenance window has closedoperations/software/gerritdeploy/wmf/stable-3.10+5 -3
gerrit: fix typo in source pathoperations/cookbooksmaster+9 -10
gerrit: test 1193860operations/cookbooksmaster+98 -97
gerrit: switchover from gerrit1003 to gerrit2003operations/dnsmaster+8 -8
gerrit: avoid hardcoded hostnames, replace with hiera lookupsoperations/puppetproduction+5 -3
gerrit: typo on --systemd argoperations/cookbooksmaster+5 -3
gerrit: bugfixes on failoveroperations/cookbooksmaster+24 -9
gerrit: replace host names in replica config with variablesoperations/puppetproduction+2 -2
gerrit: Switchover gerrit1003 → gerrit2003operations/puppetproduction+46 -30
gerrit: add spare fqdn to apache vhostoperations/puppetproduction+19 -0
gerrit: add certificate request for gerrit-spare.w.ooperations/puppetproduction+1 -0
gerrit: site.pp cleanupoperations/puppetproduction+1 -6
gerrit: add httpbb monitoring to gerrit-spareoperations/puppetproduction+19 -0
gerrit: add service ip address for gerrit2003operations/puppetproduction+5 -3
Gerrit: Add service ip for gerrit2003operations/dnsmaster+8 -3
gerrit: Bugfixes - dry run testsoperations/cookbooksmaster+64 -218
gerrit: rename host and replica_hosts array to primary/replica_vhostoperations/puppetproduction+18 -21
gerrit: enable monitoring for other instancesoperations/puppetproduction+25 -0
gerrit: also rename "passive" to "spare" server in MOTDoperations/puppetproduction+1 -1
gerrit: replace hardcoded host name and codfw string for replicaoperations/puppetproduction+11 -2
Add readonly puginoperations/software/gerritdeploy/wmf/stable-3.10+3 -0
gerrit: standardize expected rc on systemctl checkoperations/cookbooksmaster+1 -1
gerrit: sanity checks cookbook implementationoperations/cookbooksmaster+52 -150
gerrit: sanity checks as a cookbookoperations/cookbooksmaster+304 -0
gerrit: add readonly pluginoperations/software/gerritwmf/stable-3.10+4 -0
gerrit: add readonly pluginoperations/software/gerritwmf/stable-3.10+4 -0
gerrit: add readonly.jaroperations/software/gerritdeploy/wmf/stable-3.10+0 -0
gerrit: introduce second daemon_user nameoperations/puppetproduction+3 -2
gerrit: replace gerrit2003 RSA host key with ed25519 host keyoperations/puppetproduction+1 -1
gerrit: add a second replica, start replicating to gerrit2003operations/puppetproduction+16 -8
gerrit: add ssh_host_rsa public key for gerrit2003operations/puppetproduction+1 -0
gerrit: switchover to gerrit1003operations/puppetproduction+5 -3
gerrit: switchover to gerrit1003operations/dnsmaster+8 -8
gerrit: switchover to gerrit2002operations/puppetproduction+3 -3
gerrit: switchover to gerrit2002operations/dnsmaster+8 -8
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT407557 OpenSSH 10.1+ warns that Wikimedia SSH does not use post-quantum key exchange algorithm
 OpenNoneT407844 Gerrit ssh daemon does not offer post-quantum kex leading to a warning with OpenSSH 10
 OpenNoneT392448 Upgrade to Gerrit 3.12
 OpenNoneT379714 Upgrade to Gerrit 3.11
 OpenNoneT392465 Switch Gerrit from Java 17 to Java 21
 OpenNoneT384595 Upgrade Collab hosts to Bookworm
 OpenNoneT392464 Upgrade Gerrit hosts from Bullseye to Bookworm
 OpenNoneT387831 Standardize failover procedures for Collab services
 ResolvedNoneT393239 ProbeDown
 In ProgressABran-WMFT387833 Gerrit failover process
 OpenNoneT388507 Investigate missing commit from old changes
 ResolvedABran-WMFT260666 Create a cookbook to automate gerrit's switchover
 ResolvedABran-WMFT393050 ProbeDown
 In ProgressABran-WMFT338470 Rename gerrit2 unix user to gerrit and assign a fixed uid
 ResolvedDzahnT379213 PuppetFailure - gerrit2003
 ResolvedMatthewVernonT392186 Grant Gerrit admin to arnaudb
 ResolvedLSobanskiT393034 Investigate out of date refs following gerrit switchover
 ResolvedABran-WMFT395440 Gerrit read-only plugin test
 OpenABran-WMFT406334 Gerrit switchover between secondary instances
 ResolvedLSobanskiT406482 SystemdUnitFailed (jenkins.service on contint1002)
 ResolvedABran-WMFT406762 gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space
 OpenNoneT406856 Reduce size of analytics/superset/deploy.git Gerrit repo
Restricted Task

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Jelto added a comment.Aug 4 2025, 2:40 PM

In T400971 it was noticed that nftables connection tracking/metering might change when switching from bullseye to bookworm to a newer nftables version. So before switching the hosts we should make sure reasonable thresholds are configured. Alternatively we can monitor the denylist after the switchover closely.

hashar added a parent task: T392464: Upgrade Gerrit hosts from Bullseye to Bookworm.Aug 10 2025, 12:30 PM
ABran-WMF updated the task description. (Show Details)Aug 11 2025, 2:12 PM
Paladox subscribed.Aug 11 2025, 9:47 PM
gerritbot added a comment.Aug 13 2025, 5:46 AM

Change #1178172 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: add spare fqdn to apache vhost

https://gerrit.wikimedia.org/r/1178172

gerritbot added a comment.Aug 14 2025, 5:23 AM

Change #1178172 merged by Arnaudb:

[operations/puppet@production] gerrit: add spare fqdn to apache vhost

https://gerrit.wikimedia.org/r/1178172

ABran-WMF added a comment.Aug 21 2025, 5:10 AM
In T387833#11057848, @Jelto wrote:

In T400971 it was noticed that nftables connection tracking/metering might change when switching from bullseye to bookworm to a newer nftables version. So before switching the hosts we should make sure reasonable thresholds are configured. Alternatively we can monitor the denylist after the switchover closely.

thanks for highlighting this blocking issue @Jelto, with T400971: Troubleshoot GitLab nftables throttling after switchover we managed to convert the throttling logic to something properly interpreted by both nftables versions.

gerritbot added a comment.Aug 26 2025, 6:09 PM

Change #1170433 merged by Dzahn:

[operations/puppet@production] gerrit: replace host names in replica config with variables

https://gerrit.wikimedia.org/r/1170433

gerritbot added a comment.Sep 15 2025, 1:31 PM

Change #1188351 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: Switchover gerrit1003 → gerrit2003

https://gerrit.wikimedia.org/r/1188351

gerritbot added a comment.Sep 15 2025, 1:47 PM

Change #1172625 abandoned by Arnaudb:

[operations/puppet@production] gerrit: Switchover gerrit1003 → gerrit2003

Reason:

replaced by 1188351

https://gerrit.wikimedia.org/r/1172625

ABran-WMF updated the task description. (Show Details)Fri, Sep 26, 8:16 AM
ABran-WMF mentioned this in T405706: CI error on operations/cookbooks.Fri, Sep 26, 8:19 AM
gerritbot added a comment.Fri, Sep 26, 8:23 AM

Change #1191431 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: bugfixes on failover

https://gerrit.wikimedia.org/r/1191431

gerritbot added a comment.Mon, Sep 29, 12:27 PM

Change #1191431 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: bugfixes on failover

https://gerrit.wikimedia.org/r/1191431

ABran-WMF added a comment.EditedMon, Sep 29, 1:48 PM

the dry run switchover output looks alright {P83480}

I've added the entry to the deployment calendar - next switchover is scheduled on next Monday (Oct. 6)

ABran-WMF updated the task description. (Show Details)Mon, Sep 29, 1:48 PM
gerritbot added a comment.Thu, Oct 2, 7:11 AM

Change #1193017 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/software/gerrit@deploy/wmf/stable-3.10] Add a banner for a Gerrit switch over maintenance

https://gerrit.wikimedia.org/r/1193017

hashar added a comment.Thu, Oct 2, 7:13 AM

On top of your wikitech-l announce, I have proposed a change to add a banner at the top of the Gerrit web UI that will looks like:

gerrit_switch_over_20251006.png (76×952 px, 19 KB)

Follows up on https://gerrit.wikimedia.org/r/c/operations/software/gerrit/+/1193017

ABran-WMF awarded a token.Thu, Oct 2, 7:36 AM
gerritbot added a comment.Thu, Oct 2, 8:35 AM

Change #1193017 merged by jenkins-bot:

[operations/software/gerrit@deploy/wmf/stable-3.10] Add a banner for a Gerrit switch over maintenance

https://gerrit.wikimedia.org/r/1193017

Stashbot added a comment.Thu, Oct 2, 8:35 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T08:35:34Z] <hashar@deploy2002> Started deploy [gerrit/gerrit@3ef5714]: Add a banner for a Gerrit switch over maintenance - T387833

Stashbot added a comment.Thu, Oct 2, 8:35 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T08:35:40Z] <hashar@deploy2002> deploy aborted: Add a banner for a Gerrit switch over maintenance - T387833 (duration: 00m 00s)

Stashbot added a comment.Thu, Oct 2, 8:35 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T08:35:45Z] <hashar@deploy2002> Started deploy [gerrit/gerrit@3ef5714]: Add a banner for a Gerrit switch over maintenance - T387833

Stashbot added a comment.Thu, Oct 2, 8:35 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-02T08:35:52Z] <hashar@deploy2002> Finished deploy [gerrit/gerrit@3ef5714]: Add a banner for a Gerrit switch over maintenance - T387833 (duration: 00m 12s)

gerritbot added a comment.Thu, Oct 2, 9:01 AM

Change #1193051 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: typo on --systemd arg

https://gerrit.wikimedia.org/r/1193051

ABran-WMF updated the task description. (Show Details)Thu, Oct 2, 9:03 AM
gerritbot added a comment.Thu, Oct 2, 9:51 AM

Change #1193051 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: typo on --systemd arg

https://gerrit.wikimedia.org/r/1193051

gerritbot added a comment.Thu, Oct 2, 11:38 AM

Change #1193082 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/dns@master] gerrit: switchover from gerrit1003 to gerrit2003

https://gerrit.wikimedia.org/r/1193082

ABran-WMF updated the task description. (Show Details)Thu, Oct 2, 11:58 AM
ABran-WMF changed the status of subtask T338470: Rename gerrit2 unix user to gerrit and assign a fixed uid from Stalled to In Progress.Thu, Oct 2, 12:05 PM
ABran-WMF updated the task description. (Show Details)
Tacsipacsi subscribed.Thu, Oct 2, 8:18 PM
ABran-WMF mentioned this in T406334: Gerrit switchover between secondary instances.Fri, Oct 3, 1:35 PM
gerritbot added a comment.Mon, Oct 6, 4:49 AM

Change #1193590 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: add a local backup cookbook

https://gerrit.wikimedia.org/r/1193590

gerritbot added a comment.Mon, Oct 6, 5:59 AM

Change #1193599 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: remove localbackup logic from failover

https://gerrit.wikimedia.org/r/1193599

Jelto mentioned this in T403946: Split repository backups (gerrit and gitlab) into its own dedicated storage daemons.Mon, Oct 6, 9:22 AM
gerritbot added a comment.Mon, Oct 6, 12:02 PM

Change #1193082 merged by Arnaudb:

[operations/dns@master] gerrit: switchover from gerrit1003 to gerrit2003

https://gerrit.wikimedia.org/r/1193082

gerritbot added a comment.Mon, Oct 6, 12:02 PM

Change #1188351 merged by Arnaudb:

[operations/puppet@production] gerrit: Switchover gerrit1003 → gerrit2003

https://gerrit.wikimedia.org/r/1188351

Jelto added a subtask: T406482: SystemdUnitFailed (jenkins.service on contint1002).Mon, Oct 6, 12:54 PM
Jelto added a comment.Mon, Oct 6, 1:20 PM

I tried to extract the error from the failed cookbook execution from the cumin log:

2025-10-06 12:29:12,789 arnaudb 1516308 [WARNING cookbooks.sre.gerrit.failover:337 in sync_files] There will be a sync retry, expect no more than 10 of these.
2025-10-06 12:29:12,790 arnaudb 1516308 [DEBUG spicerack.remote:750 in _execute] Executing commands ['/usr/bin/rsync -avpPz --stats --delete /var/lib/gerrit/review_site  rsync://gerrit2003.wikimedia.org/gerrit-var-lib/'] on 1 hosts: gerrit1003.wikimedia.org
2025-10-06 12:29:12,791 arnaudb 1516308 [INFO cumin.transports.clustershell.ClusterShellWorker:78 in execute] Executing commands [cumin.transports.Command('/usr/bin/rsync -avpPz --stats --delete /var/lib/gerrit/review_site  rsync://gerrit2003.wikimedia.org/gerrit-var-lib/')] on '1' hosts: gerrit1003.wikimedia.org
2025-10-06 12:29:12,798 arnaudb 1516308 [DEBUG cumin.transports.clustershell.SyncEventHandler:590 in ev_pickup] node=gerrit1003.wikimedia.org, command='/usr/bin/rsync -avpPz --stats --delete /var/lib/gerrit/review_site  rsync://gerrit2003.wikimedia.org/gerrit-var-lib/'
2025-10-06 12:29:14,019 arnaudb 1516308 [DEBUG cumin.transports.clustershell.SyncEventHandler:783 in ev_hup] node=gerrit1003.wikimedia.org, rc=23, command='/usr/bin/rsync -avpPz --stats --delete /var/lib/gerrit/review_site  rsync://gerrit2003.wikimedia.org/gerrit-var-lib/'
2025-10-06 12:29:14,019 arnaudb 1516308 [INFO cumin.transports.clustershell.SyncEventHandler:853 in ev_timer] Completed command '/usr/bin/rsync -avpPz --stats --delete /var/lib/gerrit/review_site  rsync://gerrit2003.wikimedia.org/gerrit-var-lib/'
2025-10-06 12:29:14,020 arnaudb 1516308 [ERROR spicerack._menu:292 in _run] Exception raised while executing cookbook sre.gerrit.failover:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 265, in _run
    raw_ret = runner.run()
              ^^^^^^^^^^^^
  File "/srv/deployment/spicerack/cookbooks/sre/gerrit/failover.py", line 250, in run
    self.sync_files(idempotent=True, all_dirs=True)
  File "/usr/lib/python3/dist-packages/wmflib/decorators.py", line 231, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/srv/deployment/spicerack/cookbooks/sre/gerrit/failover.py", line 363, in sync_files
    self.switch_from_host.run_sync(
  File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 556, in run_sync
    return self._execute(
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 763, in _execute
    raise RemoteExecutionError(ret, "Cumin execution failed", worker.get_results())
spicerack.remote.RemoteExecutionError: Cumin execution failed (exit_code=2)
2025-10-06 12:29:14,027 arnaudb 1516308 [DEBUG spicerack.locking:234 in release] Releasing lock for key sre.gerrit.failover with ID ddcb30ae-82c5-462c-a63b-14f50d593d0f
2025-10-06 12:29:14,027 arnaudb 1516308 [DEBUG etcd.lock:23 in __init__] Initiating lock for /spicerack/locks/etcd with uuid 42039cefcf5449208340942e1a7afa7e
2025-10-06 12:29:14,027 arnaudb 1516308 [DEBUG etcd.client:582 in read] Issuing read for key /spicerack/locks/etcd with args {'recursive': True}
2025-10-06 12:29:14,032 arnaudb 1516308 [DEBUG etcd.lock:67 in acquire] Lock not found, writing it to /spicerack/locks/etcd
2025-10-06 12:29:14,032 arnaudb 1516308 [DEBUG etcd.client:471 in write] Writing 42039cefcf5449208340942e1a7afa7e to key /spicerack/locks/etcd ttl=15 dir=False append=True
ABran-WMF added a comment.Mon, Oct 6, 1:22 PM

thanks for that @Jelto I'll try and reproduce the error in a controlled environment

gerritbot added a comment.Mon, Oct 6, 1:45 PM

Change #1193860 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: fix ownership at rsync time

https://gerrit.wikimedia.org/r/1193860

gerritbot added a comment.Mon, Oct 6, 1:53 PM

Change #1193846 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/dns@master] Revert^2 "gerrit: switchover from gerrit1003 to gerrit2003"

https://gerrit.wikimedia.org/r/1193846

gerritbot added a comment.Mon, Oct 6, 1:54 PM

Change #1193845 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] Revert^2 "gerrit: Switchover gerrit1003 → gerrit2003"

https://gerrit.wikimedia.org/r/1193845

gerritbot added a comment.Mon, Oct 6, 2:08 PM

Change #1193865 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: test 1193860

https://gerrit.wikimedia.org/r/1193865

LSobanski closed subtask T406482: SystemdUnitFailed (jenkins.service on contint1002) as Resolved.Mon, Oct 6, 3:40 PM
LSobanski mentioned this in T406482: SystemdUnitFailed (jenkins.service on contint1002).
gerritbot added a comment.Tue, Oct 7, 7:04 AM

Change #1193865 abandoned by Arnaudb:

[operations/cookbooks@master] gerrit: test 1193860

Reason:

test OK

https://gerrit.wikimedia.org/r/1193865

ABran-WMF added a comment.Tue, Oct 7, 7:09 AM
DRY-RUN: Executing commands ['/usr/bin/rsync -avpPz --stats --delete /var/lib/gerrit2/review_site  rsync://gerrit2003.wikimedia.org/gerrit-var-lib/ --no-o --no-g --chown=gerrit:gerrit '] on 1 hosts: gerrit1003.wikimedia.org
DRY-RUN: Releasing lock for key sre.gerrit.failover with ID abe63737-d8ed-498c-af9d-f71d5fe4d64c

I initially thought the issue came from target directory ownership, but it was a typo on the source path used for rsync. Fixed rsync has been tested in operations/cookbooks/+/1193865 and the fix has been backported to operations/cookbooks/+/1193860

Stashbot added a comment.Tue, Oct 7, 8:37 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-07T08:37:55Z] <hashar> Stopped Gerrit on gerrit2003, deleted /srv/gerrit/git/* and restarted a full replication due to bad files ownership # T387833

gerritbot added a comment.Tue, Oct 7, 9:18 AM

Change #1193860 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: fix typo in source path

https://gerrit.wikimedia.org/r/1193860

ABran-WMF updated the task description. (Show Details)Tue, Oct 7, 2:04 PM
gerritbot added a comment.Tue, Oct 7, 2:55 PM

Change #1194220 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/software/gerrit@deploy/wmf/stable-3.10] Disable motd banner: maintenance window has closed

https://gerrit.wikimedia.org/r/1194220

gerritbot added a comment.Tue, Oct 7, 3:01 PM

Change #1194220 merged by jenkins-bot:

[operations/software/gerrit@deploy/wmf/stable-3.10] Disable motd banner: maintenance window has closed

https://gerrit.wikimedia.org/r/1194220

Stashbot added a comment.Tue, Oct 7, 3:03 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-07T15:03:16Z] <hashar@deploy2002> Started deploy [gerrit/gerrit@21d2848]: Disable motd banner: maintenance window has closed - T387833

Stashbot added a comment.Tue, Oct 7, 3:03 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-07T15:03:37Z] <hashar@deploy2002> Finished deploy [gerrit/gerrit@21d2848]: Disable motd banner: maintenance window has closed - T387833 (duration: 00m 30s)

gerritbot added a comment.Tue, Oct 7, 3:21 PM

Change #1194225 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/software/gerrit@deploy/wmf/stable-3.10] Disable component rather than motd plugin

https://gerrit.wikimedia.org/r/1194225

gerritbot added a comment.Tue, Oct 7, 3:23 PM

Change #1194225 merged by jenkins-bot:

[operations/software/gerrit@deploy/wmf/stable-3.10] Disable component rather than motd plugin

https://gerrit.wikimedia.org/r/1194225

ABran-WMF updated the task description. (Show Details)Thu, Oct 9, 6:06 AM
gerritbot added a comment.Thu, Oct 9, 12:01 PM

Change #1193845 merged by Arnaudb:

[operations/puppet@production] Revert^2 "gerrit: Switchover gerrit1003 → gerrit2003"

https://gerrit.wikimedia.org/r/1193845

gerritbot added a comment.Thu, Oct 9, 12:02 PM

Change #1193846 merged by Arnaudb:

[operations/dns@master] Revert^2 "gerrit: switchover from gerrit1003 to gerrit2003"

https://gerrit.wikimedia.org/r/1193846

ABran-WMF added a subtask: T406762: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space.Thu, Oct 9, 1:17 PM
gerritbot added a comment.Thu, Oct 9, 1:20 PM

Change #1194932 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/dns@master] Revert^4 "gerrit: switchover from gerrit1003 to gerrit2003"

https://gerrit.wikimedia.org/r/1194932

gerritbot added a comment.Thu, Oct 9, 1:21 PM

Change #1194931 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] Revert^4 "gerrit: Switchover gerrit1003 → gerrit2003"

https://gerrit.wikimedia.org/r/1194931

gerritbot added a comment.Thu, Oct 9, 1:58 PM

Change #1194949 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: local backup on source server only

https://gerrit.wikimedia.org/r/1194949

ABran-WMF added a subtask: T406856: Reduce size of analytics/superset/deploy.git Gerrit repo.Thu, Oct 9, 2:23 PM
ABran-WMF added a comment.EditedMon, Oct 13, 4:47 AM

things are looking better now:

arnaudb@gerrit2003:git $ fd | wc -l
236218
arnaudb@gerrit2003:git $ pwd
/srv/gerrit/git
arnaudb@gerrit2003:git $ ls -l /srv/backup/
total 0

vs the previous situation:

In T406762#11259697, @hashar wrote:
gerrit 2003$ find /srv/gerrit/git |wc -l
5037253

That is 5 millions files.

[...]

 JobId  Level      Files    Bytes   Status   Finished        Name 
====================================================================
656136  Incr    3,745,079    54.98 G  OK       08-Oct-25 13:09 gerrit2003.wikimedia.org-Hourly-Mon-productionEqiad-gerrit-repo-data
ABran-WMF mentioned this in T406762: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space.Mon, Oct 13, 4:49 AM
gerritbot added a comment.Mon, Oct 13, 4:52 AM

Change #1195432 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: re-enable backups on gerrit2003

https://gerrit.wikimedia.org/r/1195432

gerritbot added a comment.Mon, Oct 13, 5:38 AM

Change #1195437 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: add dry run rsync

https://gerrit.wikimedia.org/r/1195437

ABran-WMF updated the task description. (Show Details)Mon, Oct 13, 7:29 AM
gerritbot added a comment.Mon, Oct 13, 8:34 AM

Change #1194949 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: local backup on source server only

https://gerrit.wikimedia.org/r/1194949

gerritbot added a comment.Tue, Oct 14, 9:00 AM

Change #1194932 merged by Arnaudb:

[operations/dns@master] Revert^4 "gerrit: switchover from gerrit1003 to gerrit2003"

https://gerrit.wikimedia.org/r/1194932

gerritbot added a comment.Tue, Oct 14, 9:00 AM

Change #1194931 merged by Arnaudb:

[operations/puppet@production] Revert^4 "gerrit: Switchover gerrit1003 → gerrit2003"

https://gerrit.wikimedia.org/r/1194931

LSobanski added a subtask: Restricted Task.Tue, Oct 14, 11:58 AM
gerritbot added a comment.Tue, Oct 14, 12:31 PM

Change #1196051 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: typo fix in post_sync_validation

https://gerrit.wikimedia.org/r/1196051

ABran-WMF closed subtask Restricted Task as Resolved.Tue, Oct 14, 12:43 PM
gerritbot added a comment.Wed, Oct 15, 6:33 AM

Change #1196227 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: ask the operator to merge puppet earlier

https://gerrit.wikimedia.org/r/1196227

gerritbot added a comment.Wed, Oct 15, 6:27 PM

Change #1195432 merged by Dzahn:

[operations/puppet@production] gerrit: re-enable backups on gerrit2003

https://gerrit.wikimedia.org/r/1195432

gerritbot added a comment.Wed, Oct 15, 6:28 PM

Change #1196051 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: typo fix in post_sync_validation

https://gerrit.wikimedia.org/r/1196051

gerritbot added a comment.Thu, Oct 16, 8:45 AM

Change #1196629 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: disable gerrit service to enable backups

https://gerrit.wikimedia.org/r/1196629

gerritbot added a comment.Thu, Oct 16, 8:47 AM

Change #1196227 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: ask the operator to merge puppet earlier

https://gerrit.wikimedia.org/r/1196227

gerritbot added a comment.Thu, Oct 16, 1:21 PM

Change #1196684 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: rsync and chown fixes

https://gerrit.wikimedia.org/r/1196684

gerritbot added a comment.Thu, Oct 16, 1:56 PM

Change #1196694 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: stop puppet across all instances

https://gerrit.wikimedia.org/r/1196694

gerritbot added a comment.Thu, Oct 16, 2:00 PM

Change #1196695 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] gerrit: stop stopping gerrit.service

https://gerrit.wikimedia.org/r/1196695

gerritbot added a comment.Thu, Oct 16, 8:30 PM

Change #1196629 merged by Dzahn:

[operations/puppet@production] gerrit: disable gerrit service to enable backups

https://gerrit.wikimedia.org/r/1196629

Dzahn added a comment.Thu, Oct 16, 8:46 PM

Here are the notes / commands from a gerrit failover in the past:

https://phabricator.wikimedia.org/P47782

Dzahn added a comment.Thu, Oct 16, 8:48 PM

Here is how we did the DNS change without having to merge while Gerrit is down:

  1. merge DNS change that removes gerrit-new and switches IP of gerrit.wikimedia.org - in web UI of gerrit(-old)
  2. run authdns-update on ns0.wikimedia.org, see the diff but do NOT commit yet
  3. disable puppet, stop gerrit, do the rsync, run chmod -R ...
  4. say "yes" to authdns-update and actually merge DNS change that removes gerrit-new and switches IP of gerrit.wikimedia.org
gerritbot added a comment.Fri, Oct 17, 6:13 AM

Change #1196792 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: unmask service & disable backup temporarily

https://gerrit.wikimedia.org/r/1196792

ABran-WMF added a comment.Fri, Oct 17, 6:40 AM
In T387833#11283327, @Dzahn wrote:

Here is how we did the DNS change without having to merge while Gerrit is down:

Thanks for the dig! I tweaked the process and the cookbook to be closer to this, the puppet merge timing was inconsistent with that approach.

ABran-WMF closed subtask T406762: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space as Resolved.Fri, Oct 17, 6:45 AM
gerritbot added a comment.Fri, Oct 17, 6:49 AM

Change #1196684 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: rsync and chown fixes

https://gerrit.wikimedia.org/r/1196684

gerritbot added a comment.Fri, Oct 17, 6:50 AM

Change #1196694 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: stop puppet across all instances

https://gerrit.wikimedia.org/r/1196694

gerritbot added a comment.Fri, Oct 17, 7:49 AM

Change #1196695 merged by jenkins-bot:

[operations/cookbooks@master] gerrit: stop stopping gerrit.service

https://gerrit.wikimedia.org/r/1196695

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits