In T408032#11300632, @Paladox wrote:

Primary git hosting is Gerrit (https://review.opendev.org/) with gitea used as a repo viewer.

Can we build a POC for this in Cloud VPS somewhere? My naive assumption is that we would need a fair amount of disk storage (150G?) to keep a clone of every repo in gerrit, but that seems to be the biggest expense.

@Xqt I see passing tests upstream marked as using "wpbeta" (e.g. https://github.com/wikimedia/pywikibot/actions/runs/18721896853/job/53396228584). Does this mean that things are working now? Can this task be updated with info about whay y'all had to change and resolved if so?

The background is what has changed, not the logo. But yes it looks bad now.

@JJMC89 has called for a vote on the proposed wording the mailing list. The vote will end 31 October 2025 AoE.

I suspect this check is no longer relevant and could be removed entirely:

Already resolved by the time I manually checked the report:

bd808@deployment-webperf21:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-webperf21.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(7f48520a8f) gitpuppet - [BETA HACK] Changes to profile::puppetserver::volatile'
Notice: /Stage[main]/Webperf::Statsv/Systemd::Service[statsv]/Service[statsv]/ensure: ensure changed 'stopped' to 'running' (corrective)
Info: /Stage[main]/Webperf::Statsv/Systemd::Service[statsv]/Service[statsv]: Unscheduling refresh on Service[statsv]
Notice: Applied catalog in 8.29 seconds

Already resolved by the time I manually checked the report:

bd808@deployment-mwmaint03:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-mwmaint03.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(7f48520a8f) gitpuppet - [BETA HACK] Changes to profile::puppetserver::volatile'
Notice: Applied catalog in 25.95 seconds

In T396924#11153735, @bd808 wrote:

At the moment a job will still need to set a KOKKURI_REGISTRY_PUBLIC: registry.cloud.releng.team envvar in the job config to point kokkuri at the registry. That is a relatively simple addition and also something we can push up into the shared config at some point.

In T405596#11282717, @A_smart_kitten wrote:

In T405596#11244081, @Aklapper wrote:

• For Diffusion repos which observe a gerrit or gitlab URI and also mirror to github [1], change that setup not to have Diffusion "in the middle", per T405596#11242761.

I can't immediately cite a source for this but I feel like GitLab repos don't automatically mirror to GitHub (in the way that Gerrit repos do). Wouldn't disabling GitHub mirroring for these GitLab Diffusion-repo-mirrors stop the GitHub-repo-mirrors from being updated (or am I misunderstanding something)?

I maintain/co-maintain a number of tools that could be offered up for tribute here if desired:

• https://www.mediawiki.org/wiki/Wikibugs - webservice, irc bouncer, irc bot, ssh to gerrit, SSE to https://gitlab-webhooks.toolforge.org/, phabricator API polling
• https://wikitech.wikimedia.org/wiki/Tool:Stashbot - legacy python irc bot with Elasticsearch storage backend, custom k8s Deployment
• https://wikitech.wikimedia.org/wiki/Tool:SAL - legacy php webservice exposing Elasticsearch storage backend maintained by Stashbot
• https://wikitech.wikimedia.org/wiki/Tool:Bash - legacy php webservice exposing Elasticsearch storage backend maintained by Stashbot, Elasticsearch writes
• https://wikitech.wikimedia.org/wiki/Tool:Bridgebot - irc bouncer + https://github.com/42wim/matterbridge
• https://wikitech.wikimedia.org/wiki/Tool:Versions - legacy php webservice reading data from noc
• https://wikitech.wikimedia.org/wiki/Tool:Gitlab-account-approval - bot using data from gerrit, gitlab, ldap, and phabricator
• https://meta.wikimedia.org/wiki/IRC/Bots/ircservserv - irc bot
• https://wikitech.wikimedia.org/wiki/Tool:Jouncebot - legacy python irc bot
• https://wikitech.wikimedia.org/wiki/Tool:Phab-ban - legacy python webservice using OAuth to talk to Phabricator
• https://wikitech.wikimedia.org/wiki/Tool:Schedule-deployment - webservice
• https://mwdemo.toolforge.org/wiki/Main_Page - legacy webservice, MediaWiki using ToolsDB, continuous job for jobrunner, periodic job to update NFS data

In T407403#11279030, @bd808 wrote:

When the offending code is tracked down, https://www.mediawiki.org/wiki/Manual:Coding_conventions/PHP#Don't_use_built_in_serialization and T161647: RFC: Deprecate using php serialization inside MediaWiki should be justification for rewriting the blob serialization for the cached data.

Changing the 3v4l.org test @hashar did in T407403#11279288 to one that includes legacy versions of PHP so that 8.1 output shows up reproduces the serialization diff between 8.1 and 8.3.

We have PHP 8.4 jobs in CI already, so I think that could be the next dev target without too much hassle. @Krinkle may have an idea if prod is likely to update to 8.4 or if we will be more likely to skip to 8.5 (RC2 today, likely to release in November 2025).

When the offending code is tracked down, https://www.mediawiki.org/wiki/Manual:Coding_conventions/PHP#Don't_use_built_in_serialization and T161647: RFC: Deprecate using php serialization inside MediaWiki should be justification for rewriting the blob serialization for the cached data.

Implementing the data entry and storage as text on https://wikitech.wikimedia.org/wiki/Tool:Phab-ban/Log is relatively straight forward. As an actual audit tool however it feels pretty weak. The folks complaining on JJMC89's talk page would still be complaining if there was a summary line that said "Faster_than_Thunder was disabled by JJMC89 for doing whatever". Blocks are trivially set and trivially reversed. This is by design.

From https://wikitech.wikimedia.org/wiki/SRE/LDAP/Renaming_users :

We do not rename developer accounts. If you are very unhappy with the name of your account, you can create a new one with the name you prefer, request that it be granted equivalent permissions to your existing account, and request that your existing account be deactivated.

We used to rename accounts, but it frequently led to various errors throughout the many separate systems which consume developer accounts, due to their local databases and authentication methods getting out of sync. In the long term, this may change if we develop better tooling for identity management.

https://wikitech.wikimedia.org/wiki/Help:Toolforge/Redis and https://wikitech.wikimedia.org/wiki/Help:Toolforge/Elasticsearch are more shared services.

https://wikitech.wikimedia.org/wiki/Portal:Toolforge/About_Toolforge says that multi-maintainer tool accounts with access to shared storage, $TOOLNAME.toolforge.org hosting, cronjobs, daemons, per-tool databases, and access to Dumps and Wiki Replicas are part of Toolforge.

With https://gitlab.wikimedia.org/ladsgroup/Phabricator-maintenance-bot/-/merge_requests/31 merged and deployed, @Maintenance_bot will now check for tasks tagged as Beta-Cluster-Infrastructure that were authored by @wmcs-alerts and having titles containing "deployment-cache". It will add Traffic when matching tasks are found. The job that does this work currently runs once per hour, so there will be some lag before changes are applied. If this lag starts to seem problematic we can probably work with Amir to tune the bot so that it runs more frequently.

Fixed by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1194156. Thanks @Muehlenhoff!

Since T364249: New upstream release for Pygments (2.18.0) a Python 3.8+ runtime is required.

In T406650#11252612, @bd808 wrote:

In T406650#11252502, @Aklapper wrote:

I don't know the skillset of @Maintenance_bot which could be another option, in theory.

Yeah, that might be the right place to look if trying to go the bot route. It looks like the project_grouper.py task there would need a bit of feature enhancement to be able to add the "title contains" and "author is" criteria. Hacking python bots is in my wheelhouse though. :)

In T406650#11252502, @Aklapper wrote:

@bd808: I do not know how/where alertmanager/@wmcs-alerts currently sets the Beta-Cluster-Infrastructure tag and task title for the Phab task to be created. Maybe the code on that side could be expanded to pass additional tags to Phab based on substrings in the task title to be created? If not feasible, then Herald makes most sense.

@Aklapper what do you think about using Herald for this sort of thing? It would need to be a global rule something like:

https://cheatsheet.krishnaneupane.com/posts/haproxy has been helpful as more blocking has moved to haproxy. In our deployment the socket for talking to the running instance is at /var/run/haproxy/haproxy.sock.

In T403105#11251706, @bd808 wrote:

Hack achieved. :/

local isp_dbpath = "/usr/share/GeoIP/GeoIP2-ISP.mmdb"
local datacenter_dbpath = "/usr/share/GeoIP/datacenter.mmdb"

$ ls -alh /usr/share/GeoIP/
total 62M
drwxr-xr-x   2 root root 4.0K Aug 26 23:38 ./
drwxr-xr-x 116 root root 4.0K Aug 26 22:49 ../
lrwxrwxrwx   1 root root   18 Jun 22  2023 GeoIP2-City.mmdb -> GeoLite2-City.mmdb
lrwxrwxrwx   1 root root   21 Jun 22  2023 GeoIP2-Country.mmdb -> GeoLite2-Country.mmdb
lrwxrwxrwx   1 root root   36 Aug 26 23:38 GeoIP2-ISP.mmdb -> /usr/share/GeoIP/GeoIP2-Country.mmdb
lrwxrwxrwx   1 root root   15 Jun 22  2023 GeoIPCity.dat -> GeoLiteCity.dat
lrwxrwxrwx   1 root root   18 Jun 22  2023 GeoIP.dat -> GeoLiteCountry.dat
-rw-r--r--   1 root root    0 Jun 22  2023 .geoipupdate.lock
-rw-r--r--   1 root root  37M Jun 22  2023 GeoLite2-City.mmdb
-rw-r--r--   1 root root 2.0M Jun 22  2023 GeoLite2-Country.mmdb
-rw-r--r--   1 root root 3.9M Jun 22  2023 GeoLiteASNum.dat
-rw-r--r--   1 root root  19M Jun 22  2023 GeoLiteCity.dat
-rw-r--r--   1 root root 792K Jun 22  2023 GeoLiteCountry.dat

/usr/share/GeoIP/datacenter.mmdb appears to be the missing file this time.

Something has changed making this an active problem again. While debugging an unblocking failure (T406091) I found:

bd808@deployment-cache-text08.deployment-prep.eqiad1:~$ haproxy -f /etc/haproxy/haproxy.cfg -c
[NOTICE]   (2862995) : haproxy version is 2.8.16-1~bpo11+1
[NOTICE]   (2862995) : path to executable is /sbin/haproxy
[ALERT]    (2862995) : config : parsing [/etc/haproxy/haproxy.cfg:13] : Lua runtime error: Error opening the specified MaxMind DB file
[ALERT]    (2862995) : config : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
[ALERT]    (2862995) : config : Fatal errors found in configuration.

I wonder if @Don-vip actually tried to add you to the repo or not. I can see that he has the "role: admin" at https://github.com/toolforge/video2commons/settings/access. That should have let him add you I would have thought, but github org related permissions can be weird.

Caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1191680

bd808@deployment-puppetserver-1.deployment-prep.eqiad1:~$ sudo -i puppet agent -tv
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud
Info: Applying configuration version '(b88963634a) gitpuppet - [BETA HACK] Changes to profile::puppetserver::volatile'
Notice: /Stage[main]/Profile::Puppetserver::Volatile/File[/srv/puppet_fileserver/volatile/datacenter_vendors]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Profile::Puppetserver::Volatile/File[/srv/puppet_fileserver/volatile/datacenter_vendors]/ensure: removed (corrective)
Notice: Applied catalog in 12.96 seconds

Whatever it was, it's fixed now.

This was probably a stale cherry-pick that someone cleaned up without seeing the ticket.

In T395401#11232222, @JJMC89 wrote:

If it were up to me, the repo wouldn't be migrated to GitLab since GitLab is an inferior experience to GitHub and Gerrit. (I won't move the backend repo off GitHub.)

In T395401#11232164, @Samwilson wrote:

@brennen @bd808 Is there a preference for VPS tools to have their repos under /cloudvps-repos rather than /toolforge-repos?

See https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Pretrain/Progress_reports/2025-09-19 for the final report on this hypothesis. We are declaring the experiment a success and planning to follow up with work to put the new container to use in 2025.

In T398868#11197783, @dancy wrote:

@bd808 The work in the title of this ticket was completed a while ago. Shall we mark this resolved?

In T401533#11072047, @bd808 wrote:

Half of the time is apparently a workaround for T390251: docker-registry.wikimedia.org keeps serving bad blobs:
01:06:59 [mediawiki-publish-83] Waiting 300 seconds for swift after full mediawiki image build (T390251)