Sorry about this, this should now be fixed. And glad to see that Traffic was added automatically, thanks to @bd808 and @Ladsgroup for their work on this!

In T404826#11298704, @dancy wrote:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1197986 has caused puppet to break on deployment-cache-upload08.deployment-prep. Please help!

This is because of:

In T390813#11296819, @Papaul wrote:

@ssingh @Vgutierrez hello just checking in to see if you have a day and time for this for drmrs.
Thanks

Hi @CRoslof: This is another ticket that we would like to take up and will need your help with so that we can reflect it in downstream services as well. Let me know if I should create a Zendesk thread for tracking by other Legal members? Thanks a lot of for bearing with us and helping us clean the ownership.

It looks like spicerack should check that alerts for the downtimed host have been resolved (not in firing state) before deleting the silence/downtime with ALERTS{alertstate="firing", instance=~"cp5018:.*"}

Thanks for filing this task! I think this is a good idea to reduce the manual updates to this list, and something we have failed to keep updated. We will triage this after discussion in Traffic.

Nice job indeed in pursuing this over the years, Brett!

[Adding Raine @kamila as well.]

Thanks for filling the task, @JVanderhoop-WMF. As per the discussion on Slack, the above sounds good.

Thanks to @Jhancock.wm for the help with this!

FWIW doing one or two hosts is more than enough. We will reimage them again anyway so it doesn't make sense IMO for you both to spend time upgrading all of them to trixie. If one or two reimage fine, please leave the rest to us.

In T407320#11276943, @colewhite wrote:

Do you happen to have a trixie host available that we can try the existing package on?

I was also looped into a new request today. As part of the birthday initiative, the Fundraising team is developing a customized donation portal under the donate.wiki domain. Would it be possible to set up a redirect for this new portal as well? I don’t have the final destination URL yet, but we’d like to create the domain donate.wikipedia25.org to redirect to the donation portal once it’s ready. Is this something you could help with too?

In T405499#11275363, @cmooney wrote:

In T405499#11273763, @ssingh wrote:

FWIW we have typically reimaged for this in the past. I am not suggesting, just sharing! And given that this is lvs1020, that might be OK? (Leaving to you both for the final decision.)

This is lvs1018. I'm comfortable enough either way, can I leave the decision with traffic?

I was thinking of trying to schedule this for tomorrow, Thurs Oct 16th if that worked for you guys?

Thanks for working on this! We will try our best to follow up on our end in making sure that Puppet is not broken on the cache hosts in Beta.

FWIW we have typically reimaged for this in the past. I am not suggesting, just sharing! And given that this is lvs1020, that might be OK? (Leaving to you both for the final decision.)

Thanks @MoritzMuehlenhoff for working on this and researching it. I am closing this for the reason mentioned above.

Cross-posting the comment from T405102#11273708,

Traffic discussed this in the team meeting today. We decided that given the above blocker, we should simply move to trixie and use OpenSSL (3.5.0) as shipped by trixie. The reasons for doing so are this: it doesn't make sense to upgrade to bookworm as that ships OpenSSL 3.0 and we have concerns around the performance (we haven't evaluated that as we have with 3.5 but we believe that to be the case from what we have seen).

Traffic has a dependency on Data Engineering for this to be executed, as we will need their help to run the query and generate the data set. That being said, we will make sure that we take this on when Data Engineering is ready (including doing the leg work).

(Is there anything -- including input -- required from Traffic on this? I am asking since we were added and can triage the task accordingly.)

Rolled out.

Thanks to a review by @bking, we have merged this. I think we can consider this as resolved. Thanks @LSobanski and @Gehel.

Thanks @MoritzMuehlenhoff, that sounds like a good plan to me but leaving to @CDobbins for the final word.

@bd808 and I discussed this today and decided to split this up in two parts:

hcaptcha200[1-2].wikimedia.org are ready.

In T406166#11249385, @cmooney wrote:

@ssingh FYI I ran the sre.dns.netbox cookbook just now as it alerted on being a diff, it removed the entries for hcaptcha1001.

The VM doesn't exist in Netbox right now and the IPs assigned for it were unreachable so I figured it was safe.

In T406141#11249404, @LSobanski wrote:

Now that the other endpoints were added, is there anything else that needs to happen before the patch is deployed?

Hi @akosiaris: Following up on this after a discussion during Traffic's planning with @Vgutierrez, and on behalf of the team.

Thanks for following up @cmooney. Other than the fact that we owed you a response on your last comment and never did -- that's on me of course -- yes, I think we can close it.

In T392851#11242196, @elukey wrote:

Status update: I was able to upgrade idrac+bios of most of the cp hosts, I'll review the remaining ones on Monday and I'll give a precise list in here if there will be outstanding problems or not.

The code to make the firmware upgrade cookbook to work is still under review, but I tested it and it works :)

We will be resuming work on this in Q3 2025-2026.

In T343000#11241334, @Vgutierrez wrote:

yes, it's still hapenning https://grafana.wikimedia.org/goto/SHdP6s3HR?orgId=1:

I believe this will be fixed when we upgrade to HAProxy 3.0 given it provides persistent stats.

@Shisma: Hi, this still needs a Wikimedia affiliate approval as per https://lists.wikimedia.org/pipermail/maps-l/2020-August/001729.html. Can you provide one, and after which we will need to check the technical feasibility?

Can someone help me double-check if this is still a problem? I don't see it in the dashboards above, selecting a more recent time interval.

In T406141#11235857, @Gehel wrote:

I'm not entirely sure which alerts paged during the last WDQS outage. I think it was about the number of servers depooled from LVS and not from error rate. The currently linked gerrit patch removes pages in cases of high number of backend errors, but I don't think it removes the page in case of too many servers depooled.

@BCornwall from Traffic will be working on this, thanks!

@BCornwall from Traffic is handling this and will sync with @JAbrams and @jrbs for when the change is made live, to ensure incoming and outgoing email works as expected.

In T394789#11229562, @bking wrote:

Thanks @ssingh , that's completely understandable. If PyBal is going away, it's probably not worth the effort to fix it.

But I'm wondering if Liberica has the same issue? In other words, is it possible to create an invalid Liberica config via Puppet patches, that would be caught by the ConfdResourceFailed alerts? If so, then the task is still valid, it just needs to change scope. Let me know what you think.

In T404219#11226595, @Lupascriptix wrote:

I admittedly don't know enough of what OpenRefine is doing under the hood to get the query path, but I have the json for the reconciliation call if that helps. I can also share the python script that's erroring..

In T404219#11226332, @Lupascriptix wrote:

Hey there, sorry about the delay, I was away from the office for a bit. The folks at T402959 don't think this issue is due to the SPARQL queries, but I just checked and I'm still having the same issues as I was a couple weeks ago.

In T405165#11226025, @Ciencia_Al_Poder wrote:

El T405165#11225510, @ssingh escribió:

I don't think this has anything to do with the UA. Is the command correct? Maybe try downloading via curl/wget and then piping to gpg --import '. curl https://www.mediawiki.org/keys/keys.txt | gpg --import - works for me for example.

I used tcpdump to inspect the traffic (using http instead of https URL) on both Debian and OpenSuSE, and no User-Agent was sent on Debian (see description).
I later confirmed the user-agent issue using curl to request the same URL, with the default user agent and also explicitly removing the user agent, with consistent results (failure when no user-agent was present)

How easy/difficult is to whitelist a specific path or URL? It was working before your (WMF) changes, requesting a particular static txt file shouldn't cause any significant amount of pressure on the servers, and retrieving it from gpg directly is very common. Debugging such a cryptic error message can be time consuming.

Note that @KOfori is out, this should be directed to @Kappakayala in the meantime.

I don't think this has anything to do with the UA. Is the command correct? Maybe try downloading via curl/wget and then piping to gpg --import '. curl https://www.mediawiki.org/keys/keys.txt | gpg --import - works for me for example.

[Removing Traffic since this is a MW issue. Please re-add if you disagree.]

I am curious: should we keep this open or should this be resolved now given that we have requestctl.wikimedia.org and that takes care of most of the pain points?

This was merged upstream in 9.2.x so we have inherited this change. Since we have not revisited this since 2021 but can pursue if required, I am taking the liberty to mark this as resolved.

We have had this for a while and the responses are padded. Marking as resolved.

In T368694#11225139, @BCornwall wrote:

The goal is to have ncmonitor run much more frequently than it currently is (e.g. once a day rather than once a week) so that we can accelerate the turnaround time for when new domains are registered. If ncmonitor were to run once a day right now then we'd end up with daily CRs of the same patchset. By bailing out we give the team some grace to review rather than have to deal with a small pile of the same patch.

What is the update on this, given that it has been a while and I am a bit confused reading the text and trying to follow the CRs.

And to be clear, by that I mean that this change is better suited for MW and not the CDN.

This is being moved on the Traffic workboard to "Radar/Not for service" as I don't think there is anything on our end to do here. Please let me know if you disagree and adjust accordingly.

My two cents: I think like you mention above, is this really required? I think this may be more work and doesn't give us a tangible benefit? Put differently I guess, what is the concern with multiple CRs?

Awaiting a response from @Lupascriptix before we can triage it again.

Paging has been disabled on this alert since July 2025. But even other than that, looking at the 90-day view, this has not happened in a while.

OK thank you. I am marking this as resolved for now. We can re-open as required.

Commenting from Traffic's side: this is in some ways, a trivial patch for us because we are simply setting an additional header. The challenge here, though, is understanding the header itself and the associated ramifications of setting it and also keeping it updated. For that, the Security should be/needs to be consulted, so this patch currently blocks on that happening.

The new cp hosts in codfw, Intel(R) Xeon(R) 6730P, support QAT, so we can experiment there and see if that helps.

This is worth investigating/researching but I am marking this as "Low" and we can revisit it for Q3 2025-2026.

We have made progress in T301605, and specific to this task, we ramped up traffic to magru already.

There has been no follow-up on this after Jun 2024. @MatthewVernon: should we keep this open?

Because of the regressions observed in OpenSSL 3.x, we never finished the upgrade for the cp hosts to bookworm. This task is stalled while we go through the possible upgrade paths for either OpenSSL, or an upgrade to trixie itself, depending on the former.

This has been open for a while and there hasn't been any follow up from either side. @MGChecker: Please re-open if this issue still persists for you. Thanks!

@Fabfur: Is there any follow-up to do here or can we close this given the issue at hand was resolved? Note that in the recent switchover, we didn't observe this issue.

OK great, resolving this for now but yeah, let's add this to the docs and see how it serves us next time. Thanks and sorry for the delay. Adding Traffic to the tags as Nat has done now should prevent this from taking a week next time! :)

In T404974#11210662, @JKelsoteel-WMF wrote:

Understood, thank you! Going forward, as part of the process for handling these requests, once the verification is completed for the domain and ITS can see the domain property in GSC, should we (ITS) go ahead and provide the TXT record in the phab ticket so that oktaservice@ can get domain ownership as well in GSC? I'm writing process docs for ITS on this type of request.

In T404974#11210631, @JKelsoteel-WMF wrote:

Yes, I am using the ITS service account designated to grant others access to our various GSC properties. The account is oktaservice@, which Nat gave account owner permissions a few weeks ago.

In T404974#11210617, @JKelsoteel-WMF wrote:

No problem. Here it is: google-site-verification=m7jEgoI4DOUy0u6cebxtp7oJT7s3nnNyPWgmPQmNEjc

In T404974#11210610, @JKelsoteel-WMF wrote:

Hi @ssingh , I just tested this with our service account (I am part of ITS), and I am still seeing the window prompting me to verify domain ownership for wikimediafoundation.org (copy the TXT record, upload it to the DNS config for the domain). When I click Verify, it gives me an error still, saying "ownership verification failed." I assume this is because it may take a few hours for the changes you made to propagate, but I just wanted to check if this is expected behavior? Thank you again!