(Update: the Spark query is still dying...)

@AndrewTavis_WMDE I got at least part of the Spark query working--the part without the regexes.

Thanks @AndrewTavis_WMDE!

Hey thanks so much @dcausse and many apologies for the delay in replying!

Merge request for the notebook for the main missing-sitelinks metric: https://gitlab.wikimedia.org/repos/wmde/analytics/-/merge_requests/3

Gitlab merge request: https://gitlab.wikimedia.org/repos/wmde/analytics/-/merge_requests/2

Gitlab merge request: https://gitlab.wikimedia.org/repos/wmde/analytics/-/merge_requests/1

Thanks again for all the help!! :)

Thanks, @Volans. I can confirm that I have access to JupyterHub again, yaaaayyyy ;)

I received the the NDA at my WMDE address and signed it.

Yaayy thanks so much @Volans, @Dzahn, @kamila, @WMDECyn!

In T367681#9923213, @Dzahn wrote:

Hi Andy, this ticket is currently stalled and waiting for your input to continue before we can merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047473

In T367681#9904412, @kamila wrote:

@AndyRussG Can you please confirm the new end date? I assume you meant July 30, 2024, correct?

In T361742#9706455, @Milimetric wrote:

Approved, welcome back Andy :)

Thanks so much, @MoritzMuehlenhoff , @RLazarus! :) :)

In T361665#9685550, @RLazarus wrote:

@AndyRussG Welcome back!

Thanks @Quiddity! Certainly that's fine! Though it might also be clearer to mention that the settings to disable certain banner types are only available to logged-in users? What about something like this:

Initial, tentative work on this: this fr-dev patch and this related dev-images merge request (which I should break into multiple merge requests, as suggested by @jgleeson, so the unrelated Civi bullseye upgrade can be considered separately).

Recent example: See failmail with the subject, "Fail Mail (civi1001) run-job: Donations queue consume failed with code 135", from 2023-03-23. Referenced log file is /var/log/process-control/donations_queue_consume/donations_queue_consume-20230323-094701.log.

Hi! The linked DonationInterface patch has been deployed, and the form does ask for UPI ID when we specify recurring=0 and payment_submethod=upi on the URL.

A bit more discussion here (internal WMF Slack).

Notes for current sprint: https://etherpad.wikimedia.org/p/fr-tech_chores_2023_fish_HEAD%5E

In T308614#8718372, @Ciell wrote:

@Damilare , @XenoRyet I noticed you lowered priority and removed the Chaos Crew. The bug is still very relevant though - I am at this moment in a call where it is mentioned the CN banner for Wikimania had reviewed translations that did not go live.

Why was the tag removed and the priority lowered?

Just to confirm, the issue with user preferences not correctly hiding banners based on type: this part of the system now seems to be working as expected. Is anyone still experiencing this specific bug?

Thanks so much for this @jgleeson! The attached patch looks great.

In T322612#8703441, @Jdlrobson wrote:

Can this ticket be closed @AndyRussG given I think you took another direction?

In T331671#8699130, @DerHexer wrote:

@Pcoombe Yes, I sadly know. And in my opinion, I consider this a bug and not a feature. Many of us are active on more than one project and get annoyed by the same banners multiple times.

In T331671#8690787, @DerHexer wrote:

Thanks @AndyRussG for fixing the preference bug!

Heyyy here are some quick notes:

• I was able to reproduce this on production just by disabling banner categories in my preferences and seeing what happened. Not sure why you're not seeing any banners, @TheresNoTime.
• The fix turned out to be quite simple. It has been merged and placed on this week's deploy train. It should be deployed to most Wikipedias on Thursday, and to most other projects and a few Wikipedias on Wednesday.

Apologies for the delay in getting to this, and for not writing more details just now. (FR-Tech is rushing to get a new payments integration finished this week.) I'll try to find time later this week to explain more and reply to some comments above.
Thanks again!!

Thanks so much for this, it's been immensely helpful!! As noted here T331671#8688951, it looks like there is indeed be a bug. I'll post more on the parent task as soon as I can.

Thanks so much everyone for your input and effort on this, and many apologies for the delay.

Attached patches add the required fields to the donations queue message. Test and UI elements will be included in follow-on patches.

Thanks so much @Krinkle, @kostajh !!

Summary

• The updated code for this is merged and deployed, but I'm unable to turn on the process-control job myself due to a permissions issue. (Can't push to fran1001:/var/lib/git/localsettings.git/)
• Comparing the output of the old and new versions of the script, via spot-checking, we see the following:
  • Data in banner impressions seems identical.
  • Data in landingpageimpression_raw seems almost identical.
  • Data in donatewiki_unique seems to get additional entries with the new script. These entries are not present in data generated previously by the old script in the production pgehres database. However, this was probably due to an issue with the old script, or something related to the setup at that time, causing the old script to incorrectly drop data.

Instructions for running under fundraising-dev with bullseye: