Ran relforge reports for adjusting the near_match_weight on commonswiki, along with mediawikiwiki to see if this has different effects in different places. I can't share the full reports as they contain user search queries, but the top level stats are shareable. I'm only including commonswiki here as it was the only interesting one, one mediawiki.org increasing the near_match weights had almost no effect, suggesting this fix is specific to how commonswiki is organized.

In T407521#11290416, @fkaelin wrote:

Related question regarding flow of data, based on the comment from the thread you linked.

The wikitext -> html happens inside the mediawiki application using the default mediawiki parser. I'm not sure what exactly happens under the hood, i expect it's a full php parser that runs in-process but i haven't paid enough attention to exactly what they do. This is indeed quite expensive, we are running hundreds of pages a second through the parser. Part of the reason i suggest we could do this is because we already parse this flow of data. Even at this high rate, it still takes a long time to get through everything. We have a loop that re-renders everything even if not edited, but it works on 16 week cycles.

A html dataset (T360794) is a request for data engineering with a number of use cases, and has been discussed in related phab tasks for years. The linked phab task is for an incremental html dataset, which is "the easier" part of a html dataset and will hopefully get prioritized soon. I have focused on that part to get something off the ground. The more challenging part is creating the html dataset of historical revisions (e.g. render with what mediawiki version, what to do with templates, etc..).

• do I understand this right that the full re-render loop taking 16 weeks is for the "current" content of all pages (i.e. not historical)? That is indeed a long time.

MR: https://gitlab.wikimedia.org/repos/maven/wmf-jvm-parent-pom/-/merge_requests/27

related slack discussion: https://wikimedia.slack.com/archives/C0975D4NLQY/p1759903593949559

Updated .deb is available from gitlab. This should be ready for hand-off to SRE to upload the deb to apt.wikimedia.org and restart the clusters. Once the .deb is available from apt.wikimedia.org we will also need to:

built a new release and deployed to maven central, as 1.3.20-wmf4. For example: https://central.sonatype.com/artifact/org.wikimedia.search.highlighter/cirrus-highlighter-core

An initial, simple, proposal would be to split the text field on section boundaries, and retain the section title as a header. This would mean having duplicates of the headings (in both the headings and text fields), increasing the importance of the heading content, but probably not a big deal.

it might be convenient if we had some tool that could walk the category graph on wiki, then query the same thing out of blazegraph and compare them. Some quick way to identify where the issues might be.

From our review of the initial reports there is also a bit of surprise around the opening_text language model performing worse than the default language model. One plausible explanation is that there are word patterns seen in queries but not the opening text, only in the title fields. As such it would be interesting to run a follow-up test comparing title+redirect.title vs title+redirect.title+opening_text. For that I've created T407432.

We reviewed the report in the wed meeting, where a report against major spaceless languages was requested. A quick runthrough of a report restricted to zhwiki and jawiki finds that the default_1v profile is significantly better than the others, suggesting that the variant does potentially have benefits, but it may depend on which language. As such I've run a batch of reports against the top few wikis by size, and a few selected languages that have unique language features:

I mistakenly posted this to the parent ticket, but it belongs here:

Preliminary reports. They might become final, but they haven't been reviewed by anyone else yet:

We experiemented with this, and a model is available in production (example query, but the results just aren't good enough. Calling this complete without implementing it into mjolnir.

Copying comment from merged task:

Patch to configure and start the tests was prepped in the earlier patch, shipped the test today. Can turn it off Oct 13.

Something like this should be reasonable, along with the link to the documentation

I suspect the underlying technology is now sufficient to support alphabetical sorts (although we would have to evaluate it to be sure). The main sticking point in Cirrus today is going to be that the way keyword fields work in cirrus today allow doc_values to be enabled. We would need to migrate all the existing psuedo-keyword mappings to use normalizers, which then allows us to enable doc_values on appropriate keyword fields. Once the index mapping is in place the new sort is only a few lines of configuration in Cirrus.

Rough outline of a plan, I expect this will first be worked up in a notebook and evaluated. We should be able to upload the models direct from the notebook to see them operate in prod. I'm not sure if we have a way to call out models by name in a debug manner, we might have to define a prod rescore profile that can access the model variant.

yea lets create a separate ticket as it will likely involve a few days work.

Similarly, should we filter searches in main article namespace only? (though I assume that there are very few queries that are not in main namespace).

an additional difficulty with using mediasearch directly from commons is that file search is against both the local wiki and commons. It would be a change in functionality for it to start only displaying results from commons.

The T403593 subtask is now deployed to production and ready for use. The feature is documented in Help:CirrusSearch.

Tested the keywords in prod, looks to be working as expected. Updated Help:CirrusSearch on mw.org with the proposed documentation from above.

In T404822#11213120, @MGerlach wrote:

@EBernhardson Thanks for putting together the notebook. Looks really good, I appreciate the level of detail with respect to manual verification and having confidence intervals.

• from what I understand, you operationalize natural language queries as all queries which contain one of the words who|what|where|when|why|how (and later do some additional manual filtering). Could you confirm? I think that approach makes sense and is sufficient to get a rough idea of the order of mangitude.

In T401590#11213299, @Cparle wrote:

I guess the open question is, should an exact title match somehow go around deboosts?

Intuitively - yes, I think so. Is there any way to confirm that with data?

If a request is a web request and contains no cookies and contains an offset -> Automated

In T405360#11206505, @amastilovic wrote:

How big are the individual files we need to move for this?

Not having the final confidence intervals was unsatisfying, so i went through and worked it up properly with references for how this is supposed to work within a stratified sample. Notebook has been updated to contain the calculation (please review! I am not an expert here).

In T405360#11206183, @amastilovic wrote:

Blunderbuss could easily do this for you, with minimal resource usage on the Airflow executor side :-)

Initial estimate for the week of Sept 8 - 15.

What do we think is the right way forward here? If SRE will be prioritizing implementing a newer method of getting data from hdfs to the public sites in the next month or so then it seems like this could wait around, but if it's uncertain when we will be prioritizing this work it seems reasonable to move forward with the existing puppet bits that invoke hdfs_tools::hdfs_rsync_job

Using the pageid filter we can get an explain that contains only the top three results and the target category:

After further consideration, I remembered that query_clicks_hourly still does not contain mobile web requests, but those will need to be included here. To include mobile web we will need to start the analysis from web requests. This is more tedious as the dataset is quite large, but likely necessary. Will have to see if we can analyze a full week, due to data sizes we may have to break analysis up into per-day numbers and aggregate those daily numbers.

I poked around the data a bit and experimented with a few things, i suspect we can do something like:

Proposed Documentation, under the Filters heading:

Seems like we have a decision. T404647 created to run the test.

I can fit hours into here if it's needed, but I do wonder if it will feel a bit awkward with consistently rounding time. What i mean is >2024 and <2024 round their comparisons to the nearest year, similarly for months or days. This feels natural (to me, at least) when working with those units. With hours we would have lasteditdate:>now-2h, do we also round that to hours? It feels more natural to me for such short timespans to be rounded to minutes, but that would lack consistency and make the system harder to explain. Not sure what the right approach is, but switching between them isnt too hard.

In T403593#11175408, @Michael wrote:

• Accepted format: (<|<=|>|>=)?(YYYY(-MM(-DD)?)?|now(-\d+[ymd])?)

We, Growth, can make that work, but I wonder if we could add h for hours too? So that we can say <now-24h (or maybe <now-36h if we want to have more slack). Though, if that is too complicated, we can also use <=now-2d for our offset to be sure that we exclude pages edited within the last 24 hours.

In T403212#11175588, @A_smart_kitten wrote:

@EBernhardson Thanks for adding these characters to the docs!
Re the \uNNNN matching, do you think it's worth clarifying the term "surrogate pairs" in those docs? I ask as it took me a few minutes to work out that — to search for the equivalent of intitle:/💔/ — I needed to use intitle:/\uD83D\uDC94/ (instead of e.g. just using intitle:/\u1F494/). Or do you think that people using this escape character would generally be expected to know what this would be referring to? (genuine question)

Looking over the date field docs and testing a few things, it looks like we can fairly easily support the syntax requested above. To me the biggest questions are around localization. As stated in the ticket there is the question of localtime vs UTC. There is also the question of date formats, is "05/04/25" in april, or may? Do we perhaps only accept YYYY, YYYY-MM, and YYYY-MM-DD?

Per discussion at wed meeting i added a couple more profiles and renamed the existing profiles to be more consistent. The names should now consistently be of the format: {profile_name}_{prefix_len}(_variant)?. The numbers are mostly but not directly comparable to above, all the queries were re-run which gave new latency numbers and different results for some queries. All of the results were re-graded into one of the 5 buckets.

To get an idea of what an appropriate weight would be i ran some stats against an hour of incoming requests. Note that the first search result is considered position 1. Also note that this is not re-running the queries, it is applying custom weights to the scores and re-sorting the results that were provided. The true mean will likely be larger than presented here as galleries are pushed down and new results come into the result list.

Trying to put some sort of judgement on the list i came up with the following categories:

Can maybe link https://www.mediawiki.org/wiki/Help:CirrusSearch#Character_Classes which documents most of the new functionality (the rest is also documented on that page, but in a different section).

In T317599#11153646, @Quiddity wrote:

Thanks! I'll postpone including this until the following edition, partially in case T403212: Support \r, \n, \t, and \uNNNN in insource and intitle queries is completed by then and can be announced in the same entry (IIUC, that would be the clearest way to do so?). Plus, we'd need to update the proposed draft-entry to include both that other task and also link to any new documentation(?) about these new features. The current proposed draft-entry is still this:

• regex search queries now support additional features including start-of-line (^) and end-of-line ($) anchors for the intitle keyword, as well as shorthand character classes for digits (\d), whitespace (\s), and word characters (\w) in both insource and intitle.

reindex has completed on all clusters

Bit of a first draft, this defines a few new profiles:

In T403212#11133330, @A_smart_kitten wrote:

(i'm guessing this isn't ready to announce yet given that the patch isn't currently merged?)