Requests for comment/Mobile domain sunsetting/status
2025-07-01
[edit]This project was expedited and started in May, ahead of the planned start for Q1 in July, because the Google SEO risk we identified back in November 2024, has now manifested in a very real sense on Wikimedia Commons. Tim Starling found that of ~140M content pages on Commons, only ~70M are "known" to Google, and over 30% of those are explicitly de-indexed from Google Search results, and this is declining by about 1 million pages a month (T54647#10887076, T214998#10911867).
Progress in May and June:
- Prep #1 (Fix improper detection in our own code): Done. Specifically, T390923: Update various MediaWiki extensions from hardcoded m-dot checks to stable mw.config.
- Prep #2 (Prep client events): In progress. Discussions are on-going about what we need and how other existing or new fields in the event object may meet those needs.
- Prep #3 (Prep purge hook): Not yet started.
- Prep #4 (Prep analytics pipeline): In progress. Specifically:
- (Done) T305794: Expose X-Analytics header in WikimediaDebug. Our plan involves adding more fields to the X-Analytics response header, and other teams then need to build on top of this. Difficulty to debug this, was first raised back in 2022. We're pulling this in now, to ease T390924 and T389696 down the line.
- (Done) T390924: Compute ismobile field at the edge and add to X-Analytics header. Our pageview counts and unique device counts datasets currently differentiate "
mobile web" from "desktop web" by checking the domain name. To prepare for the transition away from the m-dot domain, these pipelines need a new field to check for instead. This is that new field. Theismobile=1attribute represents whether a request came from an eligible mobile user agent (unless opt-ed out), or otherwise accessed a page via its m-dot URL (for now).
- Prep #5 (Fix HTTP variant compliance): In progress.
- (Done) T397387: Decouple the MobileFrontend header from the subdomain feature. This is a prequisite to being able to enable MobileFrontend on pageviews from mobile user agents directly, rather than only via a redirect to a dedicated subdomain.
- (Done) T390929: Install "Vary" header on index.php and api.php responses where MobileFrontend is active. Our articles have two possible responses today under the same canonical URL: desktop HTML or redirect to mobile. While technically in volation of the HTTP/1.1 spec, this works because we do the redirect at the edge before the cache lookup. The mobile HTML and desktop HTML are served today at different URLs (m-dot vs canonical) and so technically have no variants. In the future, these two responses will share the same URL and get generated behind the edge in MediaWiki, which, to avoid cache poisoning, requires the backend to declare that its responses vary by whether the MobileFrontend header was set. This is now done for index.php pageviews and API responses.
2025-07-18
[edit]Prep #5 (Fix HTTP variant compliance) continues to be in progress at T390929: Install "Vary" header on load.php responses where MobileFrontend is active. At glance, ResourceLoader in MediaWiki core does not behave differently for desktop or mobile. There is however an override inside MobileFrontend itself that may require us to set up variants for this entrypoint as well. I'm looking at suitable approaches, and cleaning up tech debt in MobileFrontend and CentralNotice by fixing related bugs and improving docs to gain confidence in the implementation.
Tim has deployed an edge routing patch that suppresses the mobile redirect specifically for Mobile-Googlebot on Wikimedia Commons (T397267). This is a fairly low risk change, considering how severe the issue already is on Commons. Shortly after this, Googlebot experienced throttling errors while performing first-time crawls for a lot of newly discovered Commons content for Google Search (T398668). These errors are important because the same Googlebot is also crawling Wikipedia and other sister projects. The accepted crawl rate has been increased for the time being.
2025-07-25
[edit]Prep #5 (Fix HTTP variant compliance) is done with the last patch at T390929: Install "Vary" header on load.php responses where MobileFrontend is active. The "Vary" header is needed here, because MobileFrontend disables the standard Common.css and Common.js pages in favor of Mobile.css and Mobile.js, thus varying ResourceLoader responses between canonical and m-dot. Removing this mechanism would benefit page load performance for readers, and reduce tech complexity for the community, and reduce WMF maintenance; but MobileFrontend lacks a gradual off-ramp (T390929#10997518), so we consider this out of scope for this initiative and should revisit it later as its own project.
As part of the Wikimedia Commons SEO mitigation (T400022), Tim Starling finished development of the Sitemaps API (T396684, reviewed and merged by Timo Tijhof). This is now ready to be deployed (T400023). We met with Google Search folks on Thursday (meeting notes), where we learned that T396168 (Google Search not indexing Commons videos in "Videos" search) is likely a bug in Google instead of an issue with our markup, and Google is now looking into it.
2025-08-01
[edit]It was a slow week while we wait for the two unresolved dependencies, Prep #4 (Data pipelines) and Prep #2 (Reading Web instrument, T400852).
Tim enabled the new Sitemaps API on commons.wikimedia.org and all other public wikis (T400023). Note that sitemaps are not yet advertised in robots.txt or otherwise discoverable. This was a silent launch of the new REST API, to allow for monitoring and final end-to-end testing of its performance in our production environment, before we submit it to Google next week.
2025-08-08
[edit]It was another slow week while we wait for unresolved dependencies Prep #4 (Data pipelines) and Prep #2 (Reading Web instrument, T400852).
Tim submitted the new Sitemaps for commons.wikimedia.org to Google, after which Google Search discovered 160 million new pages for its index (T400023).
In reviewing our MobileFrontend configuration, Varnish configuration, and DNS zones, I noticed that two wikis advertise m-dot URLs that don't exist and never worked (https://thankyou.wikipedia.org, and https://nostalgia.wikipedia.org). I'll be turning off MobileFrontend there to remove these dead-end pointers and avoid confusion (T400855).
2025-08-14
[edit]We wait for dependencies Prep #2 (Reading Web instrument, T400852), and Prep #4 (Data pipelines, work started at T401665 and T401666).
We disabled MobileFrontend on thankyouwiki and nostalgiawiki to fix dead-end links and reconcile the DNS/Varnish config with MediaWiki (T400855).
SRE Traffic has reviewed our Varnish implementation plan for the new way of routing mobile page views. We expect to rollout to wikitech.wikimedia.org as first pilot wiki next week (T401595).
Last week, Google Search discovered 160 million new pages through the sitemaps on commons.wikimedia.org. This week, the index page count went up from 58 million to 118 million, and continues to rise. The search result click rate for Wikimedia Commons, as reported by Google Search Console, also continues to rise (T400023).
2025-08-22
[edit]We wait for dependencies Prep #2 (Reading Web instrument, T400852), and Prep #4 (Data pipelines, work underway at T401665 and T401666).
Last week Brett Cornwall (SRE Traffic) and Timo Tijhof (MwEng) reviewed and prepared the Varnish logic for pilot rollout, and improved unit tests for CDN routing logic. We found a pre-existing bug in the ATS-backend logic of the CDN that would have caused issues. Next week we expect to roll out to testwikis in Beta Cluster, and Wikitech in production (T401595).
2025-08-29
[edit]Unified mobile routing is now live in the Beta Cluster at https://test.wikipedia.beta.wmcloud.org/. Details in Varnish patch and MediaWiki config. We found and fixed a pre-existing bug in the way MobileFrontend toggles between modes.
Next week we will rollout to wikitech.wikimedia.org and office.wikimedia.org in production (T401595).
2025-09-05
[edit]Unified mobile routing was deployed to production and is now live on Wikitech, Office Wiki, and mediawiki.org.
The unified routing regex was replaced by an automatic codegen in Varnish, managed from a YAML file, to ease future rollouts. The rollout is announced in the next issue of Tech News, which will be published on Monday. A broader announcement will be sent to Wikitech-l, also on Monday.
Following the submission of the newly developed sitemap for Wikimedia Commons to Google Search, we continue to see sharp rises in impressions (+68%), clicks (+75%), index size (+141%), and crawl statistics. Details at T400022.
The pageview pipeline update was completed by Joseph in Data Engineering and is now ready for the rollout (T401665).
2025-09-12
[edit]We enabled the new unified mobile routing on Wikitech two weeks ago. This resolved a 7-year old bug that survived various twists and turns, which you can learn about in the write up at T190384#11153446.
The unique_devices pipeline update was completed by Joseph in Data Engineering (T401666), which resolved the last blocker for the remaining rollout beyond pilot wikis.
Timo did a proactive audit of gadgets across all wikis, and fewer than 10 are expected to need updating. Of the ~100 site scripts mentioning ".m.", only 3 appear to use it for the purpose of detecting MobileFrontend or mobile browsers (1: fixed the popular XFDcloser gadget, 2: fixed the wikibugs gadget enabled on Russian Wikipedia, 3: notified maintainer of a Dutch Wikipedia gadget). The others were either duplicates of these, or already fine and not related to MobileFrontend (e.g. mentions of mobile URLs, or logic to reconstruct the current URL, or setting cross-origin parameters).
The rollout batch for this week (T403510: cawiki, hewiki, itwiki, metawiki) was pushed to next week to make space for additional testing.
2025-09-19
[edit]We rolled out unified mobile routing to 4 major wikis this week: Catalan Wikipedia, Hebrew Wikipedia, Italian Wikipedia, Persian Wikipedia, and Meta-Wiki.
We transitioned the Googlebot/Commons experiment (started back in June) to use the new unified mobile routing as well. Google's mobile crawler is now indexing the mobile version of Wikimedia Commons via the standard domain (T397267). This ends the brief two-month period during which Google's mobile crawler temporarily had to index the desktop version of Commons instead of the mobile version.
Timo continues proactive gadget and user script analysis, outreach, and clean up. Of the thousands of gadgets and user scripts that refer to a mobile domain or related pattern, nearly all are fine and require no updates because they use it redundantly or correctly in logic specific to parsing URLs (not as a proxy to detect MobileFrontend-rendered content). Besides the three gadgets updated last week, we found and fixed 1 additional gadget (Convenient Discussions, pull request). We documented commonly found OK patterns on a wiki page for easy reference (§ Example JavaScript).
2025-09-26
[edit]Unified mobile routing is now live on 311 wikis (29% of 1060 wikis), which means 6.9% of incoming mobile page views are receiving the new unified experience instead of a redirect (Full stats report). This is up from last week when we were on 10 wikis (1%) amounting to 6.3% of incoming mobile visits. This week we enabled Wikinews, Wikibooks, Wikiquote, Wikivoyage, and Wikiversity. That's 301 wikis (+28% of 1060 wikis) in one week. This represented only a 0.6% increase in traffic because these 301 wikis combined receive about one million mobile page views a day, whereas the Italian Wikipedia alone receives ten million daily.
Peter Hedenskog (Test Platform) shared results showing a ~200ms reduction (28%) in time to fetch an article after a Google search on a mobile device. This is in line what what we expected.
The rollout to he.wikipedia.org last week yielded a bug report about MobileFrontend misclassifying Samsung devices to mobile mode, even when installed on a desktop system or after a user requests "desktop" mode (T405279). This has been broken for years, but the increased attention has brought it to light and we fixed it. A similar issue was reported about Samsung smart TVs some years ago, but was misunderstood as specific to that device. The desktop identifier was already there but not used by us until now. The smart TV workaround has been removed.
Timo completed an in-depth audit of local site and user customisations on the top 10 wikinews.org wikis and the top 10 wikibooks.org wikis. No regressions found in comparing the m-dot and unified mobile versions of the Main Page, a random article, logged-out, and logged-in. We did find a pre-existing JavaScript error on ru.wikinews.org from a default-enabled gadget, which we investigated and fixed (T403510#11207686).
2025-10-03
[edit]
Unified mobile routing is now live on 67% of wikis (711/1060), up from 29% last week (311/1060). Globally this amounts to 40.1% of incoming mobile page views receiving the unified experience instead of a redirect. We see a consistent 20-25% performance improvement in response times on mobile devices (Stats report).
This week included rollout to Wikispecies, Incubator, wikimedia.org chapter wikis, Wikisource, Wiktionary, Wikidata, Commons, and six large Wikipedias (id, fr, de, es, ru, and ja). The remaining wikis are scheduled for next week, which comprise small-medium Wikipedias and English Wikipedia (Timeline). This last rollout is announced in the next issue of Tech News published on Monday.
In auditing *.wikimedia.org wikis we found 23 wikis with missing DNS entries and inert or defective MobileFrontend installations. These were disabled to avoid surprises during the rollout. This resolves issues with broken links and non-existent domains reported since 2016 (T152882). The audit of local site customisations covered the top 10 Wikiquote wikis this week. No regressions found.
2025-10-17
[edit]
Unified mobile routing is enabled on 100% of the 1060 wikis.
At the beginning of the project, 0% of mobile page views on canonical domains were served directly (100% was redirected). As of 9 Oct, we have served every day at or above 99.5% of incoming mobile page views on canonical domains directly, instead of via a redirect. This confirms our hypothesis that we would decrease redirects for mobile visits on canonical domains from 100% to 0%. The remaining 0.5% are redirects from features like Special:Random and Special:Search, unrelated to the mobile domain. (Final stats report)
We rolled out to the last 349 Wikipedias, including English Wikipedia, on Tuesday 7 and Wednesday 8 October (T403510).
We see a consistent 20% reduction in response times worldwide for mobile page views (21% at p50, 19% at p75, 18% at p80). In mid-2024, Google stopped linking mobile search results directly to our mobile domain. That meant the ~60% of incoming page views referred by Google now experience the same redirect that everyone else experienced since 2010. With the redirect gone we more than reverse last year's regression, because it applies to everyone. (RFC § Problem statement, First perf report, Final stats report)
Examples of bugs and workarounds resolved by this transition: