Strong support, in light of recent events.

Does this refer to all and any IO? For example, we mirror 26 repositories from Diffusion into Github (see also queries in T347577 and T349921).

What does “IO” mean in this context at all? Is it some kind of Phabricator jargon? In a general sense, I/O would include everything that happens with Diffusion: SSH reads, Git HTTP reads, browser HTTP reads, the disk I/O Phabricator does to store the files, etc. I guess it’s not what you mean, otherwise you would’ve simply said “Disable Diffusion repositories”.

The I/O term is in the Diffusion (Phabricator) settings. If you go to Diffusion and hit "Manage Repository" one of the repos there are a couple settings.

It's not just simply on or off.

For example a repo can be active/inactive, publishing/not publishing and there are configurable URIs for each one.

Under the URI section there can be different ways to git clone (http, https or ssh) and each one of them has a column "I/O" which can have values like "Default, Observe, None, ReadOnly, Mirror or ReadWrite ... "

--> https://we.phorge.it/book/phorge/article/diffusion_uris/#reference-i-o-types

Thanks for the pointers! At https://phabricator.wikimedia.org/source/mediawiki/manage/uris/, I see

• one No I/O entry pointing to https://github.com/wikimedia/3d2png (I have no idea how it got there…),
• one Observe entry pointing to https://gerrit-replica.wikimedia.org/r/mediawiki/core (which gives 404 in the browser, but works when I try to fetch from it with Git),
• one Mirror entry pointing to https://github.com/wikimedia/mediawiki,
• and a bunch of Read Only entries pointing to various phabricator.wikimedia.org and git-ssh.wikimedia.org URLs.

I guess the 3d2png entry is out of scope because it’s already disabled.

I hope the Gerrit entry won’t be disabled, as it’d render the mirror practically useless (it’d result in it being frozen in whatever state it currently is in).

Do you want to disable all other entries, i.e. all Read Only ones and the GitHub one? (What will update our GitHub mirror going forward?)

(What will update our GitHub mirror going forward?)

I might be wrong, but I believe that most of our GitHub mirrors (I believe including wikimedia/mediawiki) are updated directly from Gerrit, rather than via Diffusion.

For some of the repos that do mirror directly from Diffusion to GitHub (e.g. https://phabricator.wikimedia.org/source/tool-admin-web/manage/uris/ IIUC), it does feel like disabling some I/O settings there might be a problem.
(xref the list at T347577#9689792, though I guess that list may be slightly outdated if any Diffusion->GitHub mirroring has been added/removed since April 2024.)

[Side note: interestingly, looking at the GitHub activity for a random repo that's located in Gerrit but which still appears to have separate Diffusion->GitHub mirroring configured, it seems like Gerrit and Diffusion are sometimes git push-warring with each other to get commits mirrored first :P]

Sorry the ambiguous "IO" reference. As @Dzahn points out, that's a Phabricator/Phorge term. Every repo in diffusion has URIs.

URIs are ssh or https URLs that point to:

• builtin places where users can download from diffusion; e.g., phabricator.com/source/mediawiki.git
• admin-added places we want to mirror a diffusion repository; e.g., github.com
• admin-added places phabricator should observe for changes and fetch into diffusion; e.g., github.com

Each of these URIs can be enabled or disabled.

A better wording for my proposal would be:
Remove the ability to clone repositories from diffusion for all repositories except the Wikibase repos mentioned in this task

This would:

• Save resources – since all diffusion repositories should be mirrors, there's no reason folks need to clone from Phabricator.
• Enforce assumptions – we know Wikibase/WikibaseLexeme submodules are cloned from diffusion, and there's a tracking task to stop that. Hopefully that's the last legit use-case. Disabling clones of other repos will help us be sure.
• Lessen user confusion – since Phabricator can be code browser/code forge, and repository browsing works, it might make sense to a user to download a repository from Phabricator. Later that user would discover you can't push anything to Phabricator.

But it seems we cannot disable builtin repository URIs. We can only set them to "hidden", which hides them in the UI from end-users, but still allows cloning from strange URLs like diffusion/19/mediawiki.git.

So, in practice, my proposal is less straight-forward than I'd assumed. That is, it would require changes to the Diffusion application in Phorge (I think).

In T405596#11218257, @A_smart_kitten wrote:

(What will update our GitHub mirror going forward?)

I might be wrong, but I believe that most of our GitHub mirrors (I believe including wikimedia/mediawiki) are updated directly from Gerrit, rather than via Diffusion.

That is correct, repos in Gerrit are mirrored by Gerrit to GitHub.

In T405596#11218257, @A_smart_kitten wrote:

[Side note: interestingly, looking at the GitHub activity for a random repo that's located in Gerrit but which still appears to have separate Diffusion->GitHub mirroring configured, it seems like Gerrit and Diffusion are sometimes git push-warring with each other to get commits mirrored first :P]

Neat. Proposal addendum:

If a diffusion repo has a gerrit.wikimedia.org or gerrit-replica.wikimedia.org URI that is set to Observe, then it should NOT have a github.com URI set to Mirror.

I manually disabled mirroring from diffusion for the repo you flagged @A_smart_kitten , thanks for flagging that.

For the records, we have at least two customizations in this area: https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/commit/f3d0345c028babc8dbe0e252c4c5fee652c27b85 and https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/commit/54ed31d870ff4e9d6a0ee408ea1a333aa181f9ea . The latter needs to be expanded if we wanted to apply this also to repos created in the future.

As a test, I changed the I/O Type for all Read Only URIs listed on https://phabricator.wikimedia.org/source/tool-jouncebot/manage/uris/ (I picked a random project) from "Read Only" to "No I/O". This seemed to have worked:

[acko@machina ~]$ git clone https://phabricator.wikimedia.org/source/tool-jouncebot.git
Cloning into 'tool-jouncebot'...
fatal: unable to access 'https://phabricator.wikimedia.org/source/tool-jouncebot.git/': The requested URL returned error: 403
[acko@machina ~]$ git clone http://phabricator.wikimedia.org/diffusion/3307/tool-jouncebot.git
Cloning into 'tool-jouncebot'...
fatal: unable to access 'http://phabricator.wikimedia.org/diffusion/3307/tool-jouncebot.git/': The requested URL returned error: 403

In T405596#11242761, @thcipriani wrote:

If a diffusion repo has a gerrit.wikimedia.org or gerrit-replica.wikimedia.org URI that is set to Observe, then it should NOT have a github.com URI set to Mirror.

While I never fully understood the "tracking-enabled" meaning and neither the confusing ioType called "default", I believe that we have (at least) 25 repositories observing gitlab or gerrit-replica && mirroring to github according to
SELECT CONCAT("https://phabricator.wikimedia.org/diffusion/", r.id, "/manage/uris/") AS repoURI, r.name, uO.uri AS observedFromUri, uM.uri AS MirroredToUri FROM phabricator_repository.repository r INNER JOIN phabricator_repository.repository_uri uM ON r.phid = uM.repositoryPHID INNER JOIN phabricator_repository.repository_uri uO ON r.phid = uO.repositoryPHID WHERE r.details LIKE "%\"tracking-enabled\":\"active\"%" AND uM.ioType = "mirror" AND uM.isDisabled = 0 AND uM.uri LIKE "%github%" AND uO.ioType = "observe" AND uO.isDisabled = 0;

In T405596#11242761, @thcipriani wrote:

A better wording for my proposal would be:
Remove the ability to clone repositories from diffusion for all repositories except the Wikibase repos mentioned in this task

Thanks, that’s much clearer.

In T405596#11242868, @Aklapper wrote:

As a test, I changed the I/O Type for all Read Only URIs listed on https://phabricator.wikimedia.org/source/tool-jouncebot/manage/uris/ (I picked a random project) from "Read Only" to "No I/O". This seemed to have worked:

[acko@machina ~]$ git clone https://phabricator.wikimedia.org/source/tool-jouncebot.git
Cloning into 'tool-jouncebot'...
fatal: unable to access 'https://phabricator.wikimedia.org/source/tool-jouncebot.git/': The requested URL returned error: 403
[acko@machina ~]$ git clone http://phabricator.wikimedia.org/diffusion/3307/tool-jouncebot.git
Cloning into 'tool-jouncebot'...
fatal: unable to access 'http://phabricator.wikimedia.org/diffusion/3307/tool-jouncebot.git/': The requested URL returned error: 403

Great! (Next time, feel free to experiment with https://phabricator.wikimedia.org/source/tool-atiro/ instead of random other people’s projects – I don’t use the Phabricator mirror, and probably other people neither. As it turns out, at least its display has been broken ever since I renamed the default branch to main…)

aklapper opened https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/103

Draft: Set IO_NONE by default on repos; always expose observed URI

This ticket is getting harder to follow and by now covers numerous things. We may want to move some items into subtasks.

In my understanding, we want to:

• Stop having Diffusion as a proxy between two other code hosting systems
  • For Diffusion repos which observe a gerrit or gitlab URI and also mirror to github [1], change that setup not to have Diffusion "in the middle", per T405596#11242761. This involves also changing settings in other code hosting systems.
    • Update/resolve T361859: Document diffusion->github mirroring to https://github.com/toolforge/ on wikitech.
• Clean up some Diffusion URIs' settings with no consequences:
  • Never expose the built-in ssh URIs by setting [{"type": "io", "value":"none"},{"type": "display", "value":"never"}] [2]. We sunset that service in T296022 anyway, this just makes SQL queries more consistent and the "Manage URIs" UI reflect reality. See also declined T347408: Do not expose dead git-ssh.wikimedia.org service as repo Clone URLs (defined in diffusion.ssh-host setting).
    • Also handle the remaining repos/URIs which I cannot edit as I lack permissions
    • Striker: Set ssh URIs IO to none when creating repos
  • Always expose the observed external URI by setting [{"type": "display", "value":"always"}] [3]. No visible effect as we've been doing that already in the "Clone" button UI via the hack in T361997.
    • Also handle the remaining repos/URIs which I cannot edit as I lack permissions
    • T407624: Striker: Always expose observed external URI when creating Diffusion repositories
• Disallow I/O on remaining built-in Diffusion URIs
  • Never expose the built-in http URIs by setting [{"type": "io", "value":"none"},{"type": "display", "value":"never"}] on built-in http URIs
    • T407705: Striker: Always set http IO to none when creating Diffusion repositories
  • Define exception repos/URIS to exclude for now, cf T349921 and T374926
  • Public communication
  • Potentially set up a custom 403 message for CLI clients when git clone suddenly fails. This would have to be implemented on the webserver software level. See https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/profile/templates/cache/haproxy/tls_terminator.cfg.erb#L315 for inspiration.
  • Once exceptions are handled, never expose the built-in https URIs by setting [{"type": "io", "value":"none"},{"type": "display", "value":"never"}] on all remaining built-in https URIs [4]
  • Once exceptions are handled, per the code in DiffusionServeController, ultimately change https://phabricator.wikimedia.org/config/edit/diffusion.allow-http-auth/ from true to false
• Cover the future (Diffusion repos to be created):
  • Make Phab code by default set up new Diffusion repos with io=none and display=never for all future built-in URIs. Done in https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/commit/83fe6f47c65ea99c8e7a2bcb38b6aa832e7f105d
  • Make Phab code by default set [{"type": "io", "value":"none"},{"type": "display", "value":"always"}] for future observed URIs, if feasible - proposed in https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/103
  • Cleanup: Potentially revert custom code to only expose observed URIs in the "Clone" button UI, introduced in T361997, superseded by changes above: https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/commit/f3d0345c028babc8dbe0e252c4c5fee652c27b85
• Evaluate
  • Check SELECT r.name, r.id, rp.* FROM phabricator_repository.repository_pullevent rp INNER JOIN phabricator_repository.repository r ON r.phid = rp.repositoryPHID WHERE resultType = "pull" AND resultCode = "200"; which lists the pulls which are still successful
  • Check SELECT r.name, r.id, rp.* FROM phabricator_repository.repository_pullevent rp INNER JOIN phabricator_repository.repository r ON r.phid = rp.repositoryPHID WHERE resultType != "pull"; which should show an increase in This repository is not available over HTTP. entries

[1] SELECT CONCAT("https://phabricator.wikimedia.org/diffusion/", r.id, "/manage/uris/") AS repoURI, r.name, uO.uri AS observedFromUri, uM.uri AS MirroredToUri FROM phabricator_repository.repository r INNER JOIN phabricator_repository.repository_uri uM ON r.phid = uM.repositoryPHID INNER JOIN phabricator_repository.repository_uri uO ON r.phid = uO.repositoryPHID WHERE r.details LIKE "%\"tracking-enabled\":\"active\"%" AND uM.ioType = "mirror" AND uM.isDisabled = 0 AND uM.uri LIKE "%github%" AND uO.ioType = "observe" AND uO.isDisabled = 0;
[2] SELECT u.id, CONCAT("https://phabricator.wikimedia.org/diffusion/", r.id, "/manage/uris/") AS repoURI, r.name, u.uri, builtinProtocol, ioType, displayType FROM phabricator_repository.repository r INNER JOIN phabricator_repository.repository_uri u ON r.phid = u.repositoryPHID WHERE u.builtinProtocol = "ssh" AND u.ioType != "observe" AND u.ioType != "mirror" AND u.displayType != "never";
[3] SELECT u.id FROM phabricator_repository.repository r INNER JOIN phabricator_repository.repository_uri u ON r.phid = u.repositoryPHID WHERE r.details LIKE "%\"tracking-enabled\":\"active\"%" AND u.builtinProtocol IS NULL AND u.ioType = "observe" AND u.displayType != "always";
[4] SELECT u.id, CONCAT("https://phabricator.wikimedia.org/diffusion/", r.id, "/manage/uris/") AS repoURI, r.name, u.uri, builtinProtocol, ioType, displayType FROM phabricator_repository.repository r INNER JOIN phabricator_repository.repository_uri u ON r.phid = u.repositoryPHID WHERE (u.builtinProtocol = "http" OR u.builtinProtocol = "https") AND u.ioType != "observe" AND u.ioType != "mirror";

As a next step I'm tempted to mass-set all http (no s at the end) URIs which neither observe nor mirror to io=none && display=never, so afterwards only non-mirror&&non-observe URIs for https (note the s at the end) would be left.

I guess I'm looking at @thcipriani for a {yes | no wait} input whether to exclude certain repos from getting "blocked" when disabling http access, or get a green light and just do it (and afterwards find out the hard way who was not already using https).

In T405596#11244081, @Aklapper wrote:

• For Diffusion repos which observe a gerrit or gitlab URI and also mirror to github [1], change that setup not to have Diffusion "in the middle", per T405596#11242761.

I can't immediately cite a source for this but I feel like GitLab repos don't automatically mirror to GitHub (in the way that Gerrit repos do). Wouldn't disabling GitHub mirroring for these GitLab Diffusion-repo-mirrors stop the GitHub-repo-mirrors from being updated (or am I misunderstanding something)?

As a next step I'm tempted to mass-set all http (no s at the end) URIs which neither observe nor mirror to io=none && display=never

+1. Already did this for a bunch of other repos in the past.

In T405596#11282717, @A_smart_kitten wrote:

In T405596#11244081, @Aklapper wrote:

• For Diffusion repos which observe a gerrit or gitlab URI and also mirror to github [1], change that setup not to have Diffusion "in the middle", per T405596#11242761.

I can't immediately cite a source for this but I feel like GitLab repos don't automatically mirror to GitHub (in the way that Gerrit repos do). Wouldn't disabling GitHub mirroring for these GitLab Diffusion-repo-mirrors stop the GitHub-repo-mirrors from being updated (or am I misunderstanding something)?

The open source version of gitlab does support push mirroring, but that support requires configuring the push token (i.e. github access credentials) separately for each and every origin repo. This makes it a relatively untenable solution for us at scale, especially for mirroring any repo where we would rather not expose the downstream access credentials to the upstream repo owners. We can build a different system to sit between gitlab and github to enable push mirroring if diffusion is no longer suitable, but we would need to build something.

<tl;dr>: I disabled I/O for any non-observed, non-mirrored http clone URIs that I had permissions to edit. According to Phab's pull logs nobody should be affected.

Longer version: Nothing beats outages over the weekenddata-driven decisions, so I checked pull traffic via
SELECT r.name, rp.* FROM phabricator_repository.repository_pullevent rp INNER JOIN phabricator_repository.repository r ON r.phid = rp.repositoryPHID WHERE remoteProtocol != "https";
and it was empty. And also, DiffusionServeController::handleRequest() does differentiate between PROTOCOL_HTTP and PROTOCOL_HTTPS.

I handled those repositories which I could not edit before, by following https://wikitech.wikimedia.org/wiki/Phabricator#Unlocking_edit_permissions_on_random_objects. That means
SELECT u.phid, u.id, CONCAT("https://phabricator.wikimedia.org/diffusion/", r.id, "/manage/uris/") AS repoURI, r.phid, r.name, u.uri, builtinProtocol, ioType, displayType FROM phabricator_repository.repository r INNER JOIN phabricator_repository.repository_uri u ON r.phid = u.repositoryPHID WHERE (u.builtinProtocol = "http" OR u.builtinProtocol = "ssh") AND u.ioType != "observe" AND u.ioType != "mirror" AND u.displayType != "never";
and
SELECT u.phid, u.id, CONCAT("https://phabricator.wikimedia.org/diffusion/", r.id, "/manage/uris/") AS repoURI, r.phid, r.name, u.uri, builtinProtocol,u.id FROM phabricator_repository.repository r INNER JOIN phabricator_repository.repository_uri u ON r.phid = u.repositoryPHID WHERE r.details LIKE "%\"tracking-enabled\":\"active\"%" AND u.builtinProtocol IS NULL AND u.ioType = "observe" AND u.displayType != "always";
show zero results for now as it ideally should be, until Striker creates the next new repos (see the subtasks here).