rpmostree-core: Emulate new %sysusers RPM scriplet #5334

travier · 2025-03-12T15:52:43Z

rpmostree-core: Emulate new %sysusers RPM scriplet

As part of the migration to direct sysusers support in RPMs, packages
are no longer creating their users/groups in %pre scripts as they rely
on RPM to do the work in a built in %sysusers scriplet.

The RPM library does not expose a direct interface to call those
scriplets so we instead emulate the process by directly calling out to
the binary pointed by %__systemd_sysusers, if it is set.

See: https://fedoraproject.org/wiki/Changes/RPMSuportForSystemdSysusers
See: rpm-software-management/rpm@009d139

See: fedora-silverblue/issue-tracker#636
See: https://gitlab.com/fedora/ostree/sig/-/issues/70
See: #5333

tests/compose-image: Test with Fedora 40, 41 & 42

Add systemd-standalone-sysusers to container for F42+.
Starting with Fedora 42, RPM will use sysusers to create users and we
emulate that support by running sysusers in the rootfs before running
other scriptlets.
See: Implement new RPM %sysusers script to fix user/group creation with sysusers #5333
Test with Fedora 41 by default
Replace dnf with dnf5 for F41+
See: https://fedoraproject.org/wiki/Changes/SwitchToDnf5
tests/compose-image: Remove fedora-repos-modular
Support for modularity has been removed from Fedora.
See: https://fedoraproject.org/wiki/Changes/RetireModularity

keszybz · 2025-03-12T15:55:09Z

Doesn't look too terrible ;)

travier · 2025-03-12T16:00:48Z

PR description is my understanding, may not be fully accurate, suggestions welcomed (and then I'll update the commit message).

AdamWill · 2025-03-12T16:02:41Z

The code is written as it is specifically in order not to do this sort of thing. There's a long comment to that effect:

      /* We're technically deviating from RPM here by running all the %pre's
       * beforehand, rather than each package's %pre & %post in order. Though I
       * highly doubt this should cause any issues. The advantage of doing it
       * this way is that we only need to read the passwd/group files once
       * before applying the overrides, rather than after each %pre.
       */

so it seems like we might want to try and honor that goal somehow. Maybe special-case systemd and do something like this?

pre loop
read passwd
systemd post
read passwd again
post loop

if we don't want to do that, we should probably at least update the comment...

AdamWill · 2025-03-12T16:04:31Z

also, doing it this way doesn't feel 100% safe, because it seems like it would rely on systemd coming very early in the %post loop - because it must be done before any other package containing a file owned by a user or group that would be created by systemd's %post - but we don't appear to do anything to guarantee that that's the case?

travier · 2025-03-12T16:05:31Z

That should work but would make this very specific to the systemd package name. I agree that this is a bit of a brute force logic but maybe this is safer at the cost of re-reading the files a bit often?

AdamWill · 2025-03-12T16:06:21Z

well, then, how about:

pre loop
directly cause a run of systemd-sysusers in whatever context it needs to happen
read passwd
post loop

that seems most similar to what rpm is now doing, i guess?

travier · 2025-03-12T16:06:59Z

also, doing it this way doesn't feel 100% safe, because it seems like it would rely on systemd coming very early in the %post loop - because it must be done before any other package containing a file owned by a user or group that would be created by systemd's %post - but we don't appear to do anything to guarantee that that's the case?

Agree that this is not a "fix", but more of a workaround. Ideally, we would add the RPM sysusers scriptlet support to rpm-ostree as a specific phase here before running the %post.

travier · 2025-03-12T16:11:39Z

For this to be "correct" (i.e. matching how RPM does it) we would have to re-implement rpm-software-management/rpm@009d139 as far as I understand.

travier · 2025-03-12T16:12:39Z

well, then, how about:

pre loop

directly cause a run of systemd-sysusers in whatever context it needs to happen

read passwd

post loop

that seems most similar to what rpm is now doing, i guess?

That could be another option.

AdamWill · 2025-03-12T16:18:07Z

i'm trying to figure out how to subvert the 'run a package script' bits to just run a script we make up on the fly...

travier · 2025-03-12T16:26:21Z

rpm-ostree runs post-process scripts via https://github.com/coreos/rpm-ostree/blob/main/rust/src/composepost.rs#L515, so we should be able to directly add a call to sysusers using a similar logic.

AdamWill · 2025-03-12T16:39:36Z

well, Ii'm trying to hook into the various levels of script running function in src/libpriv/rpmostree-scripts.cxx , which does more or less the same. But it's kinda tedious to figure out if there's a neat way to slip in, it's all tied into RPM business logic. but I don't want to repeat all the bwrap stuff...

for now I guess I'll just test this PR as-is, if it at least fixes ostree builds for now that's better than what we have, we can slap it in as a downstream patch till Colin or someone can decide what the best long-term fix is...

AdamWill · 2025-03-12T18:31:39Z

it does look like this works around the issue for now, so I'll go ahead and backport it downstream officially. I'll try and figure out if there's a significant impact on run time, too, once the test completes.

travier · 2025-03-13T09:06:20Z

Rebased and updated the commit description

travier · 2025-03-13T16:23:30Z

From a discussion with Jonathan, this will not work for package layering as the systemd %post won't be called there, so this is not a complete workaround.

jlebon · 2025-03-13T16:50:40Z

Whipped this up while chatting with @travier:

diff --git a/src/libpriv/rpmostree-core.cxx b/src/libpriv/rpmostree-core.cxx
index 70905996..208696fe 100644
--- a/src/libpriv/rpmostree-core.cxx
+++ b/src/libpriv/rpmostree-core.cxx
@@ -4422,6 +4422,26 @@ rpmostree_context_assemble (RpmOstreeContext *self, GCancellable *cancellable, G
 
   progress->end ("");
 
+  /* if the %__systemd_sysusers macro is defined, that _very likely_
+   * means that we're using an rpm with built-in sysusers support:
+   * https://fedoraproject.org/wiki/Changes/RPMSuportForSystemdSysusers Ideally,
+   * librpm would expose that part as a public API but it's currently just
+   * done during the transaction. So here we just do it ourselves (though a bit
+   * differently; librpm checks for the user/group provides before proceeding to call
+   * sysusers _just_ for that package. Here we do it for all of them... See also:
+   * https://github.com/coreos/rpm-ostree/issues/5333 */
+  g_autofree char *sysusers_cmd = rpmExpand ("%__systemd_sysusers", NULL);
+  if (sysusers_cmd && *sysusers_cmd)
+    {
+      CXX_TRY_VAR (bwrap,
+                   rpmostreecxx::bubblewrap_new_with_mutability (
+                       tmprootfs_dfd, rpmostreecxx::BubblewrapMutability::MutateFreely),
+                   error);
+      bwrap->append_child_arg (sysusers_cmd);
+      if (!CXX (bwrap->run (*cancellable), error))
+        return glnx_prefix_error (error, "Running systemd-sysusers (%s)", sysusers_cmd);
+    }
+
   /* Some packages expect to be able to make temporary files here
    * for obvious reasons, but we otherwise make `/var` read-only.
    */

Not tested...

cgwalters · 2025-03-13T18:37:23Z

The approach of running systemd-sysusers "wholesale" does seem to match with the current design.

rpmostreecxx::BubblewrapMutability::Immutable

Hmm don't we need mutability? Though I forget if that means just /usr is immutable.

jlebon · 2025-03-13T19:45:06Z

The approach of running systemd-sysusers "wholesale" does seem to match with the current design.

rpmostreecxx::BubblewrapMutability::Immutable

Hmm don't we need mutability? Though I forget if that means just /usr is immutable.

Ahh I think you're right. I thought the reverse and that it only implied /usr. There isn't a mode currently for "/usr only", but meh.. fine with MutateFreely. Updated patch! I think @travier is going to take it for a spin and update this PR if it works.

travier · 2025-03-14T10:14:31Z

Hum, I get:

bwrap: Can't find source path etc: No such file or directory
error: Installing packages: Running systemd-sysusers (%__systemd_sysusers): bwrap(%__systemd_sysusers): Child process killed by signal 1

Looks like everything is already in /usr/etc at this point.
Moving the code a few lines below where we setup /etc.

bwrap: execvp %__systemd_sysusers: No such file or directory
Preserved workdir: rpm-ostree-compose.Um4Cmx
error: Installing packages: Running systemd-sysusers (%__systemd_sysusers): bwrap(%__systemd_sysusers): Child process killed by signal 1

So now the macro is not expanded.
Replacing the macro with /usr/bin/systemd-sysusers directly appear to make this work, so something is off in the macro expansion but I don't know what.

travier · 2025-03-14T11:00:15Z

This fails weirdly on my systems which seems to point that there is another bug somewhere else:

Writing rpmdb... done
Regenerating rpmdb for target
error: Subprocess failed: ExitStatus(unix_wait_status(134))
r for geoclue) with UID 976 and GID 976.
Creating group 'openvpn' with GID 975.
Creating user 'openvpn' (OpenVPN) with UID 975 and GID 975.
Creating group 'passim' with GID 974.
Creating user 'passim' (Local Caching Server) with UID 974 and GID 974.
Creating group 'pipewire' with GID 973.
Creating user 'pipewire' (PipeWire System Daemon) with UID 973 and GID 973.
Creating group 'rtkit' with GID 172.
Creating user 'rtkit' (RealtimeKit) with UID 172 and GID 172.
Creating group 'systemd-coredump' with GID 972.
Creating user 'systemd-coredump' (systemd Core Dumper) with UID 972 and GID 972.
Creating group 'systemd-oom' with GID 971.
Creating user 'systemd-oom' (systemd Userspace OOM Killer) with UID 971 and GID 971.
Creating group 'unbound' with GID 970.
Creating user 'unbound' (Unbound DNS resolver) with UID 970 and GID 970.
Creating group 'usbmuxd' with GID 113.
Creating user 'usbmuxd' (usbmuxd user) with UID 113 and GID 113.
Creating user 'vboxadd' (n/a) with UID 969 and GID 1.
munmap_chunk(): invalid pointer

The Creating... ouput should have been output during the bwrap call for sysusers but it apparently did not, I don't know why.

travier · 2025-03-14T11:03:33Z

The approach of running systemd-sysusers "wholesale" does seem to match with the current design.

Could you expand on what you mean? Do you mean that it's not matching the design in RPM or the one in rpm-ostree?

keszybz · 2025-03-14T11:08:34Z

munmap_chunk(): invalid pointer

Is that from systemd-sysusers?

travier · 2025-03-14T11:43:15Z

This fails similarly in CI here:

Regenerating rpmdb for target
error: Executing compose install: Subprocess failed: ExitStatus(unix_wait_status(134))
 190.
Creating group 'dbus' with GID 81.
Creating user 'dbus' (System Message Bus) with UID 81 and GID 81.
Creating group 'polkitd' with GID 114.
Creating user 'polkitd' (User for polkitd) with UID 114 and GID 114.
Creating group 'systemd-coredump' with GID 999.
Creating user 'systemd-coredump' (systemd Core Dumper) with UID 999 and GID 999.
Creating group 'tss' with GID 59.
Creating user 'tss' (Account used for TPM access) with UID 59 and GID 59.
DEBUG Ending progress Task; suffix=None
DEBUG Creating progress "Running pre scripts" with type Task
DEBUG Ending progress Task; suffix=Some("10 done")
DEBUG Creating progress "Running post scripts" with type Task
DEBUG Ending progress Task; suffix=None
DEBUG Creating progress "Running posttrans scripts" with type Task
DEBUG Ending progress Task; suffix=Some("10 done")
DEBUG Creating progress "Writing rpmdb" with type Task
DEBUG Ending progress Task; suffix=None
free(): invalid pointer

I don't think it's from sysusers. Likely something not correct in the patch here.

bwrap: execvp /usr/bin/systemd-sysusers: No such file or directory

The container build (that does not include sysusers) rightly failed on the hardcoded sysusers binary path.

keszybz · 2025-03-14T14:31:14Z

``` container: bwrap: execvp /usr/bin/systemd-sysusers: No such file or directory container: Running systemd-sysusers...done container: Preserved workdir: rpm-ostree-compose.mgrlmt container: error: Installing packages: Running systemd-sysusers (/usr/bin/systemd-sysusers): bwrap(/usr/bin/systemd-sysusers): Child process killed by signal 1 ``` Hmmm.

This was from a previous run. I didn't refresh the page.

travier · 2025-03-14T14:37:32Z

There is something unexpected with the RPM macro expansion.

Edit: The macro was being expanded on the host, not the composed rootfs, and I'm still running F41.

travier · 2025-03-17T14:20:15Z

I've reworked the test to test all Fedora releases still stable + f42.

travier · 2025-03-17T15:17:38Z

Updated backport PR as we'll need this down to F42: https://src.fedoraproject.org/rpms/rpm-ostree/pull-request/67

travier · 2025-03-17T16:05:06Z

Arg, it timed out on a Pagure access issue:

+ git clone --depth=1 https://pagure.io/workstation-ostree-config --branch f41 workstation-ostree-config.41
Cloning into 'workstation-ostree-config.41'...
fatal: unable to access 'https://pagure.io/workstation-ostree-config/': SSL connection timeout

tests/compose-image.sh

- Add systemd-standalone-sysusers to container for F42+. Starting with Fedora 42, RPM will use sysusers to create users and we emulate that support by running sysusers in the rootfs before running other scriptlets. See: coreos#5333 - Test with Fedora 41 by default - Replace dnf with dnf5 for F41+ See: https://fedoraproject.org/wiki/Changes/SwitchToDnf5 - tests/compose-image: Remove fedora-repos-modular Support for modularity has been removed from Fedora. See: https://fedoraproject.org/wiki/Changes/RetireModularity

jlebon

LGTM. One comment, but not worth a respin.
Thanks for expanding the test matrix!

jlebon · 2025-03-17T20:54:49Z

src/libpriv/rpmostree-core.cxx

+  /* if the %__systemd_sysusers macro is defined, that _very likely_ means that
+   * we're using an rpm with built-in sysusers support:
+   * https://fedoraproject.org/wiki/Changes/RPMSuportForSystemdSysusers
+   * Ideally, librpm would expose that part as a public API but it's currently
+   * just done during the transaction. So here we just do it ourselves, though
+   * a bit differently: librpm checks for the user/group provides before
+   * proceeding to call sysusers _just_ for that package. Here we do it for all
+   * of them. See also: https://github.com/coreos/rpm-ostree/issues/5333 */


That comment should probably move to the Rust side now.

jlebon · 2025-03-17T20:57:24Z

/retest

cgwalters

Only gave this a skim but looks sane to me.

cgwalters · 2025-03-17T22:31:53Z

rust/src/core.rs


+/// Run systemd-sysusers in the rootfs if the %__systemd_sysusers RPM macro is set.
+pub(crate) fn run_sysusers(rootfs_dfd: i32) -> CxxResult<()> {
+    let args: Vec<_> = vec!["rpm", "--eval=%{?__systemd_sysusers}"]


Pointless nit: This can just be a static slice i.e. drop the vec!

cgwalters · 2025-03-17T22:33:39Z

Thanks for expanding the test matrix!

Yes! Overdue for sure

jlebon · 2025-04-03T15:42:25Z

Regression in #5353.

cgwalters · 2025-04-03T16:00:41Z

Regression in #5353.

Hmm this may be that we're missing the nss-altfiles setup at this stage? That's a pretty bad bug if so

Pointed out by Colin in coreos#5334 (comment).

In the work to rebase FCOS/RHCOS to fedora-bootc/rhel-bootc, I want to make it possible to opt out of the passwd/group bits baked in the minimal manifests (the problem there of course goes back to UID/GID drift issues -- this has caused issues as well on the IoT/Edge side). The current passwd/group files in the minimal manifest are both outdated and incomplete (most of it can in fact trace its lineage all the way back to Fedora Atomic Host). But more generally, in the FROM scratch flow, users _should_ have the ability to customize which user gets which UID. And one could imagine an option like `bootc-base-imagectl --passwd mypasswd --group mygroup` but (1) while the format is well-known, using it as an interface for building root filesystems is less so, and for example has no correspondence in the dnf world, and (2) such an interface implies taking ownership of all the entries. A more universal config file for this of course is systemd-sysusers. And recently, rpm-ostree learned in coreos#5334 to run systemd-sysusers prior to running any scriptlets for parity with similar functionality in RPM. So it basically works today in Fedora to completely ditch the monolithic passwd/group stuff and instead fixate UIDs via sysusers dropins. And then, combining that with `bootc-base-imagectl --add-dir`, this becomes possible to feed into the FROM scratch flow. We want to enable this flow in RHEL as well. All we need really is to force rpm-ostree to run systemd-sysusers even if the `%__systemd_sysusers` macro is undefined. Add a new `sysusers:` knob to the treefile. Currently, it takes one of two values: - `compose-auto`: `systemd-sysusers` is run at compose time if the `%__systemd_sysusers` RPM macro is defined (status quo) - `compose-forced`: `systemd-sysusers` is always run at compose time, even if the `%__systemd_sysusers` RPM macro is undefined And the net result of this then is it moves the emphasis further to systemd-sysusers as a way of ensuring UID coherence across builds and e.g. all the flexibility it implies around configurability/composition/overrides etc... (Worth noting that the systemd %post script actually always runs systemd-sysusers currently. But we can't rely on this because it's itself part of the RPM transaction and so has no effect on packages which have run before.) Then, for the bootc-base-imagectl flow, we can add a new `--sysusers` switch which sets this new option and crucially disables all the `check-passwd`/`check-group` logic. This is related to the nss-altfiles to sysusers migration though also somewhat independent. With `compose-forced`, the split back into `/usr/lib/{passwd,group}` still happens at the end of the compose. However I still went with the `sysusers:` enum and left room for other strategies to be implemented, since any strategy around nss-altfiles migration must take into account what semantics are desired from the systemd-sysusers run at compose time.

In the work to rebase FCOS/RHCOS to fedora-bootc/rhel-bootc, I want to make it possible to opt out of the passwd/group bits baked in the minimal manifests (the problem there of course goes back to UID/GID drift issues -- this has caused issues as well on the IoT/Edge side). The current passwd/group files in the minimal manifest are both outdated and incomplete (most of it can in fact trace its lineage all the way back to Fedora Atomic Host). But more generally, in the FROM scratch flow, users _should_ have the ability to customize which user gets which UID. And one could imagine an option like `bootc-base-imagectl --passwd mypasswd --group mygroup` but (1) while the format is well-known, using it as an interface for building root filesystems is less so, and for example has no correspondence in the dnf world, and (2) such an interface implies taking ownership of all the entries. A more universal config file for this of course is systemd-sysusers. And recently, rpm-ostree learned in coreos#5334 to run systemd-sysusers prior to running any scriptlets for parity with similar functionality in RPM. So it basically works today in Fedora to completely ditch the monolithic passwd/group stuff and instead fixate UIDs via sysusers dropins. And then, combining that with `bootc-base-imagectl --add-dir`, this becomes possible to feed into the FROM scratch flow. We want to enable this flow in RHEL as well. All we need really is to force rpm-ostree to run systemd-sysusers even if the `%__systemd_sysusers` macro is undefined. Add a new experimental `sysusers:` knob to the treefile. Currently, it takes one of two values: - `compose-auto`: `systemd-sysusers` is run at compose time if the `%__systemd_sysusers` RPM macro is defined (status quo) - `compose-forced`: `systemd-sysusers` is always run at compose time, even if the `%__systemd_sysusers` RPM macro is undefined And the net result of this then is it moves the emphasis further to systemd-sysusers as a way of ensuring UID coherence across builds and e.g. all the flexibility it implies around configurability/composition/overrides etc... (Worth noting that the systemd %post script actually always runs systemd-sysusers currently. But we can't rely on this because it's itself part of the RPM transaction and so has no effect on packages which have run before.) Then, for the bootc-base-imagectl flow, we can add a new `--sysusers` switch which sets this new option and crucially disables all the `check-passwd`/`check-group` logic. This is related to the nss-altfiles to sysusers migration though also somewhat independent. With `compose-forced`, the split back into `/usr/lib/{passwd,group}` still happens at the end of the compose. However I still went with the `sysusers:` enum and left room for other strategies to be implemented, since any strategy around nss-altfiles migration must take into account what semantics are desired from the systemd-sysusers run at compose time.

travier force-pushed the fix-user-creation-in-post branch from b3217c7 to ae2c0d2 Compare March 13, 2025 08:40

travier force-pushed the fix-user-creation-in-post branch from ae2c0d2 to 36b1bd5 Compare March 14, 2025 11:00

travier force-pushed the fix-user-creation-in-post branch from 36b1bd5 to d8179cd Compare March 14, 2025 11:05

travier force-pushed the fix-user-creation-in-post branch from d8179cd to 84c4a22 Compare March 14, 2025 12:40

jlebon force-pushed the fix-user-creation-in-post branch from 84c4a22 to 7c21d84 Compare March 14, 2025 14:31

travier force-pushed the fix-user-creation-in-post branch from b1ad7fa to 8baa2a8 Compare March 17, 2025 14:03

travier force-pushed the fix-user-creation-in-post branch from 110fb83 to e90ba11 Compare March 17, 2025 15:13

travier force-pushed the fix-user-creation-in-post branch 2 times, most recently from 35f6f5f to 7918173 Compare March 17, 2025 16:29

github-advanced-security bot found potential problems Mar 17, 2025

View reviewed changes

tests/compose-image.sh Fixed Show fixed Hide fixed

travier force-pushed the fix-user-creation-in-post branch from 7918173 to 9eb66a4 Compare March 17, 2025 16:30

jlebon approved these changes Mar 17, 2025

View reviewed changes

jlebon enabled auto-merge March 17, 2025 20:57

cgwalters approved these changes Mar 17, 2025

View reviewed changes

jlebon merged commit d81fd0b into coreos:main Mar 17, 2025
20 checks passed

travier deleted the fix-user-creation-in-post branch March 18, 2025 09:27

This was referenced Apr 1, 2025

Various enhancements and tweaks for OCP node image build #5351

Merged

rpm-ostree-2025.7 creates rpc group with wrong gid 978 #5353

Closed

jlebon added a commit to jlebon/rpm-ostree that referenced this pull request Apr 3, 2025

rust/core: Drop unnecessary Vec

09be408

Pointed out by Colin in coreos#5334 (comment).

cgwalters mentioned this pull request May 26, 2025

Cannot layer any package adding a user or group on Fedora 42 #5365

Closed

jlebon mentioned this pull request Jul 4, 2025

treefile: Add sysusers: knob #5427

Merged

rpmostree-core: Emulate new %sysusers RPM scriplet #5334

rpmostree-core: Emulate new %sysusers RPM scriplet #5334

Uh oh!

Conversation

travier commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

keszybz commented Mar 12, 2025

Uh oh!

travier commented Mar 12, 2025

Uh oh!

AdamWill commented Mar 12, 2025

Uh oh!

AdamWill commented Mar 12, 2025

Uh oh!

travier commented Mar 12, 2025

Uh oh!

AdamWill commented Mar 12, 2025

Uh oh!

travier commented Mar 12, 2025

Uh oh!

travier commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

travier commented Mar 12, 2025

Uh oh!

AdamWill commented Mar 12, 2025

Uh oh!

travier commented Mar 12, 2025

Uh oh!

AdamWill commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AdamWill commented Mar 12, 2025

Uh oh!

travier commented Mar 13, 2025

Uh oh!

travier commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jlebon commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters commented Mar 13, 2025

Uh oh!

jlebon commented Mar 13, 2025

Uh oh!

travier commented Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

travier commented Mar 14, 2025

Uh oh!

travier commented Mar 14, 2025

Uh oh!

keszybz commented Mar 14, 2025

Uh oh!

travier commented Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

keszybz commented Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

travier commented Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

travier commented Mar 17, 2025

Uh oh!

travier commented Mar 17, 2025

Uh oh!

travier commented Mar 17, 2025

Uh oh!

Uh oh!

jlebon left a comment

Choose a reason for hiding this comment

Uh oh!

jlebon Mar 17, 2025

Choose a reason for hiding this comment

Uh oh!

jlebon commented Mar 17, 2025

Uh oh!

travier commented Mar 12, 2025 •

edited

Loading

travier commented Mar 12, 2025 •

edited

Loading

AdamWill commented Mar 12, 2025 •

edited

Loading

travier commented Mar 13, 2025 •

edited

Loading

jlebon commented Mar 13, 2025 •

edited

Loading

travier commented Mar 14, 2025 •

edited

Loading

travier commented Mar 14, 2025 •

edited

Loading

keszybz commented Mar 14, 2025 •

edited

Loading

travier commented Mar 14, 2025 •

edited

Loading