Page MenuHomePhabricator

Elasticsearch pods failed to start due to Init:ImagePullBackOff
Closed, ResolvedPublic

Description

At around 2025-06-23 02:50Z we went through our normal Monday GKE maintenance window which resulted in a rolling node upgrade.

It appears that the elasticsearch pod were then unable to come up on the new node due to hitting Init:ImagePullBackOff for this image docker.io/bitnami/bitnami-shell:11-debian-11-r123. This is used for an init container.

Initially I assumed this was due to hitting the new docker rate limiting but actually it was due to the image totally disappearing from dockerhub. Presumably after bitnami renamed the repo (which admittedly they do describe as deprecated).

The resolution to this was changing the image name to bitnami/bitnami-shell-archived to point at the newly renamed archived repo.

Event Timeline

Failing to pull the init image due to hitting the pullback off.
The init image is not one of our container images hosted on github container registry but instead: docker.io/bitnami/bitnami-shell:11-debian-11-r123 which is presumably now caught by the docker image pull limits.

I would assume the only good solution is to move to an alternative registry and find a comparable image there.
After resolving this we will also need to audit our set-up for any other dockerhub images and move away from them.

Nope; I'm wrong (although maybe we will get bitten by this in a bit too). As @Andrew-WMDE pointed out to me the image has just gone. It's apparently been replaced by https://hub.docker.com/r/bitnami/os-shell

image.png (840×2 px, 164 KB)

Looks like it went down at 2:40
Briefly worked at around 3:40 and 3:45
Then down until 10:10