| wmcs-alerts | |
| Thu, Aug 21, 4:21 PM |
| F65819686: Screenshot 2025-08-21 at 18.19.05.png | |
| Thu, Aug 21, 5:25 PM |
This check flapped a few times yesterday. I will dig in deeper. My fear is that we have a new set of aggressive scraper bots hitting the Beta Cluster wikis.
Mentioned in SAL (#wikimedia-releng) [2025-08-21T16:29:08Z] <bd808> deployment-cache-text08 hard reboot via horizon (T402557)
bd808@deployment-cache-text08.deployment-prep.eqiad1:~$ sudo -i puppet agent -tv Info: Using environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for deployment-cache-text08.deployment-prep.eqiad1.wikimedia.cloud Info: Applying configuration version '(5b381ba7bb) gitpuppet - varnish: Implement new direct routing for mobile views' Notice: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]/ensure: ensure changed 'stopped' to 'running' (corrective) Info: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]: Unscheduling refresh on Service[prometheus-varnishkafka-exporter] Error: Systemd start for varnishmtail@default failed! journalctl log for varnishmtail@default: -- Journal begins at Tue 2025-08-19 14:48:25 UTC, ends at Thu 2025-08-21 16:34:19 UTC. -- Aug 21 16:31:05 deployment-cache-text08 systemd[1]: Dependency failed for "Varnish mtail default instance". Aug 21 16:31:05 deployment-cache-text08 systemd[1]: varnishmtail@default.service: Job varnishmtail@default.service/start failed with result 'dependency'. Aug 21 16:34:19 deployment-cache-text08 systemd[1]: Dependency failed for "Varnish mtail default instance". Aug 21 16:34:19 deployment-cache-text08 systemd[1]: varnishmtail@default.service: Job varnishmtail@default.service/start failed with result 'dependency'. Error: /Stage[main]/Varnish::Logging/Varnish::Logging::Mtail[default]/Systemd::Service[varnishmtail@default]/Service[varnishmtail@default]/ensure: change from 'stopped' to 'running' failed: Systemd start for varnishmtail@default failed! journalctl log for varnishmtail@default: -- Journal begins at Tue 2025-08-19 14:48:25 UTC, ends at Thu 2025-08-21 16:34:19 UTC. -- Aug 21 16:31:05 deployment-cache-text08 systemd[1]: Dependency failed for "Varnish mtail default instance". Aug 21 16:31:05 deployment-cache-text08 systemd[1]: varnishmtail@default.service: Job varnishmtail@default.service/start failed with result 'dependency'. Aug 21 16:34:19 deployment-cache-text08 systemd[1]: Dependency failed for "Varnish mtail default instance". Aug 21 16:34:19 deployment-cache-text08 systemd[1]: varnishmtail@default.service: Job varnishmtail@default.service/start failed with result 'dependency'. (corrective) Error: Systemd start for varnishmtail@internal failed! journalctl log for varnishmtail@internal: -- Journal begins at Tue 2025-08-19 14:48:25 UTC, ends at Thu 2025-08-21 16:34:23 UTC. -- Aug 21 16:31:08 deployment-cache-text08 systemd[1]: Dependency failed for "Varnish mtail internal instance". Aug 21 16:31:08 deployment-cache-text08 systemd[1]: varnishmtail@internal.service: Job varnishmtail@internal.service/start failed with result 'dependency'. Aug 21 16:34:23 deployment-cache-text08 systemd[1]: Dependency failed for "Varnish mtail internal instance". Aug 21 16:34:23 deployment-cache-text08 systemd[1]: varnishmtail@internal.service: Job varnishmtail@internal.service/start failed with result 'dependency'. Error: /Stage[main]/Varnish::Logging/Varnish::Logging::Mtail[internal]/Systemd::Service[varnishmtail@internal]/Service[varnishmtail@internal]/ensure: change from 'stopped' to 'running' failed: Systemd start for varnishmtail@internal failed! journalctl log for varnishmtail@internal: -- Journal begins at Tue 2025-08-19 14:48:25 UTC, ends at Thu 2025-08-21 16:34:23 UTC. -- Aug 21 16:31:08 deployment-cache-text08 systemd[1]: Dependency failed for "Varnish mtail internal instance". Aug 21 16:31:08 deployment-cache-text08 systemd[1]: varnishmtail@internal.service: Job varnishmtail@internal.service/start failed with result 'dependency'. Aug 21 16:34:23 deployment-cache-text08 systemd[1]: Dependency failed for "Varnish mtail internal instance". Aug 21 16:34:23 deployment-cache-text08 systemd[1]: varnishmtail@internal.service: Job varnishmtail@internal.service/start failed with result 'dependency'. (corrective) Error: Systemd start for varnish-frontend failed! journalctl log for varnish-frontend: -- Journal begins at Tue 2025-08-19 14:48:25 UTC, ends at Thu 2025-08-21 16:34:27 UTC. -- Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55708 127.0.0.1 6082 Rd auth d3b79f39e33659d8bb8503d9654d7ed45f08ddf56adc8475eecf5d14cec6fbb6 Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55708 127.0.0.1 6082 Wr 200 ----------------------------- Varnish Cache CLI 1.0 ----------------------------- Linux,6.1.0-0.deb11.21-cloud-amd64,x86_64,-junix,-smalloc,-sdefault,-hcritbit varnish-7.1.1 revision 7cee1c581bead20e88d101ab3d72afb29f14d87a Type 'help' for command list. Type 'quit' to close CLI session. Type 'start' to launch worker process. Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55708 127.0.0.1 6082 Rd ping Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55708 127.0.0.1 6082 Wr 200 PONG 1755793768 1.0 Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55708 127.0.0.1 6082 Rd vcl.label wikimedia_misc vcl-6f137cd5-5f82-4fad-8273-00b00f2ac87a Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55708 127.0.0.1 6082 Wr 200 Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55716 127.0.0.1 6082 Rd auth 0f5c99cbf263ff699c337bafd475867be1c2561afe359d08860bc3a8ad92735d Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55716 127.0.0.1 6082 Wr 200 ----------------------------- Varnish Cache CLI 1.0 ----------------------------- Linux,6.1.0-0.deb11.21-cloud-amd64,x86_64,-junix,-smalloc,-sdefault,-hcritbit varnish-7.1.1 revision 7cee1c581bead20e88d101ab3d72afb29f14d87a Type 'help' for command list. Type 'quit' to close CLI session. Type 'start' to launch worker process. Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55716 127.0.0.1 6082 Rd ping Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55716 127.0.0.1 6082 Wr 200 PONG 1755793768 1.0 Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55716 127.0.0.1 6082 Rd vcl.state vcl-6f137cd5-5f82-4fad-8273-00b00f2ac87a warm Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55716 127.0.0.1 6082 Wr 200 Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55724 127.0.0.1 6082 Rd auth db8dc16fe2d5b10a5803c130f7779511f21eff8bd6bd26aaf4fb2060ce8bdff5 Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55724 127.0.0.1 6082 Wr 200 ----------------------------- Varnish Cache CLI 1.0 ----------------------------- Linux,6.1.0-0.deb11.21-cloud-amd64,x86_64,-junix,-smalloc,-sdefault,-hcritbit varnish-7.1.1 revision 7cee1c581bead20e88d101ab3d72afb29f14d87a Type 'help' for command list. Type 'quit' to close CLI session. Type 'start' to launch worker process. Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55724 127.0.0.1 6082 Rd ping Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55724 127.0.0.1 6082 Wr 200 PONG 1755793768 1.0 Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55724 127.0.0.1 6082 Rd vcl.load vcl-1690c028-9195-469d-a57e-960200f7f806 /etc/varnish/wikimedia_text-frontend.vcl Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55724 127.0.0.1 6082 Wr 106 Message from VCC-compiler: Expected ')' got '{' (program line 70), at ('/etc/varnish/text-frontend.inc.vcl' Line 183 Pos 9) { --------# Running VCC-compiler failed, exited with 2 VCL compilation failed Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[883]: Message from VCC-compiler: Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[883]: Expected ')' got '{' Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[883]: (program line 70), at Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[883]: ('/etc/varnish/text-frontend.inc.vcl' Line 183 Pos 9) Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[883]: { Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[883]: --------# Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[883]: Running VCC-compiler failed, exited with 2 Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[883]: VCL compilation failed Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[883]: Command failed with error code 106 Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: Executing: "/usr/bin/varnishadm -n frontend vcl.load vcl-6f137cd5-5f82-4fad-8273-00b00f2ac87a /etc/varnish/wikimedia_misc-frontend.vcl" Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: Executing: "/usr/bin/varnishadm -n frontend vcl.label wikimedia_misc vcl-6f137cd5-5f82-4fad-8273-00b00f2ac87a" Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: Executing: "/usr/bin/varnishadm -n frontend vcl.state vcl-6f137cd5-5f82-4fad-8273-00b00f2ac87a warm" Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: Executing: "/usr/bin/varnishadm -n frontend vcl.load vcl-1690c028-9195-469d-a57e-960200f7f806 /etc/varnish/wikimedia_text-frontend.vcl" Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: Traceback (most recent call last): Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: File "/usr/local/sbin/reload-vcl", line 205, in <module> Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: main() Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: File "/usr/local/sbin/reload-vcl", line 188, in main Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: main_vcl_id = load(vadm_cmd, args.vcl_file) Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: File "/usr/local/sbin/reload-vcl", line 149, in load Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: do_cmd(vcl_load_cmd) Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: File "/usr/local/sbin/reload-vcl", line 64, in do_cmd Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: subprocess.check_call(cmd) Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: File "/usr/lib/python3.9/subprocess.py", line 373, in check_call Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: raise CalledProcessError(retcode, cmd) Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: subprocess.CalledProcessError: Command '['/usr/bin/varnishadm', '-n', 'frontend', 'vcl.load', 'vcl-1690c028-9195-469d-a57e-960200f7f806', '/etc/varnish/wikimedia_text-frontend.vcl']' returned non-zero exit status 1. Aug 21 16:29:28 deployment-cache-text08 systemd[1]: varnish-frontend.service: Control process exited, code=exited, status=1/FAILURE Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: Manager got SIGTERM Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: manager stopping child Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: manager dies Aug 21 16:29:28 deployment-cache-text08 systemd[1]: varnish-frontend.service: Failed with result 'exit-code'. Aug 21 16:29:28 deployment-cache-text08 systemd[1]: Failed to start varnish-frontend (Varnish HTTP Accelerator). Aug 21 16:29:28 deployment-cache-text08 systemd[1]: varnish-frontend.service: Scheduled restart job, restart counter is at 6. Error: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Systemd::Service[varnish-frontend]/Service[varnish-frontend]/ensure: change from 'stopped' to 'running' failed: Systemd start for varnish-frontend failed! journalctl log for varnish-frontend: -- Journal begins at Tue 2025-08-19 14:48:25 UTC, ends at Thu 2025-08-21 16:34:27 UTC. -- Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55708 127.0.0.1 6082 Rd auth d3b79f39e33659d8bb8503d9654d7ed45f08ddf56adc8475eecf5d14cec6fbb6 Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55708 127.0.0.1 6082 Wr 200 ----------------------------- Varnish Cache CLI 1.0 ----------------------------- Linux,6.1.0-0.deb11.21-cloud-amd64,x86_64,-junix,-smalloc,-sdefault,-hcritbit varnish-7.1.1 revision 7cee1c581bead20e88d101ab3d72afb29f14d87a Type 'help' for command list. Type 'quit' to close CLI session. Type 'start' to launch worker process. Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55708 127.0.0.1 6082 Rd ping Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55708 127.0.0.1 6082 Wr 200 PONG 1755793768 1.0 Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55708 127.0.0.1 6082 Rd vcl.label wikimedia_misc vcl-6f137cd5-5f82-4fad-8273-00b00f2ac87a Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55708 127.0.0.1 6082 Wr 200 Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55716 127.0.0.1 6082 Rd auth 0f5c99cbf263ff699c337bafd475867be1c2561afe359d08860bc3a8ad92735d Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55716 127.0.0.1 6082 Wr 200 ----------------------------- Varnish Cache CLI 1.0 ----------------------------- Linux,6.1.0-0.deb11.21-cloud-amd64,x86_64,-junix,-smalloc,-sdefault,-hcritbit varnish-7.1.1 revision 7cee1c581bead20e88d101ab3d72afb29f14d87a Type 'help' for command list. Type 'quit' to close CLI session. Type 'start' to launch worker process. Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55716 127.0.0.1 6082 Rd ping Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55716 127.0.0.1 6082 Wr 200 PONG 1755793768 1.0 Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55716 127.0.0.1 6082 Rd vcl.state vcl-6f137cd5-5f82-4fad-8273-00b00f2ac87a warm Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55716 127.0.0.1 6082 Wr 200 Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55724 127.0.0.1 6082 Rd auth db8dc16fe2d5b10a5803c130f7779511f21eff8bd6bd26aaf4fb2060ce8bdff5 Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55724 127.0.0.1 6082 Wr 200 ----------------------------- Varnish Cache CLI 1.0 ----------------------------- Linux,6.1.0-0.deb11.21-cloud-amd64,x86_64,-junix,-smalloc,-sdefault,-hcritbit varnish-7.1.1 revision 7cee1c581bead20e88d101ab3d72afb29f14d87a Type 'help' for command list. Type 'quit' to close CLI session. Type 'start' to launch worker process. Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55724 127.0.0.1 6082 Rd ping Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55724 127.0.0.1 6082 Wr 200 PONG 1755793768 1.0 Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55724 127.0.0.1 6082 Rd vcl.load vcl-1690c028-9195-469d-a57e-960200f7f806 /etc/varnish/wikimedia_text-frontend.vcl Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: CLI telnet 127.0.0.1 55724 127.0.0.1 6082 Wr 106 Message from VCC-compiler: Expected ')' got '{' (program line 70), at ('/etc/varnish/text-frontend.inc.vcl' Line 183 Pos 9) { --------# Running VCC-compiler failed, exited with 2 VCL compilation failed Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[883]: Message from VCC-compiler: Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[883]: Expected ')' got '{' Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[883]: (program line 70), at Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[883]: ('/etc/varnish/text-frontend.inc.vcl' Line 183 Pos 9) Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[883]: { Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[883]: --------# Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[883]: Running VCC-compiler failed, exited with 2 Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[883]: VCL compilation failed Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[883]: Command failed with error code 106 Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: Executing: "/usr/bin/varnishadm -n frontend vcl.load vcl-6f137cd5-5f82-4fad-8273-00b00f2ac87a /etc/varnish/wikimedia_misc-frontend.vcl" Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: Executing: "/usr/bin/varnishadm -n frontend vcl.label wikimedia_misc vcl-6f137cd5-5f82-4fad-8273-00b00f2ac87a" Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: Executing: "/usr/bin/varnishadm -n frontend vcl.state vcl-6f137cd5-5f82-4fad-8273-00b00f2ac87a warm" Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: Executing: "/usr/bin/varnishadm -n frontend vcl.load vcl-1690c028-9195-469d-a57e-960200f7f806 /etc/varnish/wikimedia_text-frontend.vcl" Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: Traceback (most recent call last): Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: File "/usr/local/sbin/reload-vcl", line 205, in <module> Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: main() Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: File "/usr/local/sbin/reload-vcl", line 188, in main Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: main_vcl_id = load(vadm_cmd, args.vcl_file) Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: File "/usr/local/sbin/reload-vcl", line 149, in load Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: do_cmd(vcl_load_cmd) Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: File "/usr/local/sbin/reload-vcl", line 64, in do_cmd Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: subprocess.check_call(cmd) Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: File "/usr/lib/python3.9/subprocess.py", line 373, in check_call Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: raise CalledProcessError(retcode, cmd) Aug 21 16:29:28 deployment-cache-text08 varnish-frontend[872]: subprocess.CalledProcessError: Command '['/usr/bin/varnishadm', '-n', 'frontend', 'vcl.load', 'vcl-1690c028-9195-469d-a57e-960200f7f806', '/etc/varnish/wikimedia_text-frontend.vcl']' returned non-zero exit status 1. Aug 21 16:29:28 deployment-cache-text08 systemd[1]: varnish-frontend.service: Control process exited, code=exited, status=1/FAILURE Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: Manager got SIGTERM Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: manager stopping child Aug 21 16:29:28 deployment-cache-text08 varnishd[867]: manager dies Aug 21 16:29:28 deployment-cache-text08 systemd[1]: varnish-frontend.service: Failed with result 'exit-code'. Aug 21 16:29:28 deployment-cache-text08 systemd[1]: Failed to start varnish-frontend (Varnish HTTP Accelerator). Aug 21 16:29:28 deployment-cache-text08 systemd[1]: varnish-frontend.service: Scheduled restart job, restart counter is at 6. (corrective) Notice: /Stage[main]/Profile::Cache::Kafka::Webrequest/Varnishkafka::Instance[webrequest]/File[/etc/varnishkafka/webrequest.conf]: Dependency Service[varnish-frontend] has failures: true Warning: /Stage[main]/Profile::Cache::Kafka::Webrequest/Varnishkafka::Instance[webrequest]/File[/etc/varnishkafka/webrequest.conf]: Skipping because of failed dependencies Warning: /Stage[main]/Profile::Cache::Kafka::Webrequest/Varnishkafka::Instance[webrequest]/File[/etc/logrotate.d/varnishkafka-webrequest-stats]: Skipping because of failed dependencies Warning: /Stage[main]/Varnish::Logging/Systemd::Service[varnish-frontend-tlsinspector]/Service[varnish-frontend-tlsinspector]: Skipping because of failed dependencies Warning: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[load-new-vcl-file-frontend]: Skipping because of failed dependencies Warning: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]: Skipping because of failed dependencies Warning: /Stage[main]/Profile::Cache::Kafka::Statsv/Varnishkafka::Instance[statsv]/File[/etc/varnishkafka/statsv.conf]: Skipping because of failed dependencies Warning: /Stage[main]/Profile::Cache::Kafka::Statsv/Varnishkafka::Instance[statsv]/File[/etc/logrotate.d/varnishkafka-statsv-stats]: Skipping because of failed dependencies Warning: /Stage[main]/Profile::Cache::Kafka::Webrequest/Varnishkafka::Instance[webrequest]/Base::Service_unit[varnishkafka-webrequest]/File[/lib/systemd/system/varnishkafka-webrequest.service]: Skipping because of failed dependencies Warning: /Stage[main]/Profile::Cache::Kafka::Webrequest/Varnishkafka::Instance[webrequest]/Base::Service_unit[varnishkafka-webrequest]/Exec[systemd reload for varnishkafka-webrequest]: Skipping because of failed dependencies Warning: /Stage[main]/Profile::Cache::Kafka::Webrequest/Varnishkafka::Instance[webrequest]/Base::Service_unit[varnishkafka-webrequest]/Service[varnishkafka-webrequest]: Skipping because of failed dependencies Warning: /Stage[main]/Varnish::Logging/Systemd::Service[varnish-frontend-tlsinspector]/Systemd::Unit[varnish-frontend-tlsinspector]/Exec[systemd daemon-reload for varnish-frontend-tlsinspector.service (varnish-frontend-tlsinspector)]: Skipping because of failed dependencies Warning: /Stage[main]/Profile::Cache::Kafka::Statsv/Varnishkafka::Instance[statsv]/Base::Service_unit[varnishkafka-statsv]/File[/lib/systemd/system/varnishkafka-statsv.service]: Skipping because of failed dependencies Warning: /Stage[main]/Profile::Cache::Kafka::Statsv/Varnishkafka::Instance[statsv]/Base::Service_unit[varnishkafka-statsv]/Exec[systemd reload for varnishkafka-statsv]: Skipping because of failed dependencies Warning: /Stage[main]/Profile::Cache::Kafka::Statsv/Varnishkafka::Instance[statsv]/Base::Service_unit[varnishkafka-statsv]/Service[varnishkafka-statsv]: Skipping because of failed dependencies Warning: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Systemd::Service[varnish-frontend-slowlog]/Service[varnish-frontend-slowlog]: Skipping because of failed dependencies Warning: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Systemd::Service[varnish-frontend-hospital]/Service[varnish-frontend-hospital]: Skipping because of failed dependencies Warning: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Systemd::Service[varnish-frontend-fetcherr]/Service[varnish-frontend-fetcherr]: Skipping because of failed dependencies Error: Systemd start for prometheus-varnish-exporter@frontend failed! journalctl log for prometheus-varnish-exporter@frontend: -- Journal begins at Tue 2025-08-19 14:48:25 UTC, ends at Thu 2025-08-21 16:34:30 UTC. -- Aug 21 16:29:32 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:29:32 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:29:35 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:29:35 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:29:39 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:29:39 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:29:42 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:29:42 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:29:46 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:29:46 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:29:49 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:29:49 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:29:53 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:29:53 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:29:56 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:29:56 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:00 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:00 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:03 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:03 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:07 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:07 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:11 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:11 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:15 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:15 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:18 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:18 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:22 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:22 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:26 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:26 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:29 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:29 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:33 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:33 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:36 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:36 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:40 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:40 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:44 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:44 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:47 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:47 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:51 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:51 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:54 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:54 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:58 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:58 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Error: /Stage[main]/Profile::Prometheus::Varnish_exporter/Prometheus::Varnish_exporter[frontend]/Systemd::Service[prometheus-varnish-exporter@frontend]/Service[prometheus-varnish-exporter@frontend]/ensure: change from 'stopped' to 'running' failed: Systemd start for prometheus-varnish-exporter@frontend failed! journalctl log for prometheus-varnish-exporter@frontend: -- Journal begins at Tue 2025-08-19 14:48:25 UTC, ends at Thu 2025-08-21 16:34:30 UTC. -- Aug 21 16:29:32 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:29:32 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:29:35 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:29:35 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:29:39 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:29:39 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:29:42 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:29:42 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:29:46 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:29:46 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:29:49 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:29:49 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:29:53 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:29:53 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:29:56 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:29:56 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:00 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:00 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:03 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:03 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:07 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:07 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:11 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:11 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:15 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:15 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:18 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:18 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:22 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:22 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:26 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:26 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:29 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:29 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:33 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:33 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:36 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:36 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:40 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:40 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:44 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:44 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:47 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:47 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:51 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:51 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:54 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:54 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. Aug 21 16:30:58 deployment-cache-text08 systemd[1]: Dependency failed for Prometheus exporter for Varnish (instance frontend). Aug 21 16:30:58 deployment-cache-text08 systemd[1]: prometheus-varnish-exporter@frontend.service: Job prometheus-varnish-exporter@frontend.service/start failed with result 'dependency'. (corrective) Info: Stage[main]: Unscheduling all events on Stage[main] Notice: Applied catalog in 25.13 seconds
Puppet is generating bad VCL in /etc/varnish/text-frontend.inc.vcl. I hacked it to compile and the next puppet run broke it again with this diff:
--- /etc/varnish/text-frontend.inc.vcl 2025-08-21 16:40:03.038694218 +0000 +++ /tmp/puppet-file20250821-7983-1l7wbef 2025-08-21 16:40:49.226782174 +0000 @@ -180,7 +180,7 @@ // These GET requests are POST-like in nature and respond by setting the // stopMobileRedirect/mf_useformat cookies we check above, which disables this redirect. && req.url !~ "[?&]mobileaction=toggle_view_desktop(&|$)" - ){ + { // Separate regexps for clarity, but multiple regsubs instead of // "if host ~"/regsub matches for efficiency. Be careful to not
Mentioned in SAL (#wikimedia-releng) [2025-08-21T16:57:05Z] <bd808> Removed cherry-picks of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1180577, https://gerrit.wikimedia.org/r/c/operations/puppet/+/1180220, https://gerrit.wikimedia.org/r/c/operations/puppet/+/1180166 (T402557)
bd808@deployment-cache-text08.deployment-prep.eqiad1:~$ sudo -i puppet agent -tv Info: Using environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for deployment-cache-text08.deployment-prep.eqiad1.wikimedia.cloud Info: Applying configuration version '(c2488790b6) gitpuppet - varnish: Implement new direct routing for mobile views' Notice: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]/ensure: ensure changed 'stopped' to 'running' (corrective) Info: /Stage[main]/Prometheus::Varnishkafka_exporter/Service[prometheus-varnishkafka-exporter]: Unscheduling refresh on Service[prometheus-varnishkafka-exporter] Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Message from VCC-compiler: Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Unused sub browser_sec_synth, defined: Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: ('/etc/varnish/browsersec.inc.vcl' Line 1 Pos 5) Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: sub browser_sec_synth { Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: ----#################-- Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: (That was just a warning) Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Unused sub geoip_cookie, defined: Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: ('/etc/varnish/geoip.inc.vcl' Line 362 Pos 5) Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: sub geoip_cookie { Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: ----############-- Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: (That was just a warning) Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Unused acl wikimedia_trust, defined: Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: ('/etc/varnish/wikimedia_misc-frontend.vcl' Line 64 Pos 5) Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: acl wikimedia_trust { Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: ----###############-- Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: (That was just a warning) Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Message from VCC-compiler: Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Expected ')' got '{' Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: (program line 70), at Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: ('/etc/varnish/text-frontend.inc.vcl' Line 183 Pos 9) Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: { Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: --------# Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Running VCC-compiler failed, exited with 2 Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: VCL compilation failed Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Command failed with error code 106 Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Executing: "/usr/bin/varnishadm -n frontend vcl.load vcl-95cdc579-d5d2-43a7-a831-614aab01d90f /etc/varnish/wikimedia_misc-frontend.vcl" Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Executing: "/usr/bin/varnishadm -n frontend vcl.label wikimedia_misc vcl-95cdc579-d5d2-43a7-a831-614aab01d90f" Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Executing: "/usr/bin/varnishadm -n frontend vcl.state vcl-95cdc579-d5d2-43a7-a831-614aab01d90f warm" Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Executing: "/usr/bin/varnishadm -n frontend vcl.load vcl-c2d993fa-b06b-4b93-8dc1-627267ce0f81 /etc/varnish/wikimedia_text-frontend.vcl" Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Traceback (most recent call last): Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: File "/usr/local/sbin/reload-vcl", line 205, in <module> Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: main() Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: File "/usr/local/sbin/reload-vcl", line 188, in main Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: main_vcl_id = load(vadm_cmd, args.vcl_file) Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: File "/usr/local/sbin/reload-vcl", line 149, in load Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: do_cmd(vcl_load_cmd) Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: File "/usr/local/sbin/reload-vcl", line 64, in do_cmd Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: subprocess.check_call(cmd) Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: File "/usr/lib/python3.9/subprocess.py", line 373, in check_call Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: raise CalledProcessError(retcode, cmd) Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: subprocess.CalledProcessError: Command '['/usr/bin/varnishadm', '-n', 'frontend', 'vcl.load', 'vcl-c2d993fa-b06b-4b93-8dc1-627267ce0f81', '/etc/varnish/wikimedia_text-frontend.vcl']' returned non-zero exit status 1. Error: '/usr/local/sbin/reload-vcl -n frontend -f /etc/varnish/wikimedia_text-frontend.vcl -d 2 -a -s /etc/varnish/wikimedia_misc-frontend.vcl && (rm /var/tmp/reload-vcl-failed-frontend; true)' returned 1 instead of one of [0] Error: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: change from 'notrun' to ['0'] failed: '/usr/local/sbin/reload-vcl -n frontend -f /etc/varnish/wikimedia_text-frontend.vcl -d 2 -a -s /etc/varnish/wikimedia_misc-frontend.vcl && (rm /var/tmp/reload-vcl-failed-frontend; true)' returned 1 instead of one of [0] (corrective) Info: Stage[main]: Unscheduling all events on Stage[main] Notice: Applied catalog in 18.10 seconds
Mentioned in SAL (#wikimedia-releng) [2025-08-21T17:05:25Z] <bd808> sudo /usr/local/bin/puppetserver-deploy-code on deployment-puppetserver-1 (T402557)
This seems to have been the fix for the VCL compilation problems. @Krinkle was working on T401595: [Rollout Phase 1] Implement redirect-less mobile routing and enable for wikitech.wikimedia.org and managed to get a Puppet patch applied that has some bad VCL in it. For reasons that are currently unclear to me the git rebase --interactive origin/production I did on deployment-puppetserver-1 to remove the experimental patch chain did not fire sudo /usr/local/bin/puppetserver-deploy-code on completion via a git hook as I expected. This meant that the Puppetserver had not seen the manifests change yet; that in turn caused the puppet run in T402557#11107729 to remain broken. A manual run of sudo /usr/local/bin/puppetserver-deploy-code got things back to the desired state and then Puppet fixed things as expected on deployment-cache-text08.
I missed this error because whenever we execute run-puppet-agent on deployment-cache-text in Beta, after a Puppet patch that changes VCL files, it runs out of memory and then the process stalls / times out.
It seems since a few weeks (?) the machine runs out of memory when trying to reload a VCL change. The same happened a few days ago with a change that did not involve a syntax error: T402355: Project deployment-prep instance deployment-cache-text08 is down
Grafana WMCS: deployment-cache-text08
The alert about the host being down, then, is not about the syntax error but about the machine being unresponsive (no more metrics flushed to Prometheus).
Back to today's change, it was not until one or two retries later that the syntax error showed up. I hadn't yet looked at the output of the re-run as I was writing the next patch in line. Please correct me if I'm wrong but afaik Beta remained up (after the memory hog cleared from me killing the stuck process of the first run), but running old config. Thanks for catching the syntax error!
Mentioned in SAL (#wikimedia-releng) [2025-08-21T17:34:01Z] <bd808> Resize deployment-cache-text08 from g4.cores2.ram4.disk20 -> g4.cores4.ram8.disk20 (T402557)
Change #1180919 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/puppet@production] trafficserver: Check x-dt-host in rb-mw-mangling.lua before using