@Dzahn The error seems to have resolved. I think maybe someone altered the setting and made it valid. I'll resolve this. I just saw 6.5.18 was released today. :)

What's the new error?

Upgrade is complete though there is a new error related to one of the settings. Will close this once a resolution to the issue has been found.

Cookbook cookbooks.sre.vrts.upgrade was started by aokoth@cumin1003 on VRTS host vrts1003.eqiad.wmnet

The original scope of this task was based on outdated assumptions. While the exporter can be used for operational metrics, we don't plan on replicating the historical reports so I'm closing this task as declined.

Our current setup trains spamassassin by exporting to mails to the mbox format. This format is not supported by rspam. I'm considering how to address this issue.

Change #1178874 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] vrts: Create test role and profile

https://gerrit.wikimedia.org/r/1178874

In T378028#10995390, @Arnoldokoth wrote:

Thanks @MoritzMuehlenhoff We'll consider that... But I'm doubtful we "strictly" need to test this on hardware. A VM will just work as well me thinks. @jhathaway is out on vacation so we can wait and hear is thoughts when he's back.

Thanks all. I am not sure though if the request was for "temp testing setup" or just for "a new system to replace the old".

Thanks @MoritzMuehlenhoff We'll consider that... But I'm doubtful we "strictly" need to test this on hardware. A VM will just work as well me thinks. @jhathaway is out on vacation so we can wait and hear is thoughts when he's back.

That said, if you striclty need a physical host for the tests, you could use puppetserver2003. I decommed it today in https://phabricator.wikimedia.org/T398607 and you could reclaim it. It has 128G RAM.

@Dzahn We used to run it on VMs but we kept running into resource issues (especially with clamav) even after resizing the VM a few times. Additionally, it was also suggested that we could migrate the attachments to the disk though this is still on ice as we wait for object storage support in newer Znuny versions and compare (last I checked this was still in development).

In T378028#10993697, @Dzahn wrote:

But another question comes to mind.. and that is.. do VRTS machines really need to be physical in general (anymore)?

Well... or we could create a VM and try to install VRTS with postfix on that. If that works (where I'm not sure how to actually test yet) it would be both the test for postfix and also a test if VRTS can just run on VMs. If it can.. we could flip over to that and get 2 physical servers back (maybe as our spares for other services).

@Arnoldokoth If there is a spare server, sure, but I am not sure there is one. Back in the days dcops had a spare pool but at some point this changed and not sure now. We would have to ask them if they have any.

@Dzahn Or we could repurpose a spare server (if available)? miscweb comes to mind... Or were those VMs?

Hosts are not virtual, they are physical machines. So the biggest issue with that would be where to get hardware from (without a hardware request and the time/budget at that takes at least).

would it be possible to setup a separate vrts server, that is configured with postfix, rather than replacing exim on the current server?

Change #1140207 merged by AOkoth:

[operations/puppet@production] vrts: add junk queue count and remove mobile queue

https://gerrit.wikimedia.org/r/1140207

Change #1140207 had a related patch set uploaded (by Dzahn; author: AOkoth):

[operations/puppet@production] vrts: add junk queue count and remove mobile queue

https://gerrit.wikimedia.org/r/1140207