Did this actually end up being caused by the train? Digging in the server admin log/reading random backscroll in random places, I couldn't tell if something on the train needed to be fixed or if this was caused by a different issue.

In T371650#10037668, @Fabfur wrote:

Hi @NBaca-WMF @thcipriani, could you confirm this request please?

@LucasWerkmeister is this mirroring a work around to keep your fork up-to-date with the canonical repo?

Reason for access makes sense. Approved from my side.

For now, Catalyst folks are focused on the switchover of patchdemo to use the catalyst back end. For that, we have T370078: Add logging for catalyst WMCS k8s that should cover immediate logging needs. We may want to go this route in future, setting as low since we won't be doing this in the very near future.

I think the update to from git v2.20.1 to v.2.30.2 explains this. this commit may be what changed to make @jnuche 's fix needed.

Haven't seen this in a while, but the linked task fix rolled out in 16.3.

Now that we have https://gitlab.wikimedia.org/repos/releng/gitlab-settings running on a schedule, this mostly solves globally disabling things. Talked to KDE folks about this, too. Seems like they have a similar project using the API to disable things on a per-repo basis.

This is an interesting problem. It seems that the trusted runners should have picked up the job.

In T370664#10003514, @matmarex wrote:

This is for the instance at https://patchdemo.catalyst.wmcloud.org, right?

We had some discussion via chat about this, in particular a couple concerns with templates:

• Escaping input within the templates
• syntax highlighting within the script (naming things script.sh.tpl works with vscode, this varies by editor tho)

Thanks for that tweak @matmarex

In T355882#9535275, @kostajh wrote:

In T355882#9491485, @thcipriani wrote:

Most of these questions are in our Risky Change Template. The week something risky is going out, it's a nice template to use to give us a heads up, most of the time that suffices. This change seems pretty fundamental, so we might need more. But let's start with there and see.

That sounds good.

Overall, I think the important thing for Release Engineering and specifically train drivers to know would be, after testwiki rollout:

• testwiki would reflect an editing paradigm (temporary accounts) that is *not* the same as all other wikis. So, during that time it would probably make sense to review logs from other group0 wikis more closely. And WMF teams should probably do any anonymous editing related QA on test2.wikipedia.org

@thcipriani https://www.mediawiki.org/wiki/Help:Temporary_accounts/How_it_works is also a useful reference, that is probably a good idea for RelEng folks to read before we enable this on testwiki.

In T366959#9995535, @hubaishan wrote:

In T366959#9995482, @Pppery wrote:

That's not how train deployments work. If there's a specific patch you don't want to roll out to all wikis you could potentially revert and backport that one patch. But we won't keep one wiki on a legacy version because unspecified bots are slow (that was whoever wrote the patch's responsibility before putting it on the train).

Can tody changes in Arabic translation included in the version?

In T366742#9981166, @Lucas_Werkmeister_WMDE wrote:

Actually, it looks like this merge includes the fix, so IIUC that will roll out with this week’s train.

I still see these happening in production on wikidatawiki. Seems to be happening on PATCH requests to the wikibase rest API.

In T370110#9987108, @bd808 wrote:

Using the UnsafeAllow3F flag on the existing /index.php?... rule seems like a reasonable fix to me:

RewriteRule ^(.*)$          /index.php?__path__=$1  [B,L,QSA,UnsafeAllow3F]

We can see in the logging from T370110#9983690 that the url encoded %3F values are passing through as expected and the ? that is triggering the error is actually the one we explicitly are adding to support the front router setup of Phorge's index.php entry point.

In T370110#9983601, @bd808 wrote:

@bd808: The RewriteRule ^(.*)$ /index.php?__path__=$1 [B,L,QSA] rule at https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/phabricator/templates/phabricator-default.conf.erb#75 is going to trigger on this. Not sure what else.

This actually might not be the trigger point? The B flag here should keep the %3F in the inbound URL from being expanded. As I understand the bug & fix Apache is now checking to see if there was a %3F in the inbound URL and a ? in the redirect target and refusing to continue when this is the case.

In T370110#9983673, @brennen wrote:

If we turn up the LogLevel on a Phab node to trace2 I think we should see that before/after and maybe figure out which rule should really have a B flag added.

Could use devtools' https://phabricator.wmcloud.org/ for testing here.

In T360784#9981467, @Andrew wrote:

Hello, @hashar! The deadline for this rebuild is today :)

Likely you can just replace this VM with an identically-puppetized Bullseye host. I'm doing the same in deployment-prep, although in that case I'm blocked by not having the keyholder passphrase.

The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!

Nice work digging :)

In T368280#9977648, @SDunlap wrote:

Suggestions (in order of my recommendations):

1. host an ELK stack on our cluster
2. write logs to files on a persisted volume
3. log on the Wikimedia Observability experimental CloudVPS logstash
4. continue to rely on Kubernetes logs only Ostrich Algorithm

Getting database free disk space in k8s may require more effort than I think it's worth at this point. Let's just drop that information from the front end and if we miss it, we can always explore options after that. As long as we have disk space monitoring in some other form, I don't think we'll miss it in the front-end of the application.

Thanks for documenting this, ran into the same thing in deployment prep (T327742), reboot also fixed it there.