Page MenuHomePhabricator
Create Task
Maniphest T210621

Internal api error: CannotCreateActorException
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

It is almost impossible today to close deletion request, i am getting tons of such errors:

API request failed (internal_api_error_CannotCreateActorException): [W-66MwpAAE0AAAVLeqgAAACD] Caught exception of type CannotCreateActorException
trace
#0 /srv/mediawiki/php-1.33.0-wmf.6/includes/ActorMigration.php(224): User->getActorId(Wikimedia\Rdbms\DatabaseMysqli)
#1 /srv/mediawiki/php-1.33.0-wmf.6/includes/page/WikiPage.php(2902): ActorMigration->getInsertValues(Wikimedia\Rdbms\DatabaseMysqli, string, User)
#2 /srv/mediawiki/php-1.33.0-wmf.6/includes/page/WikiPage.php(2681): WikiPage->archiveRevisions(Wikimedia\Rdbms\DatabaseMysqli, integer, boolean)
#3 /srv/mediawiki/php-1.33.0-wmf.6/includes/page/WikiPage.php(2624): WikiPage->doDeleteArticleBatched(string, boolean, User, NULL, string, boolean)
#4 /srv/mediawiki/php-1.33.0-wmf.6/includes/FileDeleteForm.php(198): WikiPage->doDeleteArticleReal(string, boolean, integer, boolean, string, User, NULL)
#5 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiDelete.php(171): FileDeleteForm::doDelete(Title, LocalFile, NULL, string, boolean, User, NULL)
#6 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiDelete.php(73): ApiDelete::deleteFile(WikiFilePage, User, NULL, string, boolean, NULL)
#7 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiMain.php(1576): ApiDelete->execute()
#8 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiMain.php(531): ApiMain->executeAction()
#9 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiMain.php(502): ApiMain->executeActionWithErrorHandling()
#10 /srv/mediawiki/php-1.33.0-wmf.6/api.php(87): ApiMain->execute()
#11 /srv/mediawiki/w/api.php(3): include(string)
#12 {main}

Logstash query: message:"Cannot create actor ID for user_id"

Over 30 days:

We had a few occurrences in November. it started spamming heavily 2018-11-26 around 15:20 UTC

Details

SubjectRepoBranchLines +/-
User: Bypass repeatable-read when creating an actor_idmediawiki/coreREL1_31+9 -2
User: Bypass repeatable-read when creating an actor_idmediawiki/coreREL1_32+9 -2
User: Bypass repeatable-read when creating an actor_idmediawiki/coremaster+8 -2
User: Bypass repeatable-read when creating an actor_idmediawiki/corewmf/1.33.0-wmf.6+8 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

Steinsplitter created this task.Nov 28 2018, 3:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 28 2018, 3:56 PM
Steinsplitter updated the task description. (Show Details)Nov 28 2018, 3:57 PM
Aklapper added a project: Wikimedia-production-error.Nov 28 2018, 4:15 PM
exception.file	       	/srv/mediawiki/php-1.33.0-wmf.6/includes/user/User.php:2555
exception.message	       	Cannot create actor ID for user_id=xxxxxxx user_name=XXXXXXXXX

#0 /srv/mediawiki/php-1.33.0-wmf.6/includes/ActorMigration.php(224): User->getActorId(Wikimedia\Rdbms\DatabaseMysqli)
#1 /srv/mediawiki/php-1.33.0-wmf.6/includes/page/WikiPage.php(2902): ActorMigration->getInsertValues(Wikimedia\Rdbms\DatabaseMysqli, string, User)
#2 /srv/mediawiki/php-1.33.0-wmf.6/includes/page/WikiPage.php(2681): WikiPage->archiveRevisions(Wikimedia\Rdbms\DatabaseMysqli, integer, boolean)
#3 /srv/mediawiki/php-1.33.0-wmf.6/includes/page/WikiPage.php(2624): WikiPage->doDeleteArticleBatched(string, boolean, User, NULL, string, boolean)
#4 /srv/mediawiki/php-1.33.0-wmf.6/includes/FileDeleteForm.php(198): WikiPage->doDeleteArticleReal(string, boolean, integer, boolean, string, User, NULL)
#5 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiDelete.php(171): FileDeleteForm::doDelete(Title, LocalFile, NULL, string, boolean, User, NULL)
#6 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiDelete.php(73): ApiDelete::deleteFile(WikiFilePage, User, NULL, string, boolean, NULL)
#7 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiMain.php(1576): ApiDelete->execute()
#8 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiMain.php(531): ApiMain->executeAction()
#9 /srv/mediawiki/php-1.33.0-wmf.6/includes/api/ApiMain.php(502): ApiMain->executeActionWithErrorHandling()
#10 /srv/mediawiki/php-1.33.0-wmf.6/api.php(87): ApiMain->execute()
#11 /srv/mediawiki/w/api.php(3): include(string)
#12 {main}
Krinkle added a project: Platform Engineering.Nov 28 2018, 6:07 PM
gerritbot subscribed.Nov 29 2018, 2:09 PM

Change 476508 had a related patch set uploaded (by Anomie; owner: Anomie):
[mediawiki/core@master] User: Bypass repeatable-read when creating an actor_id

https://gerrit.wikimedia.org/r/476508

gerritbot added a project: Patch-For-Review.Nov 29 2018, 2:09 PM
Anomie merged a task: T210740: Cannot create actor ID for user_id=156 user_name=Sz-iwbot.Nov 29 2018, 4:26 PM
Anomie added a subscriber: hashar.
greg triaged this task as Unbreak Now! priority.Nov 29 2018, 6:32 PM
greg added a parent task: T206660: 1.33.0-wmf.6 deployment blockers.
Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptNov 29 2018, 6:32 PM
greg assigned this task to Anomie.Nov 29 2018, 6:34 PM
Krinkle moved this task from Untriaged to Jan2020/1.35-wmf.14 on the Wikimedia-production-error board.Nov 29 2018, 8:06 PM
hashar updated the task description. (Show Details)Nov 29 2018, 8:48 PM
hashar added a comment.Nov 29 2018, 8:51 PM

I have looked at logstash over 30 days, there were a few occurrences in November that pre date 1.33.0-wmf.6 so it is no more a blocker.

The log raised starting 2018-11-26 around 15:20 UTC which would correspond to:

15:24 <anomie@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-both/read-old on group 1 (T188327) (duration: 00m 46s)

hashar added a parent task: T188327: Deploy refactored actor storage.Nov 29 2018, 8:54 PM
hashar removed a parent task: T206660: 1.33.0-wmf.6 deployment blockers.
hashar mentioned this in T206660: 1.33.0-wmf.6 deployment blockers.
greg lowered the priority of this task from Unbreak Now! to Medium.Nov 29 2018, 10:25 PM
greg subscribed.
In T210621#4786991, @hashar wrote:

I have looked at logstash over 30 days, there were a few occurrences in November that pre date 1.33.0-wmf.6 so it is no more a blocker.

Anomie added a project: Platform Team Workboards (Waiting for Review).Dec 6 2018, 5:47 PM
Paladox merged a task: T211450: importDump does not work with actor if the user name is not in canonical form.Dec 7 2018, 8:00 PM
Paladox added subscribers: Paladox, Anomie.
Dzahn subscribed.Dec 7 2018, 8:43 PM
This comment was removed by Dzahn.
Dzahn unsubscribed.Dec 7 2018, 8:46 PM
gerritbot added a comment.Dec 7 2018, 11:37 PM

Change 476508 merged by jenkins-bot:
[mediawiki/core@master] User: Bypass repeatable-read when creating an actor_id

https://gerrit.wikimedia.org/r/476508

gerritbot added a comment.Dec 7 2018, 11:40 PM

Change 478363 had a related patch set uploaded (by Paladox; owner: Anomie):
[mediawiki/core@REL1_32] User: Bypass repeatable-read when creating an actor_id

https://gerrit.wikimedia.org/r/478363

gerritbot added a comment.Dec 7 2018, 11:40 PM

Change 478365 had a related patch set uploaded (by Paladox; owner: Anomie):
[mediawiki/core@REL1_31] User: Bypass repeatable-read when creating an actor_id

https://gerrit.wikimedia.org/r/478365

ReleaseTaggerBot added a project: MW-1.33-notes (1.33.0-wmf.8; 2018-12-11).Dec 8 2018, 12:00 AM
gerritbot added a comment.Dec 10 2018, 2:45 PM

Change 478685 had a related patch set uploaded (by Anomie; owner: Anomie):
[mediawiki/core@wmf/1.33.0-wmf.6] User: Bypass repeatable-read when creating an actor_id

https://gerrit.wikimedia.org/r/478685

gerritbot added a comment.Dec 10 2018, 3:01 PM

Change 478685 merged by jenkins-bot:
[mediawiki/core@wmf/1.33.0-wmf.6] User: Bypass repeatable-read when creating an actor_id

https://gerrit.wikimedia.org/r/478685

Stashbot subscribed.Dec 10 2018, 3:05 PM

Mentioned in SAL (#wikimedia-operations) [2018-12-10T15:05:24Z] <anomie@deploy1001> Synchronized php-1.33.0-wmf.6/includes/user/User.php: Backport fix for T210621 (duration: 00m 46s)

Anomie closed this task as Resolved.Dec 10 2018, 3:08 PM

Fixed and backported.

ReleaseTaggerBot edited projects, added MW-1.33-notes (1.33.0-wmf.6; 2018-11-27); removed MW-1.33-notes (1.33.0-wmf.8; 2018-12-11).Dec 10 2018, 4:00 PM
Stashbot added a comment.Dec 11 2018, 6:06 PM

Mentioned in SAL (#wikimedia-operations) [2018-12-11T18:06:35Z] <anomie@deploy1001> Synchronized php-1.33.0-wmf.6/includes/user/User.php: Backport fix for T210621, for real this time (duration: 00m 53s)

Anomie added a comment.Dec 11 2018, 6:08 PM

Bad me, I forgot the "git rebase" step so the scap yesterday didn't deploy the actual fix.

Liuxinyu970226 unsubscribed.Dec 12 2018, 2:38 PM
CCicalese_WMF moved this task from Inbox to Needs Cleaning - Security, stability, performance, and scalability (TEC1) on the Platform Engineering board.Dec 13 2018, 8:06 PM
CCicalese_WMF edited projects, added Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1)); removed Platform Engineering.
CCicalese_WMF moved this task from Waiting for Review to Done with CPT on the Platform Team Workboards board.Dec 13 2018, 8:09 PM
CCicalese_WMF edited projects, added Platform Team Workboards (Done with CPT); removed Platform Team Workboards (Waiting for Review).
gerritbot added a comment.Dec 27 2018, 9:31 PM

Change 478363 merged by jenkins-bot:
[mediawiki/core@REL1_32] User: Bypass repeatable-read when creating an actor_id

https://gerrit.wikimedia.org/r/478363

ReleaseTaggerBot added a project: MW-1.32-notes.Dec 27 2018, 10:00 PM
gerritbot added a comment.Dec 27 2018, 11:43 PM

Change 478365 merged by jenkins-bot:
[mediawiki/core@REL1_31] User: Bypass repeatable-read when creating an actor_id

https://gerrit.wikimedia.org/r/478365

ReleaseTaggerBot added a project: MW-1.31-release-notes.Dec 28 2018, 12:01 AM
Anomie mentioned this in T213663: Actor table migration interfering with page deletion.Feb 10 2019, 12:57 AM
Krinkle moved this task from Jan2020/1.35-wmf.14 to Resolved on the Wikimedia-production-error board.Mar 19 2019, 3:20 PM
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:08 PM
Maintenance_bot removed a project: Patch-For-Review.Aug 28 2019, 11:39 PM
Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits