Page MenuHomePhabricator
Create Task
Maniphest T313395

User impact API: Create maintenance script for refreshing data
Closed, ResolvedPublic
Actions

Description

T313393: User impact API: Create GrowthExperimentsUserImpactManager, GrowthExperimentsUserImpactLookup, and GrowthExperimentsUserImpactCompute services describes the services needed to compute, lookup, and store user impact data.

This task is about creating a maintenance script that will eventually run periodically (hourly? daily?) in production.

The maintenance script should:

Details

SubjectRepoBranchLines +/-
refreshUserImpactData: Add feature flagmediawiki/extensions/GrowthExperimentsmaster+11 -0
refreshUserImpactData: Add feature flagmediawiki/extensions/GrowthExperimentswmf/1.40.0-wmf.8+11 -0
GrowthExperiments: Set feature-flag for RefreshUserImpactDataMaintenanceScriptEnabledoperations/mediawiki-configmaster+7 -0
Add UserRegistrationLookupHelpermediawiki/extensions/GrowthExperimentswmf/1.40.0-wmf.8+54 -7
Add UserRegistrationLookupHelpermediawiki/extensions/GrowthExperimentsmaster+54 -7
Fix RefreshUserImpactData in MySQL strict modemediawiki/extensions/GrowthExperimentsmaster+11 -3
RefreshUserImpactData: Fix queries that filesortmediawiki/extensions/GrowthExperimentsmaster+8 -22
Add maintenance script for deleting expired user impact datamediawiki/extensions/GrowthExperimentsmaster+88 -1
Add maintenance script to refresh user impact datamediawiki/extensions/GrowthExperimentsmaster+206 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

kostajh created this task.Jul 20 2022, 10:48 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 20 2022, 10:48 AM
kostajh moved this task from Incoming to Ready for Development on the Growth-Team (Sprint 0 (Growth Team)) board.Jul 20 2022, 10:48 AM
kostajh mentioned this in T309747: Positive Reinforcement: Technical Research Spike.Jul 20 2022, 10:56 AM
Ladsgroup subscribed.Jul 20 2022, 10:59 AM
KStoller-WMF triaged this task as Medium priority.Jul 22 2022, 10:06 PM
DMburugu subscribed.Jul 25 2022, 3:10 PM

@kostajh: Will we need any DBA help on this?

DMburugu added a comment.Jul 25 2022, 3:14 PM

No problem, I just saw this: https://phabricator.wikimedia.org/T313393

KStoller-WMF added a parent task: T222310: [EPIC] Positive reinforcement: New Impact Module.Sep 3 2022, 12:38 AM
KStoller-WMF edited parent tasks, added: T315732: [EPIC] Structured Tasks: Patroller Focus; removed: T222310: [EPIC] Positive reinforcement: New Impact Module.Sep 9 2022, 9:44 PM
KStoller-WMF edited parent tasks, added: T222310: [EPIC] Positive reinforcement: New Impact Module; removed: T315732: [EPIC] Structured Tasks: Patroller Focus.
• Tgr claimed this task.Sep 12 2022, 12:55 PM
• Tgr moved this task from Ready for Development to In Progress on the Growth-Team (Sprint 0 (Growth Team)) board.
• Tgr added a subtask: T317534: Create growthexperiments_user_impact table in Wikimedia production.Sep 12 2022, 1:12 PM
• Tgr added a comment.Sep 13 2022, 9:17 AM
In T313395#8101920, @DMburugu wrote:

@kostajh: Will we need any DBA help on this?

The necessary DBA work is in T317534: Create growthexperiments_user_impact table in Wikimedia production. @kostajh pointed out that we could also use the main stash though, that has some restrictions but does not require any DB setup.

kostajh changed the status of subtask T317534: Create growthexperiments_user_impact table in Wikimedia production from Open to In Progress.Sep 16 2022, 8:33 AM
gerritbot added a comment.Sep 21 2022, 4:44 PM

Change 833828 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/GrowthExperiments@master] Add maintenance script to refresh user impact data

https://gerrit.wikimedia.org/r/833828

gerritbot added a project: Patch-For-Review.Sep 21 2022, 4:44 PM
gerritbot added a comment.Sep 23 2022, 8:11 PM

Change 834612 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/GrowthExperiments@master] Add maintenance script for deleting expired user impact data

https://gerrit.wikimedia.org/r/834612

gerritbot added a comment.Sep 27 2022, 10:35 AM

Change 835573 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/GrowthExperiments@master] RefreshUserImpactData: Fix queries that filesort

https://gerrit.wikimedia.org/r/835573

gerritbot added a comment.Sep 28 2022, 12:24 PM

Change 833828 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Add maintenance script to refresh user impact data

https://gerrit.wikimedia.org/r/833828

gerritbot added a comment.Sep 28 2022, 12:37 PM

Change 834612 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Add maintenance script for deleting expired user impact data

https://gerrit.wikimedia.org/r/834612

gerritbot added a comment.Sep 28 2022, 12:46 PM

Change 835573 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] RefreshUserImpactData: Fix queries that filesort

https://gerrit.wikimedia.org/r/835573

ReleaseTaggerBot added a project: MW-1.40-notes (1.40.0-wmf.4; 2022-10-03).Sep 28 2022, 1:00 PM
Maintenance_bot removed a project: Patch-For-Review.Sep 28 2022, 1:30 PM
Urbanecm_WMF changed the task status from Open to In Progress.Sep 29 2022, 10:05 AM
gerritbot added a comment.Oct 2 2022, 9:50 AM

Change 837215 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/GrowthExperiments@master] Fix RefreshUserImpactData in MySQL strict mode

https://gerrit.wikimedia.org/r/837215

gerritbot added a project: Patch-For-Review.Oct 2 2022, 9:50 AM
gerritbot added a comment.Oct 2 2022, 10:14 AM

Change 837215 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Fix RefreshUserImpactData in MySQL strict mode

https://gerrit.wikimedia.org/r/837215

Maintenance_bot removed a project: Patch-For-Review.Oct 2 2022, 10:30 AM
• Tgr moved this task from In Progress to Code Review on the Growth-Team (Sprint 0 (Growth Team)) board.Nov 7 2022, 2:37 AM
ReleaseTaggerBot edited projects, added MW-1.40-notes (1.40.0-wmf.10; 2022-11-14); removed MW-1.40-notes (1.40.0-wmf.4; 2022-10-03).Nov 7 2022, 1:00 PM
kostajh moved this task from Code Review to QA on the Growth-Team (Sprint 0 (Growth Team)) board.Nov 7 2022, 1:04 PM
kostajh mentioned this in T322541: UserImpact: Set up maintenance script to run in betalabs and production.
kostajh added a parent task: T322541: UserImpact: Set up maintenance script to run in betalabs and production.
• Tgr removed a parent task: T322541: UserImpact: Set up maintenance script to run in betalabs and production.Nov 7 2022, 11:45 PM
• Tgr added a subtask: T322541: UserImpact: Set up maintenance script to run in betalabs and production.
• Tgr changed the status of subtask T317534: Create growthexperiments_user_impact table in Wikimedia production from In Progress to Open.Nov 8 2022, 12:45 AM
ReleaseTaggerBot edited projects, added MW-1.40-notes (1.40.0-wmf.8; 2022-10-31); removed MW-1.40-notes (1.40.0-wmf.10; 2022-11-14).Nov 8 2022, 2:00 AM
• Tgr changed the status of subtask T322541: UserImpact: Set up maintenance script to run in betalabs and production from In Progress to Open.Nov 8 2022, 3:52 AM
Urbanecm_WMF subscribed.Nov 10 2022, 2:07 PM

The script fatal errors if there are zero edits tagged with newcomer task (which happens to be the case in my dev environment). Since the script doesn't appear to be feature flagged, it'll run on all wikis, even on wikis where suggested edits are not set up (the smallest ones).

Stacktrace from my dev environment:

urbanecm@notebook  ~/unsynced/gerrit/mediawiki/extensions/GrowthExperiments
$ php maintenance/refreshUserImpactData.php --wiki=awiki --editedWithin=1year --registeredWithin=1year
processing 100 users starting with 0
MediaWiki\Storage\NameTableAccessException from line 43 of /home/urbanecm/unsynced/gerrit/mediawiki/core/includes/Storage/NameTableAccessException.php: Failed to access name from change_tag_def using name = newcomer task
#0 /home/urbanecm/unsynced/gerrit/mediawiki/core/includes/Storage/NameTableStore.php(261): MediaWiki\Storage\NameTableAccessException::newFromDetails()
#1 /home/urbanecm/unsynced/gerrit/mediawiki/extensions/GrowthExperiments/includes/UserImpact/ComputedUserImpactLookup.php(214): MediaWiki\Storage\NameTableStore->getId()
#2 /home/urbanecm/unsynced/gerrit/mediawiki/extensions/GrowthExperiments/includes/UserImpact/ComputedUserImpactLookup.php(161): GrowthExperiments\UserImpact\ComputedUserImpactLookup->getEditData()
#3 /home/urbanecm/unsynced/gerrit/mediawiki/extensions/GrowthExperiments/maintenance/refreshUserImpactData.php(70): GrowthExperiments\UserImpact\ComputedUserImpactLookup->getExpensiveUserImpact()
#4 /home/urbanecm/unsynced/gerrit/mediawiki/core/maintenance/includes/MaintenanceRunner.php(309): GrowthExperiments\Maintenance\RefreshUserImpactData->execute()
#5 /home/urbanecm/unsynced/gerrit/mediawiki/core/maintenance/doMaintenance.php(85): MediaWiki\Maintenance\MaintenanceRunner->run()
#6 /home/urbanecm/unsynced/gerrit/mediawiki/extensions/GrowthExperiments/maintenance/refreshUserImpactData.php(186): require_once('/home/urbanecm/...')
#7 {main}
urbanecm@notebook  ~/unsynced/gerrit/mediawiki/extensions/GrowthExperiments
$
kostajh added a comment.Nov 10 2022, 2:10 PM
In T313395#8386637, @Urbanecm_WMF wrote:

The script fatal errors if there are zero edits tagged with newcomer task (which happens to be the case in my dev environment). Since the script doesn't appear to be feature flagged, it'll run on all wikis, even on wikis where suggested edits are not set up (the smallest ones).

Stacktrace from my dev environment:

urbanecm@notebook  ~/unsynced/gerrit/mediawiki/extensions/GrowthExperiments
$ php maintenance/refreshUserImpactData.php --wiki=awiki --editedWithin=1year --registeredWithin=1year
processing 100 users starting with 0
MediaWiki\Storage\NameTableAccessException from line 43 of /home/urbanecm/unsynced/gerrit/mediawiki/core/includes/Storage/NameTableAccessException.php: Failed to access name from change_tag_def using name = newcomer task
#0 /home/urbanecm/unsynced/gerrit/mediawiki/core/includes/Storage/NameTableStore.php(261): MediaWiki\Storage\NameTableAccessException::newFromDetails()
#1 /home/urbanecm/unsynced/gerrit/mediawiki/extensions/GrowthExperiments/includes/UserImpact/ComputedUserImpactLookup.php(214): MediaWiki\Storage\NameTableStore->getId()
#2 /home/urbanecm/unsynced/gerrit/mediawiki/extensions/GrowthExperiments/includes/UserImpact/ComputedUserImpactLookup.php(161): GrowthExperiments\UserImpact\ComputedUserImpactLookup->getEditData()
#3 /home/urbanecm/unsynced/gerrit/mediawiki/extensions/GrowthExperiments/maintenance/refreshUserImpactData.php(70): GrowthExperiments\UserImpact\ComputedUserImpactLookup->getExpensiveUserImpact()
#4 /home/urbanecm/unsynced/gerrit/mediawiki/core/maintenance/includes/MaintenanceRunner.php(309): GrowthExperiments\Maintenance\RefreshUserImpactData->execute()
#5 /home/urbanecm/unsynced/gerrit/mediawiki/core/maintenance/doMaintenance.php(85): MediaWiki\Maintenance\MaintenanceRunner->run()
#6 /home/urbanecm/unsynced/gerrit/mediawiki/extensions/GrowthExperiments/maintenance/refreshUserImpactData.php(186): require_once('/home/urbanecm/...')
#7 {main}
urbanecm@notebook  ~/unsynced/gerrit/mediawiki/extensions/GrowthExperiments
$

Thanks. That is addressed in the as-yet-unmerged https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/853501

gerritbot added a comment.Nov 10 2022, 2:17 PM

Change 855575 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] refreshUserImpactData: Add feature flag

https://gerrit.wikimedia.org/r/855575

gerritbot added a project: Patch-For-Review.Nov 10 2022, 2:17 PM
gerritbot added a comment.Nov 10 2022, 2:20 PM

Change 855576 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] GrowthExperiments: Set feature-flag for RefreshUserImpactDataMaintenanceScriptEnabled

https://gerrit.wikimedia.org/r/855576

gerritbot added a comment.Nov 10 2022, 2:29 PM

Change 855587 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@wmf/1.40.0-wmf.8] refreshUserImpactData: Add feature flag

https://gerrit.wikimedia.org/r/855587

gerritbot added a comment.Nov 10 2022, 2:30 PM

Change 855576 merged by jenkins-bot:

[operations/mediawiki-config@master] GrowthExperiments: Set feature-flag for RefreshUserImpactDataMaintenanceScriptEnabled

https://gerrit.wikimedia.org/r/855576

Stashbot added a comment.Nov 10 2022, 2:30 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-10T14:30:55Z] <kharlan@deploy1002> Started scap: Backport for [[gerrit:855576|GrowthExperiments: Set feature-flag for RefreshUserImpactDataMaintenanceScriptEnabled (T313395)]]

Stashbot added a comment.Nov 10 2022, 2:31 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-10T14:31:15Z] <kharlan@deploy1002> kharlan and kharlan: Backport for [[gerrit:855576|GrowthExperiments: Set feature-flag for RefreshUserImpactDataMaintenanceScriptEnabled (T313395)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet

Stashbot added a comment.Nov 10 2022, 2:35 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-10T14:35:53Z] <kharlan@deploy1002> Finished scap: Backport for [[gerrit:855576|GrowthExperiments: Set feature-flag for RefreshUserImpactDataMaintenanceScriptEnabled (T313395)]] (duration: 04m 57s)

gerritbot added a comment.Nov 10 2022, 2:58 PM

Change 855587 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@wmf/1.40.0-wmf.8] refreshUserImpactData: Add feature flag

https://gerrit.wikimedia.org/r/855587

Stashbot mentioned this in T322706: User impact API: Maintenance scripts should defer work to the job queue.Nov 10 2022, 2:58 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-10T14:58:23Z] <kharlan@deploy1002> Started scap: Backport for [[gerrit:855525|refreshUserImpactData: Add option to use job queue (T322706)]], [[gerrit:855587|refreshUserImpactData: Add feature flag (T313395)]]

Stashbot added a comment.Nov 10 2022, 2:58 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-10T14:58:42Z] <kharlan@deploy1002> kharlan and kharlan: Backport for [[gerrit:855525|refreshUserImpactData: Add option to use job queue (T322706)]], [[gerrit:855587|refreshUserImpactData: Add feature flag (T313395)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet

Stashbot added a comment.Nov 10 2022, 3:03 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-10T15:03:10Z] <kharlan@deploy1002> Finished scap: Backport for [[gerrit:855525|refreshUserImpactData: Add option to use job queue (T322706)]], [[gerrit:855587|refreshUserImpactData: Add feature flag (T313395)]] (duration: 04m 47s)

gerritbot added a comment.Nov 10 2022, 3:15 PM

Change 855575 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] refreshUserImpactData: Add feature flag

https://gerrit.wikimedia.org/r/855575

ReleaseTaggerBot edited projects, added MW-1.40-notes (1.40.0-wmf.10; 2022-11-14); removed MW-1.40-notes (1.40.0-wmf.8; 2022-10-31).Nov 10 2022, 4:00 PM
kostajh added a project: GrowthExperiments-ImpactModule.Nov 21 2022, 4:44 PM
kostajh closed subtask T317534: Create growthexperiments_user_impact table in Wikimedia production as Resolved.Dec 5 2022, 8:35 AM
kostajh moved this task from Triage to NewImpact Release (Completed/QA) on the GrowthExperiments-ImpactModule board.
Etonkovidova closed this task as Resolved.Dec 16 2022, 9:38 PM
Etonkovidova subscribed.

Checked in betalabs - the proper error is displayed for a wiki without GrowthExperiments. I could not find an example of wiki in betalabs where GrowthExperiments extension is enabled, but SuggestedEdits are not.

$ mwscript extensions/GrowthExperiments/maintenance/refreshUserImpactData.php --wiki=hewiktionary --editedWithin=1year --registeredWithin=1year
The "GrowthExperiments" extension must be installed for this script to run. Please enable it and then try again.
Etonkovidova closed subtask T322541: UserImpact: Set up maintenance script to run in betalabs and production as Resolved.Dec 19 2022, 10:12 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits