HomepageModule seems like the logical place for these. The opening and closing for the welcome popup or drawer should probably be a new event (welcome-impression / welcome-close? the schema already contains a mobile flag so using the same event for desktop/mobile is fine); the different types of closing can be handled via action_data. Clicking the welcome popup/drawer's button could be a dedicated event, or just link-click, or a variant of welcome-close. I think what would make the most sense is using welcome-close in variant D (where it doesn't do anything other than closing) and se-cta-click in variant C (where it opens the onboarding).

Clicking the info button could also use se-cta-click (which means we need to add action_data to differentiate). In variant D, the user can select task types as well as topics during onboarding, so we'll need to add task types to the se-activate action_data (topics are already there). No change for the other onboarding events (se-cta-difficulty, se-cta-back, se-activate in variant C, se-cancel-activation - although in variant C confirming the form doesn't really activate anything, but close enough), other than they will be logged for the suggested edits module, not startediting.

So, schema changes:

• welcome-impression
• welcome-close with type=close-icon, type=outside-click or type=button as action_data
• se-cta-click gets type=welcome / type=info-icon action_data
• se-activate gets taskTypes=[...] action_data all SE module events log task types, so there is no need.

Do we want to log the variant as part of the event? It can be reconstructed from the user ID, but that seems inconvenient.

schema diff

Some questions:

• what is the preferred behavior of clicking the welcome notice button in Variant C? Should that log a welcome notice close event or an onboarding open (impression) event or both?
• how should the mobile task preview be logged? A normal task impression? Then the same task impression again when the user clicks through? (Also, how important is it? Since we show it before the suggested edits module is loaded, using the same logging logic might be cumbersome.)

@Tgr -- what you wrote in T262485#6486155 mostly sounds right to me. I have these notes:

• You wrote that for Variant C, "se-cta-click gets type=welcome / type=info-icon action_data". I would like se-cta-difficulty and se-cta-back to also get those same action_datas.
• How are you planning to do Variant D-desktop? Since the first onboarding screen is already on the page when the user arrives, should it have se-cta-click with type=welcome or something like that? Because se-cta-click essentially means "topic overlay"?
• What is type=close-icon? Is that when the user closes with the "X"?

Your specific questions:

Do we want to log the variant as part of the event? It can be reconstructed from the user ID, but that seems inconvenient.

When you say it can be reconstructed from the user ID, you mean that we could find it in user_properties via the work done on T251477, right? I agree that it would be more convenient to store it in the event. Let's do it!

what is the preferred behavior of clicking the welcome notice button in Variant C? Should that log a welcome notice close event or an onboarding open (impression) event or both?

It is easiest for our analysis if it logs both. Is that okay, or are we starting to add weight and slow down the experience?

how should the mobile task preview be logged? A normal task impression? Then the same task impression again when the user clicks through? (Also, how important is it? Since we show it before the suggested edits module is loaded, using the same logging logic might be cumbersome.)

I think we should log the preview the same way we log a normal task impression (like including what the article is and stuff), but we'll know that it was a preview from the mode field, which will say mobile-summary. How does that sound?

I would like se-cta-difficulty and se-cta-back to also get those same action_datas.

Ack. And then se-activate / se-cancel-activation too I presume.

Since the first onboarding screen is already on the page when the user arrives, should it have se-cta-click with type=welcome or something like that?

I haven't thought about that, but we can do it if you prefer. I'd probably add a new type=impression for that, to make it clearer that it is not really a click.

• What is type=close-icon? Is that when the user closes with the "X"?

Yes.

When you say it can be reconstructed from the user ID, you mean that we could find it in user_properties via the work done on T251477, right?

Yeah, you can get the variant with a few SQL joins. It's trivial to add it in Javascript though.

It is easiest for our analysis if it logs both. Is that okay, or are we starting to add weight and slow down the experience?

Dozens of evens are logged on an average pageview. I'm not sure how much of a performance impact that has (individually they are tiny, and modern web protocols reuse the same connection for many requests, making them very cheap) but in any case making it one more or less is not going to make any difference. I'll just add both then. It's marginally more convenient in terms of code structure as well.

I think we should log the preview the same way we log a normal task impression (like including what the article is and stuff), but we'll know that it was a preview from the mode field, which will say mobile-summary. How does that sound?

It's not any worse than other ways of logging it. Logging it at all might be complicated, since the suggested edits module code is not loaded yet at this point, but there might be some easy ways around that. I'll try and see.

combined schema diff

@Tgr

Ack. And then se-activate / se-cancel-activation too I presume.

Yes, please.

I haven't thought about that, but we can do it if you prefer. I'd probably add a new type=impression for that, to make it clearer that it is not really a click.

That sounds fine, and then we should use that same type=impression for se-cta-difficulty, se-cta-back, and se-activate.

Yeah, you can get the variant with a few SQL joins. It's trivial to add it in Javascript though.

Great, let's do it. This may make our reporting notebooks easier (@nettrom_WMF).

It's not any worse than other ways of logging it. Logging it at all might be complicated, since the suggested edits module code is not loaded yet at this point, but there might be some easy ways around that. I'll try and see.

Thanks.

Change 629937 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Allow logging homepage events via mw.track

https://gerrit.wikimedia.org/r/629937

Change 629938 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] [WIP] Variant C/D instrumentation

https://gerrit.wikimedia.org/r/629938

final schema diff.

There are some trivial differences, this is the final log of changes:

• An action_data.user_variant field is added to all events.
• There are two new event types, welcome-impression (when the welcome popup or drawer appears) and welcome-close (when it is closed by the user).
  • For close, the action_data.type field contains the method by which it was closed: close-icon for the X icon on the desktop popup, outside-click for clikcing outside the dialog/drawer, button for the "Got it" / Get suggestions" button.
• The se-cta-click, se-cta-difficulty, se-cta-back, se-activate and se-cancel-activation events are logged where appropriate.
  • They have a new action_data.trigger field, which contains how onboarding was triggered: welcome for the welcome popup or drawer, info-icon for the variant C icon, suggested-edits for the variant D mobile summary button, impression for variant D desktop when it is visible immediately. For variant A (where it is triggered via the StartEditing module button) the field is omitted for compatibility with the existing log records.
  • se-activate now logs the task types the user selected. (The field existed before but wasn't filled correctly.)
• In variant C mobile summary mode a se-task-impression event is logged (or se-task-pseudo-impression in the case of error).

Change 629937 abandoned by Gergő Tisza:
[mediawiki/extensions/GrowthExperiments@master] Allow logging homepage events via mw.track

Reason:

https://gerrit.wikimedia.org/r/629937

@MMiller_WMF what should be logged when the user clicks "Done" in the onboarding dialog in variant C? It doesn't really activate anything, and can be done any number of time unlike D's proper activation, but it isn't cancelling the dialog either. Should there be a new action for that? Or just shoehorn it into se-activate or se-cancel-activation?

Per Gerrit discussion, a new generic pseudo-module was added, for events that don't really belong to a module. This is only the case for the welcome events (drawer / popup) currently.

@Tgr -- I think we should shoehorn into se-activate, because that fits with how we shoehorned the initiation of that workflow into se-cta-click. That makes things consistent, it's just that the names of the actions themselves no longer make sense. Like, I wish se-activate were called, like se-finish-onboarding. But it's too late for that. When the dust settles here, I'd like to write down a clear table of what all the action/action_data/mode combos mean.

Change 629938 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Variant C/D instrumentation

https://gerrit.wikimedia.org/r/629938

Change 631163 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] Instrumentation: Adjust start editing teardown logging

https://gerrit.wikimedia.org/r/631163

@Tgr -- in testing EventLogging on desktop, I've noticed these issues so far. I'll be testing mobile next.

• In Variant A right now, we log events for the start-email module. But in Variants C and D, it's startemail. Those should actually be the same, right?
• The startemail event should come with a value for state, like it does in Variant A now. In Variants C and D, it has no value for state. The values should be like confirmed, unconfirmed, etc.
• In Variant C, for the impression and se-task-impression events I get for the suggested edits module upon arrival to the homepage, they have action_data that says taskTypes=null;taskCount=null;topics=null. I think those should not be null, because they do have values when the user first arrives. These values do look right, however, in the se-fetch-tasks event.
• In Variant D, when I arrive on the homepage, I get an impression event with module: "start-startediting". That event shouldn't happen, right? That's the event for getting Variant A's sub-module that has the CTA button.
• In Variant D, when I first arrive on the page, I get se-cta-click with trigger="impression". But then when I advance to the difficulty screen (se-cta-difficulty) and then to the module (se-activate), both those latter events have trigger="suggested-edits". They should all have trigger="impression", right?

Update:
I've now tested on mobile, and found no additional issues, but most of the issues I listed above also exist on mobile.

Change 631163 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Instrumentation: Adjust start editing teardown logging

https://gerrit.wikimedia.org/r/631163

Change 632175 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] HomepageModule: Update schema version for startemail module

https://gerrit.wikimedia.org/r/632175

In Variant A right now, we log events for the start-email module. But in Variants C and D, it's startemail. Those should actually be the same, right?

They're different. The start module contains a bunch of child modules (these are shown in variant A), but variant C/D doesn't have a start module, it just has the "add email" module. I've uploaded a patch to add "startemail" to the HomepageModule schema.

Change 632179 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] StartEmail: Export state to homepage logger

https://gerrit.wikimedia.org/r/632179

The startemail event should come with a value for state, like it does in Variant A now. In Variants C and D, it has no value for state. The values should be like confirmed, unconfirmed, etc.

Oops, good catch, another patch uploaded for this one now.

In Variant D, when I arrive on the homepage, I get an impression event with module: "start-startediting". That event shouldn't happen, right? That's the event for getting Variant A's sub-module that has the CTA button.

We reuse the start module's start-startediting module for variant D to show the onboarding dialog. (I realize that's conceptually different than what I just wrote about start module's start-email module and the new startemail module we created separate from the existing start / child-task paradigm.)

Change 632193 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] HomepageModule: Export taskcount, tasks and topics on server-side

https://gerrit.wikimedia.org/r/632193

In Variant C, for the impression and se-task-impression events I get for the suggested edits module upon arrival to the homepage, they have action_data that says taskTypes=null;taskCount=null;topics=null. I think those should not be null, because they do have values when the user first arrives. These values do look right, however, in the se-fetch-tasks event.

Right; we can export this data on the server-side, I've attached a patch to do so.

In Variant D, when I first arrive on the page, I get se-cta-click with trigger="impression". But then when I advance to the difficulty screen (se-cta-difficulty) and then to the module (se-activate), both those latter events have trigger="suggested-edits". They should all have trigger="impression", right?

I believe this is by design; @Tgr wrote in the code "For technical reasons we implement this dialog with the StartEditing module, but conceptually it is the pre-initiation view of the SuggestedEdits module." @MMiller_WMF @nettrom_WMF would you like this to be changed?

In T262485#6510991, @MMiller_WMF wrote:

@Tgr -- in testing EventLogging on desktop, I've noticed these issues so far. I'll be testing mobile next.

1. In Variant A right now, we log events for the start-email module. But in Variants C and D, it's startemail. Those should actually be the same, right?
2. The startemail event should come with a value for state, like it does in Variant A now. In Variants C and D, it has no value for state. The values should be like confirmed, unconfirmed, etc.
3. In Variant C, for the impression and se-task-impression events I get for the suggested edits module upon arrival to the homepage, they have action_data that says taskTypes=null;taskCount=null;topics=null. I think those should not be null, because they do have values when the user first arrives. These values do look right, however, in the se-fetch-tasks event.
4. In Variant D, when I arrive on the homepage, I get an impression event with module: "start-startediting". That event shouldn't happen, right? That's the event for getting Variant A's sub-module that has the CTA button.
5. In Variant D, when I first arrive on the page, I get se-cta-click with trigger="impression". But then when I advance to the difficulty screen (se-cta-difficulty) and then to the module (se-activate), both those latter events have trigger="suggested-edits". They should all have trigger="impression", right?

Update:
I've now tested on mobile, and found no additional issues, but most of the issues I listed above also exist on mobile.

So, patches exist for 2 and 3 (well, for item 1 also but just to define "startemail" as a valid module name in Schema:HomepageModule). The other items (1, 4, 5) are working as we designed it in code but we can make adjustments there if @MMiller_WMF and @nettrom_WMF prefer.

Change 632175 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] HomepageModule: Update schema version for startemail module

https://gerrit.wikimedia.org/r/632175

Change 632179 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] StartEmail: Export state to homepage logger

https://gerrit.wikimedia.org/r/632179

Change 632193 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] HomepageModule: Export taskcount, tasks and topics on server-side

https://gerrit.wikimedia.org/r/632193

Checked https://phabricator.wikimedia.org/T262485#6516491

2. The startemail event should come with a value for state, like it does in Variant A now. In Variants C and D, it has no value for state. The values should be like confirmed, unconfirmed, etc.

2 is Fixed.

3. In Variant C, for the impression and se-task-impression events I get for the suggested edits module upon arrival to the homepage, they have action_data that says taskTypes=null;taskCount=null;topics=null. I think those should not be null, because they do have values when the user first arrives. These values do look right, however, in the se-fetch-tasks event.

Specific topics would be displayed when actual topics selection is made; the default topic selection is presented as topics="
Otherwise
impression

action: "impression", action_data: "taskTypes=copyedit,links;taskCount=250;topics=", user_id: 40769, user_editcount: 0, user_variant: "C", …}
action: "impression"
action_data: "taskTypes=copyedit,links;taskCount=250;topics="
homepage_pageview_token: "cu5c93isiqe1qosqr52tqcs0sfr9rvin"
is_mobile: false
mode: "desktop"
module: "suggested-edits"
state: "activated"
user_editcount: 0
user_id: 40769
user_variant: "C"

se-task-impression

action: "se-task-impression", action_data: "taskTypes=copyedit,links;taskCount=250;topics=;newcomerTaskToken=e4b972b8221f12edf869", user_id: 40769, user_editcount: 0, user_variant: "C", …}
action: "se-task-impression"
action_data: "taskTypes=copyedit,links;taskCount=250;topics=;newcomerTaskToken=e4b972b8221f12edf869"
homepage_pageview_token: "cu5c93isiqe1qosqr52tqcs0sfr9rvin"
is_mobile: false
mode: "desktop"
module: "suggested-edits"
state: "activated"
user_editcount: 0
user_id: 40769
user_variant: "C"

It's interesting that there are different taskCount numbers between se-fetch-tasksse-task-navigation and se-task-impression/impression when no topic selection is made (for the initial page load.)

action: "se-fetch-tasks", action_data: "taskTypes=copyedit,links;taskCount=200;topics=", user_id: 40769, user_editcount: 0, user_variant: "C", …}
action: "se-fetch-tasks"
action_data: "taskTypes=copyedit,links;taskCount=200;topics="
homepage_pageview_token: "cu5c93isiqe1qosqr52tqcs0sfr9rvin"
is_mobile: false
mode: "desktop"
module: "suggested-edits"
state: "activated"
user_editcount: 0
user_id: 40769
user_variant: "C"

For PM review - checked all specs - looks ok.

The only thing that I could not see it's mobile-details - both modes - mobile-summary and mobile-overlay are present but not mobile-details.

In T262485#6527396, @Etonkovidova wrote:

For PM review - checked all specs - looks ok.

The only thing that I could not see it's mobile-details - both modes - mobile-summary and mobile-overlay are present but not mobile-details.

mobile-details only appears if the user has JavaScript disabled (on a phone, not so likely) or if JavaScript fails to load due to a script error, connectivity issues, etc. I think we could talk through removing the mobile-details mode entirely; we built it as the first pass of mobile implementation of Special:Homepage, and haven't given it too much attention since then. That's probably deserving of its own task, so I made T264992: Remove mobile-details mode from Special:Homepage