FileImporter metrics doc can be found here:

https://github.com/wikimedia/mediawiki-extensions-FileImporter/blob/master/docs/metrics.md

Beta-Cluster test wiki with the file importer installed:

https://test.wikimedia.beta.wmflabs.org/wiki/Special:ImportFile

A convenient way to get random files from ( non-beta ) commons to test:

https://commons.wikimedia.org/wiki/Special:Random/File

• Expected to be testable on June 10th, 2018.

@Lea_WMDE @Tobi_WMDE_SW @WMDE-Fisch

You can track the development of this dashboard at: https://grafana.wikimedia.org/dashboard/db/mediawiki-fileimporter

@Lea_WMDE Please take a look at https://grafana.wikimedia.org/dashboard/db/mediawiki-fileimporter

The templating variable $FAILTYPE on this Dashboard needs to be checked by someone who is more experienced in Grafana templating than I am. I have followed the same templating syntax as is used in the Wikidata Entity Usage Projects dashboard - with apparently no success to get to even a preview of multiple values only.

@GoranSMilovanovic thanks for the board!
I found some smaller things

• FileImporter Pageviews and ImportSuccess do not show 0 views/ imports, which makes the graph look a bit weird
• FileImporter Files MB moved: The numbers look surprisingly big, so I just want to make sure it shows what I had in mind: For each file that we want to move, we add up the total MBs that are going to be moved. This means all of the one file's revisions + all of its page revisions. We do not want to add up the MBs of alll the files that have ever been moved.

@thiemowmde can you help Goran with the FAILTYPE variable as mentioned by him above?

@Lea_WMDE

FileImporter Pageviews and ImportSuccess do not show 0 views/ imports, which makes the graph look a bit weird

I am working on it. Funny thing, I am on the Grafana docs and Stack Overflow to find out how to resolve the issue now: most of the people there are complaining how they can't make the graphs hide zero values :)

FileImporter Files MB moved: The numbers look surprisingly big, so I just want to make sure it shows what I had in mind: For each file that we want to move, we add up the total MBs that are going to be moved. This means all of the one file's revisions + all of its page revisions. We do not want to add up the MBs of alll the files that have ever been moved.

• A. I forgot about the following: we are currently showing bytes, not MBs. I will fix it. Do not ask: believe it or not, diving a time series by a constant in Graphite/Grafana is tricky.
• B. We have two metrics available (please refer to metrics doc) that report on file sizes:

MediaWiki.FileImporter.import.details.individualFileSizes.{AGGREGATION} - Individual file revision sizes (bytes).
MediaWiki.FileImporter.import.details.totalFileSizes.{AGGREGATION} - Total size of all revisions in a single import (bytes).

The current query uses MediaWiki.FileImporter.import.details.totalFileSizes. Please let me know exactly what do you want done with these two metrics: add them up, subtract one from another, divide, whatever that would get us to the indicator that you are looking for. Thank you.

@Lea_WMDE Re-scaled to MBs. Please take a look at the Dashboard and let me know whether the numbers in the last row now make sense.

@Lea_WMDE This is fixed now: FileImporter Pageviews and ImportSuccess do not show 0 views/ imports, which makes the graph look a bit weird.

@GoranSMilovanovic that looks great now! The only thing I spot now is that we get N/A instead of 0 in the 1-value displays. But if that is hard to fix, I can live with that.
Thanks again Goran!

@Lea_WMDE Please: what do you mean by 1-value displays? I will see to fix it, just let me know what display do you mean.

@Lea_WMDE Oh, I see... at this point, I can't find a fix. What the dashboard is showing is essentially correct (N/A; and in fact there are no data points for the respective singlestat). I've tried with value mappings, no changes. Give me some time to browse the docs.

@Lea_WMDE Oh oh wait - that is the {FAILTYPE} variable templating problem. Someone else needs to take a look at my queries and the templating setup and let me know if he or she sees something wrong. Otherwise, everything is done in the same way as on the Wikidata dashboards that make use of templating.

Hi @GoranSMilovanovic I'll talk to the WMDE-TechWish today about the variable templating problem.
Now that we are live, though (yay :) ) the numbers below are looking a bit weird:

• the max import time is smaller than the average import time, and the average import time is different to what is shown in the graph visualisation
• file and text revision mean and median are bigger than the top 75% and 90% numbers, but that cannot really be the case, can it?
• The max MB used numbers look very much like KB numbers again, is that possible?

These story points are dedicated to looking into the FAILTYPE variable problem

I tried to have a look, but the issue is so specific, I don't know what I could do. @GoranSMilovanovic, to which specific Wikidata board are you referring to? I tried to find something that uses the same syntax, but was not successful.

@thiemowmde For example, the Wikidata Entity Usage Project uses templating.
Essentially, it is a matter of how the templating variable is defined and used; I cannot guarantee that I have done it properly.

The FileImporter Failure Type singlestat (in the fourth row of the MediaWiki FileImporter dashboard) is the only one that should be affected by the $FAILTYPE templating variable. You can take a look at the variable definition by selecting </>Templating from the dashboard menu (first left to the save icon).

@thiemowmde @Lea_WMDE

• I think I've managed to fix the templating (now you can select the Failure Type, top left corner of the Dashboard).
• Let's wait for some data and then see whether the singlestat delivers what we want.

• FileImporter Failure Type singlestat moved to the first row.

• Templating seems to be fixed now @Lea_WMDE @thiemowmde please check the behavior of the rightmost singlestat/graph in the first and the second row (FileImporter Fail Type) and let me know whether it displays the data that you need.

@GoranSMilovanovic based on our meeting yesterday I removed all metrics that grafana can't really support (i.e. median and percentiles)

@Lea_WMDE The following

how often the import fails because of any exception, seperated by the reasons.

remains unchecked in the task description, however the dashboard seems reactive to any change in the Failure Type selection.

The numbers are still off (c.f. the median), however, we have already seen and discussed these inconsistencies during our session with @Addshore.

Should this task be closed?

P.S.

Once again, my suggestion is: switch to R and Shiny entirely for any task similar to this one. That technology stack is more complicated, and also more timely consuming, but when used - it delivers, like it did for the Advanced Search Extension.