Page MenuHomePhabricator

Add confidence scores to MachineVision prop=imagelabels output
Closed, ResolvedPublic

Description

Provide advice and code review support to Cooltey for making this change.

The coding work will involve updating the existing image labels fetching query in order to retrieve the confidence scores, and then updating ApiQueryImageLabels to include them in the imagelabels API response.

The relevant DB query is defined in Repository::getLabels using the methods of MediaWiki's database abstraction object; that object is documented here. The existing code demonstrates the syntax for a join; you will need to update the query to retrieve the per-provider confidence scores for each label suggestion (which will involve a join on machine_vision_suggestion to get the confidence scores, and another on machine_vision_provider to get the provider name associated with the provider ID in machine_vision_suggestion).

Current query in raw SQL:

SELECT mvi_sha1, mvl_wikidata_id, mvl_review, mvl_reviewer_id 
FROM machine_vision_image
  INNER JOIN machine_vision_label on mvi_id = mvl_mvi_id
WHERE mvi_sha1 = $sha1
ORDER BY mvl_id;

Desired query in raw SQL:

SELECT mvi_sha1, mvl_wikidata_id, mvl_review, mvl_reviewer_id, mvs_confidence, mvp_name
FROM machine_vision_image
  INNER JOIN machine_vision_label on mvi_id = mvl_mvi_id
  INNER JOIN machine_vision_suggestion on mvl_id = mvs_mvl_id
  INNER JOIN machine_vision_provider on mvs_provider_id = mvp_id
WHERE mvi_sha1 = $sha1;

(The ORDER BY clause can be removed when the raw confidence scores are returned, since clients will be able to sort by confidence if desired.)
Current response structure:

{
    "batchcomplete": "",
    "query": {
        "pages": {
            "87316785": {
                "pageid": 87316785,
                "ns": 6,
                "title": "File:9058Jose Aspiras Highway Tubao, La Union 23.jpg",
                "imagelabels": [
                    {
                        "wikidata_id": "Q10390031",
                        "state": "unreviewed",
                        "label": "public road"
                    },
                    ...
               ]
            },
            ...
        }
    }
}

Desired response structure:

{
    "batchcomplete": "",
    "query": {
        "pages": {
            "87316785": {
                "pageid": 87316785,
                "ns": 6,
                "title": "File:9058Jose Aspiras Highway Tubao, La Union 23.jpg",
                "imagelabels": [
                    {
                        "wikidata_id": "Q10390031",
                        "state": "unreviewed",
                        "label": "public road",
                        "confidence": {
                            "google": 0.957253
                        }
                    },
                    ...
               ]
            },
            ...
        }
    }
}

Event Timeline

Mholloway renamed this task from Support MachineVision updates needed for Android to Support addition of confidence scores to MachineVision prop=imagelabels output.Mar 13 2020, 5:32 PM
Mholloway updated the task description. (Show Details)
Mholloway added a project: MachineVision.
Charlotte added subscribers: cooltey, Charlotte.

Hey @Mholloway - can you add a little more info about what needs to be done so that @cooltey can get going? Cheers.

Updated the description. @cooltey Does this make sense? Feel free to put a Hangout on my calendar if you'd like to discuss in greater detail.

Mholloway updated the task description. (Show Details)
Mholloway updated the task description. (Show Details)
Mholloway updated the task description. (Show Details)
Mholloway updated the task description. (Show Details)

Great! This is in the mediawiki/extensions/MachineVision repository.

Change 585027 had a related patch set uploaded (by Cooltey; owner: Cooltey):
[mediawiki/extensions/MachineVision@master] [WIP] Support addition of confidence scores to the imagelabels output

https://gerrit.wikimedia.org/r/585027

Change 585027 merged by jenkins-bot:
[mediawiki/extensions/MachineVision@master] Support addition of confidence scores to the imagelabels output

https://gerrit.wikimedia.org/r/585027

Mholloway renamed this task from Support addition of confidence scores to MachineVision prop=imagelabels output to Add confidence scores to MachineVision prop=imagelabels output.Apr 7 2020, 3:17 PM

Moving this to PI kanban for monitoring rollout and verifying in production.

Hi @schoenbaechler and cc @Charlotte
According to the message on the android Slack channel, I have put the confidence scores inside each image tag and sorted the tags as descending order.

Please note that the confidence score will only display on the Alpha app.

ABorbaWMF subscribed.

I'm not exactly sure how to test this, but it appears to have been verified already on production. Moving it on to sign off.