Commons:Bots/Work requests

From Wikimedia Commons, the free media repository
Jump to navigation Jump to search
Shortcuts
1, 2, 3, 4, 5, 6, 7, 8, 9, 10
11, 12, 13, 14, 15, 16, 17


This is a page for requesting work to be done by a bot. This is an appropriate place to simply put ideas for bots. However be aware of various tools available to all users which can be used to accomplish the work without the need for a bot:
  1. Gadget Cat-a-lot can be used for most category adding, removing and moving
  2. VisualFileChange tool can be used for mass-changes of one author's uploaded files, or files in a category, creating a mass-deletion-request, let you insert tags to the file-description-pages (even copy from exif/meta-data). You can also perform find and replace operations with or without JavaScript regular expressions (regexp). A common use case is: "I've changed my username". It's a web-tool and can be launched directly.
  3. AutoWikiBrowser can be used for large number of supervised and automatic edits
  4. Commons:Batch uploading page can be used to request mass image uploads
Please add {{Section resolved|1=~~~~}} when request is resolved.
SpBot archives all sections tagged with {{Section resolved|1=~~~~}} after 5 days.



Detect identical duplicates in file namespace

Is it possible to query the database for files with identical checksum? If possible could this be made a regular query with an output file made similar to Commons:Database reports/Largely duplicative file names? Most interesting would be files with 1 or 2 duplicates, these are most likely not intended. --Denniss (talk) 20:43, 7 January 2014 (UTC)[reply]

That might be a pretty complicated query.  Doing… --Zhuyifei1999 (talk) 05:10, 8 January 2014 (UTC)[reply]
WTF 'image' table has no id column? @Hazard-SJ: Can you do this? --Zhuyifei1999 (talk) 05:24, 8 January 2014 (UTC)[reply]
Bump --Denniss (talk) 14:27, 29 January 2014 (UTC)[reply]
I've just started to test some things, but so far I've received no results. Of course, we can't only rely on duplicate sha1 values, so I'm testing with other fields to see if I can get anything.  Hazard SJ  04:07, 7 February 2014 (UTC)[reply]
im hoping that something like this will eventually become a special page (gerrit:85446. Only pending review for 5 months now..). Feel free to copy the query used there. I doubt many other of the fields would be useful (img_metadata possibly, but that would be super slow). From a longer term prespective, i think it would be interesting to have a log of recently uploaded files that are dupes of old and deleted imaged. As for the structure of image/oldimage/filearchive, yeah they suck (from what i understand the article related tables got fixed to something saner in mediawiki 1.5, but nobody ever fixed up the image table). Bawolff (talk) 02:27, 14 February 2014 (UTC)[reply]
@Bawolff: If I'm not mistaken, that query just brings up files with the same sha1 values, correct? It is possible for different files to have the same value, so we wouldn't want to rely on that alone.  Hazard SJ  03:31, 14 February 2014 (UTC)[reply]
well there certainly exists in theory such files (by the pigeon hole principle), as far as i know, nobody has ever found any examples for sha1 thus far. If you manage to find 2 different files with the same sha1 (excluding some case of db corruption or integrity issues. I know there was at least one bug a while back where a file had the incorrect sha1 in db, but i think that has all been fixed), it would be a very big deal (to crypto folks anyways) Bawolff (talk) 05:20, 14 February 2014 (UTC)[reply]
SELECT count(*) as "Numb dupes", MIN(img_name) as "Image name" from image group by img_sha1 HAVING count(*) > 1 order by count(*) desc Gives results like [1] Bawolff (talk) 20:45, 14 February 2014 (UTC)[reply]
tldr. Is User:RLuts/duplicates the wanted report? --McZusatz (talk) 19:35, 16 February 2014 (UTC)[reply]
@Denniss: is that (Bawolff's list) okay? If so, would you still want a database reports page for it? It could link to Special:FileDuplicateSearch such as Special:FileDuplicateSearch/Abhandlung_von_den_Zähnen_(Pfaff)_219.jpg.  Hazard SJ  01:17, 18 February 2014 (UTC)[reply]
User:RLuts/duplicates is almost exactly what I want, just as a regularly updated (weekly?) database report from bottom to top (least numbers of duplicates first) and limited to ~500 entries to keep the page size in reasonable limits. The format of Bawolff's list is also OK, it seems to list only one file with existing dupes and not all existing dupes, if you can transform it into proper wikilinks it would be easier to handle than the pure text. Both pages show a hell lot of dupes.--Denniss (talk) 10:34, 18 February 2014 (UTC)[reply]
I'm not really familiar with how the formatting of commons db reports work. I put my list through a quick awk script which yielded [2]. Bawolff (talk) 22:17, 18 February 2014 (UTC)[reply]
@Denniss: Would that work?  Hazard SJ  02:51, 23 February 2014 (UTC)[reply]
Yes. --Denniss (talk) 07:19, 23 February 2014 (UTC)[reply]
Could someone integrate this report into the Databse report overview site? --Denniss (talk) 12:45, 17 March 2014 (UTC)[reply]
Could someone please create a regular Bot job for this? Bawolff's page was last updated on March 12. --Denniss (talk) 09:12, 1 April 2014 (UTC)[reply]
@Steinsplitter: Try this. You're good at dynamic DBRs. --Zhuyifei1999 (talk) 10:53, 1 April 2014 (UTC)[reply]
@Bawolff: is there an ETA for your Special:ListDuplicatedFiles? --McZusatz (talk) 12:59, 1 April 2014 (UTC)[reply]
toollabs:steinsplitter/dupe.php (update every day, ~13:00 UTC) --Steinsplitter (talk) 14:37, 1 April 2014 (UTC)[reply]
Special:ListDuplicatedFiles should be available on April 4 (It was deployed today, but cached special pages only update every 3 days). Sorry about the script on my tool labs dying. Tool labs moved clusters, and decided not to transfer cron scripts, so all my repetitive tasks stopped working. Bawolff (talk) 21:13, 1 April 2014 (UTC)[reply]
The special page finally works! I was half expecting something bad to happen to prevent it from working. Bawolff (talk) 14:16, 4 April 2014 (UTC)[reply]

PD images from Flickr with a non-free license

Hello,

There are many public domain images on Flickr which have a non-free license. They can't be uploaded with the Upload Wizard. I wonder if it would be possible to use a bot. Examples:

What information do you need to upload all this with a bot? Yann (talk) 13:59, 30 November 2013 (UTC)[reply]

@Yann: you mean to use a bot to upload them? And how do the bot show that the image is not a copyvio? --Zhuyifei1999 (talk) 07:29, 8 December 2013 (UTC)[reply]
@Zhuyifei1999: I would check that. That's what I ask: if I make a list, could a bot upload them? Sorry if I was not clear. Actually, I was also asking if the bot could make a list. Upload all images where one of this photographer is mentioned in the description, and add a category "Files to be checked". Yann (talk) 09:36, 8 December 2013 (UTC)[reply]
@Yann: If there's a list of photos (or even better, urls), yes. If there isn't, or the bot have to google it, I'm afraid it would be hard for me to do so. Not sure about other bot operators, though. --Zhuyifei1999 (talk) 09:11, 9 December 2013 (UTC)[reply]
OK, I am going to make a list at User:Yann/PD images from Flickr with a non-free license. Yann (talk) 20:12, 11 December 2013 (UTC)[reply]
@Zhuyifei1999: OK, I made a list: User:Yann/PD images from Flickr with a non-free license. Could you start with that? And I found more photographers with the same case. Thanks in advance, Yann (talk) 19:28, 30 January 2014 (UTC)[reply]
Will do. --Zhuyifei1999 (talk) 06:21, 31 January 2014 (UTC)[reply]
I completed the first part: all photographers I had initially selected above. Yann (talk) 19:19, 31 January 2014 (UTC)[reply]
@Yann: How to bypass {{Flickrreview}} and show the image really is PD? --Zhuyifei1999 (talk) 06:55, 8 February 2014 (UTC)[reply]
 Doing… --Zhuyifei1999 (talk) 07:01, 8 February 2014 (UTC)[reply]

@Zhuyifei1999: Ok, thanks, tell me when you are done. Regards, Yann (talk) 12:28, 13 February 2014 (UTC)[reply]

Sorry, will take some time. I'm just busy with everything. --Zhuyifei1999 (talk) 11:10, 14 February 2014 (UTC)Timestamp (no archiving): Zhuyifei1999 (talk) 11:00, 6 March 2014 (UTC)[reply]
@Zhuyifei1999: Hi, What's the status of this request? Regards, Yann (talk) 14:43, 16 March 2014 (UTC)[reply]
It's mostly done, but technical problem with User:Yann/PD_images_from_Flickr_with_a_non-free_license, I'll get this done before or on Friday (and a test run and a BRFA after that). --Zhuyifei1999 (talk) 15:02, 16 March 2014 (UTC)[reply]
OK, thanks. Yann (talk) 04:11, 17 March 2014 (UTC)[reply]
@Yann: Commons:Bots/Requests/YiFeiBot_(17) Also what filename would you like for http://www.flickr.com/photos/28433765@N07/5014106447/ ? --Zhuyifei1999 (talk) 12:52, 21 March 2014 (UTC)[reply]

@Yann: @Yann: All listed ones ✓ Done except if it is detected as a duplicate. --Zhuyifei1999 (talk) 08:53, 28 March 2014 (UTC)[reply]

Uncategorized Bundesarchive images

The famous photo

Hello all. I think it'ld be reasonable to make a category of Bundesarchive images having no categories, except for the hidden, like this one. Obviously this is work for a bot. Thank you beforehands! Ain92 (talk) 10:53, 18 March 2014 (UTC)[reply]

✓ Done See catscan2 report. --Fæ (talk) 10:28, 21 March 2014 (UTC)[reply]

BadSVG

Hi there,

it would be nice if somebody could generate a list of all svg files (current version) that contain the string "xlink:href="data:image". Anybody up for that?

Cheers --Cwbm (commons) (talk) 11:54, 21 March 2014 (UTC)

 This will likely take weeks/months/years --Zhuyifei1999 (talk) 03:01, 22 March 2014 (UTC)[reply]

It would also be helpful if a bot could check the recent uploads and mark them with {{BadSVG}}. --Cwbm (commons) (talk) 08:36, 24 March 2014 (UTC)

Could you explain why this would be valuable and (considering the bandwidth cost) if there are any other non-controversial checks that could be usefully added at the same time? --Fæ (talk) 08:53, 24 March 2014 (UTC)[reply]
1) There are always new users who think that uploading raster files in a svg container is a good idea and user that convert existing pngs by uploading them in a svg container. Having a bot checking the recent uploads would enable to inform the uploader in a timely manner. 2) One could also check if the file is corrupt or if the svg is valid. But I don't know if that's reasonable. --Cwbm (commons) (talk) 11:25, 24 March 2014 (UTC)

As a method of filtering out suspect files, the following table was generated in pywikipediabot by looking at the latest 2,000 files uploaded and filtering out those with "svg" extensions and those over 50,000 bytes in size. It took under 10s to run. Do you think if we only checked SVGs of this size for further possible problems, this would be useful? A bot running this once an hour might be sufficient. --Fæ (talk) 13:27, 24 March 2014 (UTC)[reply]

You are going to miss some files, particularly logos which can be under 10 kb even as embedded raster graphics. But it's better than nothing. --Cwbm (commons) (talk) 13:37, 24 March 2014 (UTC)

I have dropped the old report from this discussion. A recent run is at User:Faebot/SandboxS. This goes through the last hour of uploads, to a maximum of 2,000 images (which appears plenty), picks out the SVG files over 5k in size, and then retrieves these files and examines their contents. If they match the regex "xlink..?href.?=.?.?data:image" then {{BadSVG}} is added to the image page. It seems a bit silly to load each matching SVG if this is the only test. It would be neat if there were several things to test while the bot is looking.
I am running the script locally rather than on WMF labs, which means it would probably be faster later. At the moment this run took 4 minutes to do but this represented the last 3 hours worth of uploaded images. So, say, running every 60 minutes back through the recently uploaded log, up until the point where it last ran to a maximum of 2,000 files in each slurp, would probably cope quite easily. --Fæ (talk) 21:55, 28 March 2014 (UTC)[reply]
First test run—running hourly, updating User:Faebot/SandboxS, I expect to leave this as a soak test over the weekend:
Images marked as BadSVG:
  1. 15:16, 31 March 2014 File:Cilician Armenia-hy.svg
  2. 15:16, 31 March 2014 File:KHW logo color.svg
  3. 00:17, 31 March 2014 File:Cannabis laws.svg
  4. 23:17, 30 March 2014 File:MH370 data graphs.svg
  5. 23:16, 30 March 2014 File:Biml logo.svg
  6. 20:16, 30 March 2014 File:IndoParthianKingdom.svg
  7. 17:16, 30 March 2014 File:Roman Empire 125 general map.SVG
  8. 16:16, 30 March 2014 File:Blason ville fr Érize-la-Brûlée (Meuse).svg
  9. 13:16, 30 March 2014 File:Roman Empire 125 political map.svg
  10. 09:16, 30 March 2014 File:Arms of the Viscounts of Villamur.svg
  11. 09:16, 30 March 2014 File:Greco-BactrianKingdomMap-es.svg
  12. 00:17, 30 March 2014 File:2014 Latakia Offensive Map.svg
  13. 23:17, 29 March 2014 File:Gundam Sentinel Wikipedia espanol.svg
  14. 23:16, 29 March 2014 File:Rif Aleppo2.svg
  15. 22:17, 29 March 2014 File:Battle of Qalamoun.svg
  16. 22:16, 29 March 2014 File:Einsatz Pu-12M im Regiment 9K33.svg
  17. 22:16, 29 March 2014 File:Battle of Daraa City.svg
  18. 20:17, 29 March 2014 File:IndoGreekKingdomAndCampaigns.svg
  19. 15:17, 29 March 2014 File:Battle of Marathon Initial Situation (it).svg
  20. 15:16, 29 March 2014 File:2.5 Russian road sign.svg
  21. 12:16, 29 March 2014 File:Major-Cultural-spheres.svg
  22. 11:16, 29 March 2014 File:South East Asia location-Naja-siamensis.svg
  23. 10:17, 29 March 2014 File:HUN Visegrád Címer.svg
  24. 23:16, 28 March 2014 File:Vostok Spacecraft Diagram.svg
✓ Done
I think this request can be marked as done for the moment. I will think about moving my working script over to labs at some point, though it will need little re-writing before then. The data volumes involved and the number of images that might be templated each day are relatively small, for this specific check. As it seems non-controversial, I will keep the bot running like this once an hour (with an average run time of under 60s). --Fæ (talk) 14:51, 31 March 2014 (UTC)[reply]

Thanks a lot for your effort. Cheers --Cwbm (commons) (talk) 06:35, 1 April 2014 (UTC)

Fixing typo "transfered" to "transferred" 2

Apologies for resurrecting a two-year old thread, but since this issue with the Upload Bot has been fixed, can someone contact User:Schlurcher on their dewiki page if they're still active and let them know they've got the go-ahead to run the bot to fix the typos, or perhaps write up a bot yourself for this task? TeleComNasSprVen (talk) 10:19, 30 March 2014 (UTC)[reply]

@TeleComNasSprVen: {{Transferred from}} should be used instead of simple text. --Ricordisamoa 11:05, 30 March 2014 (UTC)[reply]
I guess this would be more in the scope of an internationalization issue then --TeleComNasSprVen (talk) 17:06, 30 March 2014 (UTC)[reply]
Then, as SchlurcherBot 1, SamoaBot 3 and OgreBot 3. --Ricordisamoa 21:56, 30 March 2014 (UTC)[reply]

split category from user template

I have uploaded lots of files on which I use the templates user:Thryduulf/cc-by-sa-all or user:Thryduulf/copyleft which categorises the files into Category:Photos by Chris McKenna. I now wish to subcategorise some of my image uploads. The easiest way I can think to do this is if someone could go through all images tagged with those templates and explicitly add them to my category so I can subcategorise using cat-a-lot as I get time. When the run is complete (please advise me here or on my talk page) I will remove the categories from the templates. Thanks. Thryduulf (talk) 19:28, 31 March 2014 (UTC)[reply]

Anyone? Thryduulf (talk) 09:53, 7 April 2014 (UTC)[reply]
✓ Done You may want to play around with AWB if you don't want to install Pywikipediabot or find a work-around for large categories using VisualFileChange.js. I think we need more people with these tool skills. --Fæ (talk) 15:55, 8 April 2014 (UTC)[reply]
Thank you. I don't think I'm the right person for doing this sort of task - as a Linux user AWB isn't an realistic option (I've never successfully used wine for anything), I know no python, don't feel comfortable with the idea of running a bot and much of the VisualFileChange page is gibberish to me. Sorry. Thryduulf (talk) 20:39, 8 April 2014 (UTC)[reply]

Spaces before file extensions

Someone suggested on Commons:Village pump/Archive/2014/03#Spaces before file extensions in titles that a bot go looking around for filenames with spaces before the file extension and rename them to eliminate the space, while still retaining the redirect for reusers linking back to the file just in case. As a subtask, can a bot also look for instances of %E2%80%8E (hidden left-to-right mark U+200E) in filenames, or perhaps upload a test file containing the hidden character? TeleComNasSprVen (talk) 01:24, 9 April 2014 (UTC)[reply]

Sounds good to me. --99of9 (talk) 04:12, 9 April 2014 (UTC)[reply]
To clarify the subtask, I'd like to see if we can query the database to see if we have any left-to-right marks in either filenames or category names and if possible fix them. TeleComNasSprVen (talk) 06:08, 9 April 2014 (UTC)[reply]
Uploading such files are impossible as far as I know. Database query:  Doing… --Zhuyifei1999 (talk) 10:06, 9 April 2014 (UTC)[reply]
Query SELECT * FROM page WHERE page_title LIKE CONCAT("%", CHAR(0x200e USING utf8), "%"); → Empty set (37.66 sec) --Zhuyifei1999 (talk) 10:32, 9 April 2014 (UTC)[reply]

TIF files with jpeg compression

The image thumbnail generating process seems to have problems with .tif files using .jpeg compression (distorted colors). Is there an easy way to detect them and re-upload them as uncompressed version? --Denniss (talk) 14:39, 16 April 2014 (UTC)[reply]

 Making a list... The first 2 tiff images with jpeg compression is grayscale (no color problems) --Zhuyifei1999 (talk) 08:24, 17 April 2014 (UTC)[reply]
SELECT img_name FROM image WHERE img_minor_mime = "tiff" AND img_metadata LIKE '%s:11:"Compression";i:7;%' → User:Zhuyifei1999/sandbox (in fact, most, even colored photos look good) --Zhuyifei1999 (talk) 08:57, 17 April 2014 (UTC)[reply]
File:AloisianumLinzMaerz.tif looks fine but check the other thumb sizes,File:Alte Schule mit Pausenhalle, Eschelbronn.tif look completely wrong colored. See Commons:Village_pump#Change_of_colours_after_a_move and linked bugzilla report. Anyway I don't think it's a good idea to have TIF images using JPEG compression, the original intention of TIF is lossless. --Denniss (talk) 10:51, 17 April 2014 (UTC)[reply]
Could you resubmit the query with something like AND img_user_text <> \"Swiss Federal Archives\" as additional parameter? It seems like most of the compressed TIF were uploaded by the Swiss Federal Archives. Those images look mostly fine and are greyscale anyway. --McZusatz (talk) 12:44, 20 April 2014 (UTC)[reply]
✓ Done User:Zhuyifei1999/sandbox --Zhuyifei1999 (talk) 13:54, 20 April 2014 (UTC)[reply]

Freesound

This may be a late request, but can a bot please query the database for any files containing the string "(F/f)reesound" or similar that is not either linked to Template:Freesound or uploaded by me? I wish to replace such instance with the new template I've created, manually if need be and if the sources are accurate. TeleComNasSprVen (talk) 09:37, 20 April 2014 (UTC)[reply]

Retrieved from "https://commons.wikimedia.org/w/index.php?title=Commons:Bots/Work_requests&oldid=121813631"