Help:Toolforge/Developing successful tools

Toolforge

• Cloud Services overview
• Toolforge user docs
• Toolforge changelog

Get started

• Quickstart: set up and get access
• How Toolforge works
• Rules you must follow
• Tutorials

Build and run tools

• Navigate tool accounts and files
• Build container images for tools
• Run a web service
• Schedule and manage jobs
• Manage tool runtime configuration (envvars)
• Language-specific details:
  • Python
  • Pywikibot
  • Node.js
  • PHP
  • ...more languages/frameworks
• Use Redis for caching
• Index content with Elasticsearch

Access shared storage and databases

• Access shared storage and public wiki dumps
• Access the Wiki Replicas databases
• Access replica search indices
• Manage tool databases

Share and maintain tools

• Set up version control and code review
• Develop successful tools
• Find and share tools on Toolhub
• Delete a tool
• Contribute to Toolforge

Get help

• How and where to get help
• Troubleshooting

Useful links

• Toolforge admin docs
• List of tools
• Toolforge Admin Console (toolsadmin)

[]

Tips for developing a successful tool

• License your source code and document that with a LICENSE or COPYING file in the tool's home directory and header comments in the source code. See Help:Toolforge/Developing § Licensing your source code for more help on why and how to select a license.
• Use public version control (gerrit, diffusion, GitHub, Bitbucket, ...) for your tool's source code and deploy changes to the Toolforge servers by updating a checkout of that public version control. See Help:Toolforge § Setting up code review and version control for additional information.
• Keep passwords and other credentials (OAuth secrets, etc) separated from the main application code so that they are not exposed publicly in your version control system of choice.
• Create a page in the Tool: namespace documenting the basics of what your tool does and how to start and stop it.
• Find co-maintainers for your tools who can help out at least with starting/stopping jobs when needed.
• Make many small tools that each do one specific task rather than a catch-all tool that does many different tasks.

Licensing your source code

All code in the ‘tools’ project must be published under an OSI approved open source license. Please add a license at the beginning!

The absence of a license means that default copyright laws apply. Without a clear license you are implicitly claiming copyright without providing an explanation of the rights you are willing to grant to others who wish to use or modify your software. This means that you retain all rights to your source code and that nobody else may reproduce, distribute, or create derivative works from your work until standard copyright lapses. In the United States today that means until 70 years after your death. This is counter to the general principles of the Wikimedia movement.

The two easiest choices are GPL-2.0+ if you want to ensure that all derivative works are made available under the same license terms (this is the license used for MediaWiki itself) or MIT if you only want to ensure that your original work is mentioned in a derivative project. This is a gross simplification of course. See choosealicense.com as a light primer on choosing a license.

Heavy processing

If you will be doing heavy processing (e.g., compiles or tool test runs), please use the development environment (tools-dev.wmflabs.org) instead of the primary login host (tools-login.wmflabs.org) so as to help maintain the interactive performance of the primary login host.

The tools-dev host is functionally identical to tools-login.

Even when running on tools-dev, your processes may be killed without notice by system administrators or automated watch processes if performance of the shared server is severely impacted. Using the job grid is recommended for any heavy processing.

Where to put shared Tool code and files

Sharing files via packages or version control

• Package shared libraries using the package manager for your implementation language: PHP Composer packages, Python PyPI packages, Ruby gems, etc. This is the recommended and most portable and future-proof method of sharing code with multiple projects.
• Shared code can be stored in git submodules, which allow users to keep a git repository within another git repository. Sharing code in this way retains the maintainability and other source controls advantages of git. For more information about git submodules, please see the git documentation.

Sharing files via NFS

Programs running on the job grid and in Kubernetes webservice containers have access to shared NFS directories. This shared access can be used to share files between multiple tool accounts. This is the least portable method of sharing and may not be supported in all future Toolforge services.

• Access to a tool's code can be delegated to other tools by adding the tools as maintainers of the tool that is hosting the code. This can be done at https://toolsadmin.wikimedia.org/tools/id/<tool>/maintainers/. (It may be appropriate to create a new tool to house the shared code.)
• Shared config or other files may be placed in the /data/project/shared directory, which is readable (and potentially writeable) by all Toolforge tools and users. In this directory are available, for instance:
  • a full MediaWiki checkout (core and all extensions in gerrit)

How to use <programming-language-X> to write tools on Toolforge

Do you have experience that might help another user? Please share it (or point to it) here!

• Help:Toolforge/Java
• Help:Toolforge/Mono
• Help:Toolforge/Pywikibot

Include useful links on a tool's public webpage

If your tool is a web application, you should include some useful links on it for users. For example:

• Feedback page link,
• Documentation link,
• Source code link and license,
• Authors name(s).

See more ideas and examples in phab:T202949.