Ultrasound (US) is a relatively new imaging modality in rheumatology that offers great potential as a diagnostic and management tool. In 2004, an OMERACT Ultrasound Special Interest Group was formed to address the metric qualities of US as a potential outcome measure. A preliminary systematic review highlighted the deficiencies in the literature, particularly with regard to the reliability of interpreting and acquiring images; as a consequence, a number of exercises were proposed to address these issues. This report describes a series of iterative studies that have resulted in improved intra- and inter-reader reliability for detecting and scoring synovitis from both static and real-time images of the hand joints of patients with rheumatoid arthritis. The reliability of acquiring images was also enhanced using standardized positions. Future studies will assess the value of US in clinical trials.