Evaluating health information technologies: A systematic review of framework recommendations

Objective: Evaluating Health Information Technologies (HITs) can be challenging, but studies are necessary so that the most beneficial interventions can be identified. Our objective was to systematically review the available recommendations for improving the methods used in HIT evaluations.

Methods: HIT evaluation frameworks were identified from database (MEDLINE, EMBASE, CINAHL) and grey literature searches. Outcome measures included framework recommendations and characteristics. Recommendations were coded and organised using thematic analysis methods. A scoring instrument was used to measure framework quality.

Results: The search identified 23 frameworks and 272 recommendations. These were organised into five evaluation domains and 42 themes. The themes included recommendations for improving the evaluation of technical aspects of HITs (e.g. describing aspects of HIT functionality) and suggestions for improving the evaluation of complex factors that may influence the overall effects of HITs (e.g. careful reporting of whether the HIT became integrated into existing working patterns). The frameworks were not generally developed in association with healthcare professionals, or with input from patients. The frameworks tended not to have been developed using systematic methods designed to reduce the risk of bias.

Discussion: HIT evaluations are important but they are challenging to conduct and appraise. This review was conducted using systematic methods enabling the organisation of framework recommendations into key themes. These findings may help investigators to successfully plan, conduct and appraise HIT evaluations. The quality appraisal demonstrated that HIT evaluation research may be improved by using more systematic methods and the involvement of participants from a range of differing backgrounds.