The evaluation of health recommender systems: A scoping review

Background: People often look online for information about health concerns, but the vast amount of available and unregulated content can cause misinformation and potential harm. Health recommender systems (HRSs) can address this issue by recommending personalised health information. Previous research has evaluated individual systems, but there is a lack of reviews synthesising their evaluation findings. Such a synthesis is needed to ensure that future recommender designs have a positive impact on target health or behavioural outcomes.

Objective: This review aimed to provide a summary of the evidence obtained from previous studies evaluating HRSs and highlight methodological considerations and gaps in the current research.

Methods: The review was developed using the PRISMA-ScR and PICOS frameworks. PubMed, ACM Digital Library, IEEE Xplore, Web of Science, ScienceDirect, and Scopus were searched for studies that evaluated at least one HRS and involved human participants. A descriptive analysis was conducted on included studies and key themes and gaps in the literature were assessed.

Results: 36 papers evaluating 34 HRSs were included. The systems targeted 13 different health conditions and provided different types of recommendations. Evaluation designs varied, with sample sizes ranging from 1 to 8057, and study durations from a single session to three years. A variety of outcome measures were used, including accuracy, engagement, clinical or behavioural outcomes, and participant perspectives.

Conclusions: The number of studies about HRSs is increasing, but there is a distinct lack of robust scientific research. The heterogeneity of outcome measures made it difficult to draw conclusions about their efficacy, but the data suggest that HRSs can help with the self-management of a wide range of conditions. There is a need to strengthen the available early-stage evidence with further research, evaluating multiple outcome measures including clinical outcomes, usability, and acceptability over a longer period to show real-world impact.