Incorporating computer vision on smart phone photographs into screening for inflammatory arthritis: results from an Indian patient cohort

Background: Convolutional neural networks (CNNs) are increasingly used to classify medical images, few studies utilize smartphone photographs. We assessed CNNs for differentiating patients from controls and detecting joint inflammation.

Methods: We included consecutive patients with early inflammatory arthritis and healthy controls, all examined by a rheumatologist (15% by two). Standardized hand photographs of the hands were taken, anonymized, and cropped around joints. Pre-trained CNN models were fine-tuned on our dataset (80% training; 20% test set). We used an Inception-ResNet-v2 backbone CNN modified for two class outputs (Patient vs Control) on uncropped photos. Separate Inception-ResNet-v2 CNNs were trained on cropped photos of Middle finger Proximal Interphalangeal (MFPIP), Index finger PIP (IFPIP) and wrist. We report accuracy, sensitivity and specificity and Area under the curve(AUC) of Receiver Operating Curves(ROC).

Results: We analysed 800 hands from 200 controls (mean age 37.8) and 200 patients (mean age 49 years). Two rheumatologists showed 0.89 concordance. The wrist was commonly involved (173/400) followed by the MFPIP (134) and IFPIP (128). The screening CNN achieved 99% accuracy and specificity and 98% sensitivity in predicting a patient compared with controls. Joint-specific CNN accuracy, sensitivity and specificity and AUC were as follows: wrist (75%, 92%, 72%, 0.86), IFPIP (73%, 89%, 72%, 0.88) and MFPIP (71%, 91%, 70%, 0.87).

Conclusion: Computer vision distinguishes patients and controls using smartphone photographs, showing promise as a screening tool. Future research will focus on validating findings in diverse populations, other joints and integrating this technology into clinical workflows.