The Use of Chatbots in Head and Neck Mucosal Malignancy Treatment Recommendations

Objective: As cancer patients increasingly use chatbots, it is crucial to recognize ChatGPT's potential in enhancing health literacy while ensuring validation to prevent misinformation. This study aims to assess ChatGPT-3.5's capability to provide appropriate staging and treatment recommendations for head and neck mucosal malignancies for vulnerable populations.

Study design and setting: Forty distinct clinical vignettes were introduced into ChatGPT to inquire about staging and treatment recommendations for head and neck mucosal malignancies.

Methods: Prompts were created based on head and neck cancer (HNC) disease descriptions (cancer location, tumor size, lymph node involvement, and symptoms). Staging and treatment recommendations according to the 2021 National Comprehensive Cancer Network (NCCN) guidelines were scored by three fellowship-trained HNC surgeons from two separate tertiary care institutions. HNC surgeons assessed the accuracy of staging and treatment recommendations, such as the completeness of surgery and the appropriateness of treatment modality.

Results: Whereas ChatGPT's responses were 95% accurate at recommending the correct first-line treatment based on the 2021 NCCN guidelines, 55% of the responses contained inaccurate staging. Neck dissection was incorrectly omitted from treatment recommendations in 50% of the cases. Moreover, 40% of ChatGPT's treatment recommendations were deemed unnecessary.

Conclusion: This study emphasizes ChatGPT's potential in HNC patient education, aligning with NCCN guidelines for mucosal malignancies, but highlights the importance of ongoing refinement and scrutiny due to observed inaccuracies in tumor, nodal, metastasis staging, incomplete surgery options, and inappropriate treatment recommendations. Otolaryngologists can use this information to caution patients, families, and trainees regarding the use of ChatGPT for HNC education without expert guidance.