Telemedicine has potential to improve the delivery, quality, and accessibility of ophthalmic care for infants with Retinopathy of Prematurity (ROP). Using a telemedicine screening strategy, three potential diagnostic cutoffs may be used to define disease that warrants ophthalmologic referral: presence of any ROP, presence of moderate ("type-2 prethreshold") ROP, or presence of severe ROP requiring treatment. This study examines the relationship between accuracy and reliability of diagnosis by three masked ophthalmologist graders at each of these diagnostic cutoffs. The sensitivity, specificity, inter-grader reliability, and intra-grader reliability showed significant variation depending on the diagnostic cutoff, with best results at cutoffs of type-2 prethreshold ROP or treatment-requiring ROP. Before the large-scale adoption of telemedicine for image-based screening of diseases such as ROP, standards defining clinically-relevant referral cutoffs must be established, and diagnostic accuracy and reliability at these cutoffs must be characterized.