The T cell receptor (TCR), despite its importance, is underutilized in single-cell analysis, with gene expression features solely driving current strategies. Here, we argue for a TCR-first approach, more suited toward T cell repertoires. To this end, we curated a large T cell atlas from 12 prominent human studies, containing in total 500,000 T cells spanning multiple diseases, including melanoma, head and neck cancer, blood cancer, and lung transplantation. Here, we identified severe limitations in cell-type annotation using unsupervised approaches and propose a more robust standard using a semi-supervised method or the TCR arrangement. We showcase the utility of a TCR-first approach through application of the STEGO.R tool for the identification of treatment-related dynamics and previously unknown public T cell clusters with potential antigen-specific properties. Thus, the paradigm shift to a TCR-first can highlight overlooked key T cell features that have the potential for improvements in immunotherapy and diagnostics.