Though the total lymphocyte density has been implicated in the pathogenesis of colorectal cancer (CRC), the clinical implications of lymphocyte subsets in CRC patients remain unclear yet. In this study, we used tissue microarrays and immunohistochemistry to evaluate the density of tumor-infiltrating CD4(+) and CD8(+) T cells as well as CD20(+) B cells in the central region and the invasive margin (IM) of tumor tissue from 67 CRC patients and then analyzed the clinical implications. We found that the subsets of tumor-infiltrating lymphocyte had no relationship with patients' age, tumor size, N category, and tumor stage. However, the density of B cells in the tumor center (TC) of colorectal patients in T1/T2 category was much higher than that of patients in T3/T4 category (P = 0.008). In addition, patients in M0 category had a higher density of CD8(+) T cells in the IM but not in the TC compared to patients in M1 category (P = 0.03). Results in this study demonstrate that higher CD8(+) T cell density in the IM of tumor is associated with lower tumor metastasis and higher B cell density in TC is associated with lower tumor T category. The results also suggest that the analysis of infiltrating lymphocyte subsets is important for understanding the detailed mechanisms of tumor pathogenesis and exploring potential interventions for the tumor.