We find for the first time that polarization mismatch of the sample and reference arms in optical-fiber-based optical coherence tomography (OCT) has critical effect on its depth resolution when the light source is partially polarized. When the polarization states of the two arms are matched, the measured point spread function (PSF) is almost identical to the theoretical prediction. When their polarization states are mismatched, the PSF can be so distorted that the depth resolution is degraded to several times the theoretical value. When we polarize the source light with a polarizer, then the degree of polarization (DOP) is unity, and the depth resolution becomes independent of the polarization mismatch. This discovery has fundamental importance for high-resolution OCT imaging of biological tissues. With DOP<1, the depth resolution can be quickly degraded by either birefringence or scattering in the sample. Adjusting polarization controllers can only improve the depth resolution at a certain depth in a sample if the polarization state of light changes along the depth. When DOP=1, uniform resolution along the depth of a sample can be achieved.