We describe the design and performance of a fiber-based confocal microscope for cryogenic operation. The microscope combines positioning at low temperatures along three space coordinates of millimeter translation and nanometer precision with high stability and optical performance at the diffraction limit. It was successfully tested under ambient conditions as well as at liquid nitrogen (77 K) and liquid helium (4 K) temperatures. The compact nonmagnetic design provides for long term position stability against helium refilling transfers, temperature sweeps, as well as magnetic field variation between -9 and 9 T. As a demonstration of the microscope performance, applications in the spectroscopy of single semiconductor quantum dots are presented.