Frequency-modulated continuous-wave (FMCW) light detection and ranging (LIDAR), which offers high depth resolution and immunity to environmental disturbances, has emerged as a strong candidate technology for active imaging applications. In general, hundreds of photons per pixel are required for accurate three-dimensional (3D) imaging. When it comes to the low-flux regime, however, depth estimation has limited robustness. To cope with this, we propose and demonstrate a photon-efficient approach for FMCW LIDAR. We first construct a FMCW LIDAR setup based on single-photon detectors where only a weak local oscillator is needed for the coherent detection. Further, to realize photon-efficient imaging, our approach borrows the data from neighboring pixels to enhance depth estimates, and employs a total-variation seminorm to smooth out the noise on the recovered depth map. Both simulation and experiment results show that our approach can produce high-quality 3D images from ∼10 signal photons per pixel, increasing the photon efficiency by 10-fold over the traditional processing method. The high photon efficiency will be valuable for low-power and rapid FMCW applications.