Twiddle factor is considered to be the dominant error source of frequency estimate by Fourier transformation (FT), and thus impacts the accuracy in FT-based heterodyne velocimetry. Here we report a novel data analysis method for heterodyne velocimetry, which utilizes the change law of frequency errors with signals' phases to develop twiddle factor neutralization method, improving the performance of heterodyne velocimetry. Numerical simulations show that this method can improve velocity resolution by many times as compared to the boxcar, Hamming, and Hann window functions under different noise conditions. A 90° optical hybrid is used to generate four phase shifted signals for this method, and 1 m/s level velocity resolution and 100 ps level time resolution are simultaneously achieved in laser-shock experiments.