Diffusion MRI is a powerful tool for non-invasive probing of brain tissue microstructure. Recent multi-center efforts in the acquisition and analysis of diffusion MRI data significantly increase sample sizes and hence improve sensitivity and reliability in detecting subtle changes associated with development, aging, and diseases. However, discrepancies resulting from different scanner vendors, acquisition protocols, and image reconstruction algorithms can cause data incompatibility across imaging centers. In this paper, we introduce a model-free method that is based on the method of moments for the direct harmonization of diffusion MRI data to reduce site-specific variations. Our method directly harmonizes diffusion-attenuated signal without the need to fit any diffusion model. Moreover, our method allows the explicit definition of well-behaved mapping functions with properties such as invertibility, smoothness, and injectivity. We show that our method is effective in lowering the variations of diffusion scalars of traveling human phantoms scanned at different sites from 1%-3% to less than 0.9% for fractional anisotropy (FA) and mean diffusivity and from 1%-2.5% to 0.3%-1.2% for generalized FA. We also demonstrate its ability in preserving individual differences and in increasing across-site consistency in tractography and white matter connectivity.