It has been reported that the Mg-insufficient bone is fragile upon mechanical loading, despite its high bone mineral density, while vitamin K2 (MK-4: menatetrenone) improved the mechanical strength of Mg-insufficient bone. Therefore, we aimed to elucidate the ultrastructural properties of bone in rats with dietary Mg insufficiency with and without MK-4 supplementation. Morphological examinations including histochemistry, transmission electron microscopy, electron probe microanalysis (EPMA) and X-ray diffraction were conducted on the femora and tibiae of 4-week-old Wistar male rats fed with 1) a normal diet (control group, 0.09% Mg), 2) a Mg-insufficient diet (low Mg group, 0.006% Mg), or 3) a Mg-insufficient diet supplemented with MK-4 (MK-4 group, 0.006% Mg, 0.03% MK-4). MK-4 appeared to inhibit the osteoclastic bone resorption that is stimulated by Mg insufficiency. EPMA analysis, however, revealed an increased concentration of Ca paralleling Mg reduction in the low Mg group. Assessment by X-ray diffraction revealed an abundance of a particular synthetic form of hydroxyapatite in the low Mg group, while control bones featured a variety of mineralized crystals. In addition, Mg-deficient bones featured larger mineral crystals, i.e., crystal overgrowth. This crystalline aberration in Mg-insufficient bones induced collagen fibrils to mineralize easily, even in the absence of mineralized nodules, which therefore led to an early collapse of the fibrils. MK-4 prevented premature collagen mineralization by normalizing the association of collagen fibrils with mineralized nodules. Thus, MK-4 appears to rescue the impaired collagen mineralization caused by Mg insufficiency by promoting a re-association of the process of collagen mineralization with mineralized nodules.