Accumulation of intramyocellular lipid (IMCL) is observed in individuals with insulin resistance as well as insulin-sensitive endurance athletes with high peak oxygen consumption (VO2peak), which is called the athlete's paradox. It remains unclear whether non-athletes with higher fitness levels have IMCL accumulation and higher insulin sensitivity in general. In this study, we investigated the association between IMCL accumulation and muscle insulin sensitivity (M-IS) in subjects with high or low VO2peak. We studied 61 nonobese (BMI, 23 to 25 kg/m2), non-athlete Japanese men. We divided the subjects into four groups based on the median value of VO2peak and IMCL in the soleus muscle. We evaluated M-IS using a two-step hyperinsulinemic-euglycemic clamp. Among subjects with higher VO2peak (n = 32), half of those (n = 16) had lower IMCL levels. Both High-VO2peak groups had higher M-IS than the Low-VO2peak groups. On the other hand, M-IS was comparable between the High-VO2peak/High-IMCL and High-VO2peak/Low-IMCL groups, whereas the High-VO2peak/High-IMCL group had IMCL levels that were twice as high as those in the High-VO2peak/Low-IMCL group. On the other hand, the High-VO2peak/High-IMCL group had significantly higher physical activity levels (approximately 1.8-fold) than the other three groups. In conclusion, in nonobese, non-athlete Japanese men, subjects with higher VO2peak and higher IMCL had higher physical activity levels. IMCL accumulation is not associated with insulin resistance in individuals with higher or lower fitness levels.