Being a key feature of a glassy state, low temperature relaxation has important implications on the mechanical behavior of glasses; however, the mechanism of low temperature relaxation is still an open issue, which has been debated for decades. By systematically investigating the influences of cooling rate and pressure on low temperature relaxation in the Zr50Cu50 metallic glasses, it is found that even though pressure does induce pronounced local structural change, the low temperature-relaxation behavior of the metallic glass is affected mainly by cooling rate, not by pressure. According to the atomic displacement and connection mode analysis, we further demonstrate that the low temperature relaxation is dominated by the dispersion degree of fast dynamic atoms rather than the most probable atomic nonaffine displacement. Our finding provides the direct atomic-level evidence that the intrinsic heterogeneity is the key factor that determines the low temperature-relaxation behavior of the metallic glasses.