Surface plasmon has the unique capability to concentrate light into subwavelength volume. Active plasmon devices using electrostatic gating can enable flexible control of the plasmon excitations, which has been demonstrated recently in terahertz plasmonic structures. Controlling plasmon resonance at optical frequencies, however, remains a significant challenge because gate-induced free electrons have very weak responses at optical frequencies. Here we achieve efficient control of near-infrared plasmon resonance in a hybrid graphene-gold nanorod system. Exploiting the uniquely strong and gate-tunable optical transitions of graphene, we are able to significantly modulate both the resonance frequency and quality factor of gold nanorod plasmon. Our analysis shows that the plasmon-graphene coupling is remarkably strong: even a single electron in graphene at the plasmonic hotspot could have an observable effect on plasmon scattering intensity. Such hybrid graphene-nanometallic structure provides a powerful way for electrical control of plasmon resonances at optical frequencies and could enable novel plasmonic sensing down to single charge transfer events.