Cellular and molecular characterization of immune responses elicited by influenza virus infection and seasonal vaccination have informed efforts to improve vaccine efficacy, breadth, and longevity. Here, we use negative stain electron microscopy polyclonal epitope mapping (nsEMPEM) to structurally characterize the humoral IgG antibody responses to hemagglutinin (HA) from human patients vaccinated with a seasonal quadrivalent flu vaccine or infected with influenza A viruses. Our data show that both vaccinated and infected patients had humoral IgGs targeting highly conserved regions on both H1 and H3 subtype HAs, including the stem and anchor, which are targets for universal influenza vaccine design. Responses against H1 predominantly targeted the central stem epitope in infected patients and vaccinated donors, whereas head epitopes were more prominently targeted on H3. Responses against H3 were less abundant, but a greater diversity of H3 epitopes were targeted relative to H1. While our analysis is limited by sample size, on average, vaccinated donors responded to a greater diversity of epitopes on both H1 and H3 than infected patients. These data establish a baseline for assessing polyclonal antibody responses in vaccination and infection, providing context for future vaccine trials and emphasizing the importance of carefully designing vaccines to boost protective responses towards conserved epitopes.