We present a global payoff-based strategy updating model for studying cooperative behavior of a networked population. We adopt the Prisoner's Dilemma game and the snowdrift game as paradigms for characterizing the interactions among individuals. We investigate the model on regular, small-world, and scale-free networks, and find multistable cooperation states depending on the initial cooperator density. In particular for the snowdrift game on small-world and scale-free networks, there exist a discontinuous phase transition and hysteresis loops of cooperator density. We explain the observed properties by theoretical predictions and simulation results of the average number of neighbors of cooperators and defectors, respectively. Our work indicates that individuals with more neighbors have a trend to preserve their initial strategies, which has strong impacts on the strategy updating of individuals with fewer neighbors; while the fact that individuals with few neighbors have to become cooperators to avoid gaining the lowest payoff plays significant roles in maintaining and spreading of cooperation strategy.