We study numerically a family of surface growth models that are known to be in the universality class of the Kardar-Parisi-Zhang equation when driven by uncorrelated noise. We find that, in the presence of noise with power-law temporal correlations with exponent θ, these models exhibit critical exponents that differ both quantitatively and qualitatively from model to model. The existence of a threshold value for θ below which the uncorrelated fixed point is dominant occurs for some models but not for others. In some models the dynamic exponent z(θ) is a smooth decreasing function, while it has a maximum in other cases. Despite all models sharing the same symmetries, critical exponents turn out to be strongly model dependent. Our results clearly show the fragility of the universality class concept in the presence of long-range temporally correlated noise.