We examine the implication of intracavity nonlinearity for harmonic mode locking (HML) by exploiting highly nonlinear fiber in a carbon nanotube film mode-locked Er-doped fiber laser. It is found that the reasonably large nonlinearity is of benefit to increase the extent of harmonic order while the excessive nonlinearity leads to some peculiar multi-pulse patterns such as noise-like pulse and soliton rain. Via appropriate nonlinearity management, nearly 4 GHz repetition rate pulses at the 91st harmonic with 936 fs pulse duration are delivered under the pump power of 280 mW. The pulse stability is evidenced by the super-mode suppression ratio of 35.6 dB. To the best of our knowledge, it is the highest repetition rate yet reported for a passively HML fiber laser based on a film-type physical saturable absorber. Furthermore, the laser exhibits steep pumping efficiency slope of ${\gt}{19}\;{\rm MHz/mW}$, which is also a record among all of the passively HML fiber lasers.