In this paper, we analyze the dependence of the output signal spectral lineshape and intensity on the modulation, waveform and modulation index in first- and second-harmonic detection of triangle modulation, square demodulation and saw modulation square demodulation. The signal lineshape in the case of saw modulation, square demodulation is quite different from, whereas the signal lineshape in the case of triangle modulation, square demodulation is similar to that in traditional sine modulation, sine demodulation. Experiments of tunable IR laser frequency modulation absorption spectrum of methane are also done to verify the theoretical analysis. Then we reach a conclusion; the lineshape and intensity of frequency modulation spectrum can be influenced by modulation waveform, demodulation waveform and modulation index. Calculation results can be used to choose better experimental parameters and optimize experimental designs.