The computation of model matrix in the iterative imaging reconstruction process is crucial for the quantitative photoacoustic tomography (PAT). However, it is challenging to establish an outstanding model matrix to improve the overall imaging quality in PAT due to the noisy signal acquisition and inevitable artifacts. In this work, we present a novel method, named as the curve-driven-based model-matrix inversion (CDMMI), to calculate the model matrix for tomographic reconstruction in photoacoustic imaging. It eliminated the use of interpolation techniques, and thus avoided all interpolation related errors. The conventional interpolated-matrix-model inversion (IMMI) method was applied to evaluate its performance in numerical simulation, tissue-mimicking phantom and in vivo small animal studies. Results demonstrated that CDMMI achieved better reconstruction accuracy until IMMI kept increasing discrete points to 10000. Furthermore, the proposed method can suppress the negative influence of noise and artifacts effectively, which benefited the overall imaging quality of photoacoustic tomography.