A pencil beam algorithm as a component of an optimization algorithm for intensity modulated proton therapy (IMPT) is presented. The pencil beam algorithm is tuned to the special accuracy requirements of IMPT, where in heterogeneous geometries both the position and distortion of the Bragg peak and the lateral scatter pose problems which are amplified by the spot weight optimization. Heterogeneity corrections are implemented by a multiple raytracing approach using fluence-weighted sub-spots. In order to derive nuclear interaction corrections, Monte Carlo simulations were performed. The contribution of long ranged products of nuclear interactions is taken into account by a fit to the Monte Carlo results. Energy-dependent stopping power ratios are also implemented. Scatter in optional beam line accessories such as range shifters or ripple filters is taken into account. The collimator can also be included, but without additional scattering. Finally, dose distributions are benchmarked against Monte Carlo simulations, showing 3%/1 mm agreement for simple heterogeneous phantoms. In the case of more complicated phantoms, principal shortcomings of pencil beam algorithms are evident. The influence of these effects on IMPT dose distributions is shown in clinical examples.