Semiconductor saturable absorber mirrors (SESAMs) using quantum dot (QD) absorbers exhibit a larger design freedom than standard quantum well absorbers. The additional parameter of the dot density in combination with the field enhancement allows for an independent control of saturation fluence and modulation depth. We present the first detailed study of the effect of QD growth parameters and post growth annealing on the macroscopic optical SESAM parameters, measuring both nonlinear reflectivity and recombination dynamics. We studied a set of self-assembled InAs QD-SESAMs optimized for an operation wavelength around 960 nm with varying dot density and growth temperature. We confirm that the modulation depth is controlled by the dot density. We present design guidelines for QD-SESAMs with low saturation fluence and fast recovery, which are for example important for modelocking of vertical external cavity surface emitting lasers (VECSELs).