In this study, we investigated the genetic and environmental origin of individual differences in respiratory sinus arrhythmia (RSA) during rest and during four stress tasks. We used a multivariate model including age, RSA, and respiration rate. Participants were 208 male and female pairs of middle-aged twins. A model without sex differences, specifying additive genetic and unique environmental factors, showed the best fit across all conditions. Heritability of RSA ranged from 28% to 43%. Correction for respiration rate yielded RSA heritabilities of similar size. The covariance between respiration rate and RSA was best explained by a combination of correlated unique environmental and correlated additive genetic factors. Combined with data from an earlier project, RSA from 317 adolescent and 712 middle-aged individuals of both sexes was available. This large data set showed that (a) sex differences in mean RSA are absent and (b) RSA decreases considerably from adolescence (111.5 ms) to middle age (60.0 ms).