There is ongoing debate whether the efficiency of local cognitive processes leads to global cognitive ability or whether global ability feeds the efficiency of basic processes. A prominent example is the well-replicated association between inspection time (IT), a measure of perceptual discrimination speed, and intelligence (IQ), where it is not known whether increased speed is a cause or consequence of high IQ. We investigated the direction of causation between IT and IQ in 2012 genetically related subjects from Australia and The Netherlands. Models in which the reliable variance of each observed variable was specified as a latent trait showed IT correlations of -0.44 and -0.33 with respective Performance and Verbal IQ; heritabilities were 57% (IT), 83% (PIQ) and 77% (VIQ). Directional causation models provided poor fits to the data, with covariation best explained by pleiotropic genes (influencing variation in both IT and IQ). This finding of a common genetic factor provides a better target for identifying genes involved in cognition than genes which are unique to specific traits.