In genome-wide association studies, failure to remove variation due to population structure results in spurious associations. In contrast, for predictions of future phenotypes or estimated breeding values from dense SNP data, exploiting population structure arising from relatedness can actually increase the accuracy of prediction in some cases, for example, when the selection candidates are offspring of the reference population where the prediction equation was derived. In populations with large effective population size or with multiple breeds and strains, it has not been demonstrated whether and when accounting for or removing variation due to population structure will affect the accuracy of genomic prediction. Our aim in this study was to determine whether accounting for population structure would increase the accuracy of genomic predictions, both within and across breeds. First, we have attempted to decompose the accuracy of genomic prediction into contributions from population structure or linkage disequilibrium (LD) between markers and QTL using a diverse multi-breed sheep (Ovis aries) data set, genotyped for 48,640 SNP. We demonstrate that SNP from a single chromosome can achieve up to 86% of the accuracy for genomic predictions using all SNP. This result suggests that most of the prediction accuracy is due to population structure, because a single chromosome is expected to capture relationships but is unlikely to contain all QTL. We then explored principal component analysis (PCA) as an approach to disentangle the respective contributions of population structure and LD between SNP and QTL to the accuracy of genomic predictions. Results showed that fitting an increasing number of principle components (PC; as covariates) decreased within breed accuracy until a lower plateau was reached. We speculate that this plateau is a measure of the accuracy due to LD. In conclusion, a large proportion of the accuracy for genomic predictions in our data was due to variation associated with population structure. Surprisingly, accounting for this structure generally decreased the accuracy of across breed genomic predictions.