Phyllodes tumors of the breast are biphasic stromal and epithelial tumors histologically similar to benign fibroadenomas, but with a neoplastic stromal component. In contrast to fibroadenoma, phyllodes tumors can recur and be locally aggressive or be malignant. This study uses SNP array analysis to present a genome-wide map of loss of heterozygosity (LOH) in a cohort of phyllodes tumors and fibroadenomas. LOH is frequent and sometimes extensive in phyllodes tumors, but is rarely seen in fibroadenomas. There is heterogeneity between phyllodes tumors of different patients and no one LOH marker identifies a majority of these lesions. However, a subset of LOH loci occur in multiple cases of phyllodes tumors and are not found in fibroadenomas. Primary phyllodes tumors and paired recurrences from the same patient share common regions of LOH. In contrast, metachronous fibroadenomas from the same patient have different LOH patterns with no indication of a shared origin. Specific LOH loci may be associated with pathologic progression in recurrent phyllodes tumors. In a single case of phyllodes tumor containing a malignant epithelial component the malignant epithelium and stroma partially share an LOH genotype, suggesting a common precursor cell for the biphasic malignant components.