Fungus-growing termites live in obligate mutualistic symbiosis with species of the basidiomycete genus Termitomyces, which are cultivated on a substrate of dead plant material. When the termite colony dies, or when nest material is incubated without termites in the laboratory, fruiting bodies of the ascomycete genus Xylaria appear and rapidly cover the fungus garden. This raises the question whether certain Xylaria species are specialised in occupying termite nests or whether they are just occasional visitors. We tested Xylaria specificity at four levels: (1) fungus-growing termites, (2) termite genera, (3) termite species, and (4) colonies. In South Africa, 108 colonies of eight termite species from three termite genera were sampled for Xylaria. Xylaria was isolated from 69% of the sampled nests and from 57% of the incubated fungus comb samples, confirming high prevalence. Phylogenetic analysis of the ITS region revealed 16 operational taxonomic units of Xylaria, indicating high levels of Xylaria species richness. Not much of this variation was explained by termite genus, species, or colony; thus, at level 2-4 the specificity is low. Analysis of the large subunit rDNA region, showed that all termite-associated Xylaria belong to a single clade, together with only three of the 26 non-termite-associated strains. Termite-associated Xylaria thus show specificity for fungus-growing termites (level 1). We did not find evidence for geographic or temporal structuring in these Xylaria phylogenies. Based on our results, we conclude that termite-associated Xylaria are specific for fungus-growing termites, without having specificity for lower taxonomic levels.