Although there is an emerging consensus that both musical and linguistic pitch processing can be problematic for individuals with a developmental disorder termed congenital amusia, the nature of such a pitch-processing deficit, especially that demonstrated in a speech setting, remains unclear. Therefore, this study tested the performance of native Mandarin speakers, both with and without amusia, on discrimination and imitation tasks for Cantonese level tones, aiming to shed light on this issue. Results suggest that the impact of the phonological deficit, coupled with that of the domain-general pitch deficit, could provide a more comprehensive interpretation of Mandarin amusics' speech impairment. Specifically, when there was a high demand for pitch sensitivity, as in fine-grained pitch discriminations, the operation of the pitch-processing deficit played the more predominant role in modulating amusics' speech performance. But when the demand was low, as in discriminating naturally produced Cantonese level tones, the impact of the phonological deficit was more pronounced compared to that of the pitch-processing deficit. However, despite their perceptual deficits, Mandarin amusics' imitation abilities were comparable to controls'. Such selective impairment in tonal perception suggests that the phonological deficit more severely implicates amusics' input pathways.