The sex chromosomes of the tropical crop papaya (Carica papaya) are evolutionarily young and consequently allow for the examination of evolutionary mechanisms that drive early sex chromosome divergence. We conducted a molecular population genetic analysis of four X/Y gene pairs from a collection of 45 wild papaya accessions. These population genetic analyses reveal striking differences in the patterns of polymorphism between the X and Y chromosomes that distinguish them from other sex chromosome systems. In most sex chromosome systems, the Y chromosome displays significantly reduced polymorphism levels, whereas the X chromosome maintains a level of polymorphism that is comparable to autosomal loci. However, the four papaya sex-linked loci that we examined display diversity patterns that are opposite this trend: the papaya X alleles exhibit significantly reduced polymorphism levels, whereas the papaya Y alleles maintain greater than expected levels of diversity. Our analyses suggest that selective sweeps in the regions of the X have contributed to this pattern while also revealing geographically restricted haplogroups on the Y. We discuss the possible role sexual selection and/or genomic conflict have played in shaping the contrasting patterns of polymorphism found for the papaya X and Y chromosomes.