The Jumonji C domain is a catalytic motif that mediates histone lysine demethylation. The Jumonji C-containing oxygenase JMJD2A specifically demethylates tri- and dimethylated lysine-9 and lysine-36 of histone 3 (H3K9/36 me3/2). Here we present structures of the JMJD2A catalytic core complexed with methylated H3K36 peptide substrates in the presence of Fe(II) and N-oxalylglycine. We found that the interaction between JMJD2A and peptides largely involves the main chains of the enzyme and the peptide. The peptide-binding specificity is primarily determined by the primary structure of the peptide, which explains the specificity of JMJD2A for methylated H3K9 and H3K36 instead of other methylated residues such as H3K27. The specificity for a particular methyl group, however, is affected by multiple factors, such as space and the electrostatic environment in the catalytic center of the enzyme. These results provide insights into the mechanisms and specificity of histone demethylation.