Phytochromes are the main plant photoreceptors regulating multiple developmental processes. However, the regulatory network of phytochrome-mediated plant reproduction has remained largely unexplored. There are three phytochromes in rice, phyA, phyB and phyC. No changes in fertility are observed in the single mutants, whereas the seed-setting rate of the phyA phyB double mutant is significantly reduced. Histological and cytological analyses showed that the reduced fertility of the phyA phyB mutant was due to defects in both anther and pollen development. The four anther lobes in the phyA phyB mutant were developed at different stages with fewer pollen grains, most of which were aborted. At the mature stage, more than one lobe in the double mutant was just consisted of several cell layers. To identify genes involved in phytochrome-mediated anther development, anther transcriptomes of phyA, phyB and phyA phyB mutants were compared to that of wild-type rice respectively. Analysis of 2,241 double-mutant-specific differentially expressed transcripts revealed that the metabolic profiles, especially carbohydrate metabolism, were altered greatly, and heat-shock responses were activated in the double mutant. This study firstly provides valuable insight into the complex regulatory networks underlying phytochrome-mediated anther and pollen development in plants, and offers novel clues for hybrid rice breeding.