The effect of N doping on the crystal structure, electronic, and optical properties of α-Bi2O3 and β-Bi2O3 has been studied in detail based with first principle calculations. The crystallographic features of Bi2O3 polymorphs are not substantially changed through N doping, whereas charge transfer from Bi to N results in large variations of charge density distribution. N-doped β-Bi2O3 exhibits improved thermal stability due to stronger Bi-N covalent bonds and lower defect formation energy, and the convenient preparative access agrees well with experimental observations. Calculated band structures and optical properties indicate that N doping does not induce major band gap narrowing, but leads to the presence of isolated bands above the VBM induced by N 2p for both α-Bi2O3 and β-Bi2O3 which induce large red-shifts of their visible light absoprtion properties. These isolated bands act as acceptor levels and facilitate electron transition under visible light illumination through introduction of steps between VB and CB, thereby rendering the materials quite promising for photocatalytic applications.