Nano-scale surface roughness in transparent ITO films was artificially formed by sputtering a mixed Ag and ITO layer and wet etching of segregated Ag nanoparticles from the surface of the ITO film. Effective removal of self-segregated Ag particles from the grain boundaries and surface of the crystalline ITO film led to a change in only the nano-scale surface morphology of ITO film without changes in the sheet resistance and optical transmittance. A nano-scale rough surface of the ITO film led to an increase in contact area between the hole transport layer and the ITO anode, and eventually increased the hole extraction efficiency in the organic solar cells (OSCs). The heterojunction OSCs fabricated on the ITO anode with a nano-scale surface roughness exhibited a higher power conversion efficiency of 3.320%, than that (2.938%) of OSCs made with the reference ITO/glass. The results here introduce a new method to improve the performance of OSCs by simply modifying the surface morphology of the ITO anodes.