In recent experiments the transverse normalized rms emittance of laser-accelerated MeV ion beams was found to be < 0.002 mm mrad, which is at least 100 times smaller than the emittance of thermal ion sources used in accelerators [T. E. Cowan, Phys. Rev. Lett. 92, 204801 (2004)]. We investigate the origin for the low emittance of laser-accelerated proton beams by studying several candidates for emittance-growth mechanisms. As our main tools, we use analytical models and one- and two-dimensional particle-in-cell simulations that have been modified to include binary collisions between particles. We find that the dominant source of emittance is filamentation of the laser-generated hot electron jets that drive the ion acceleration. Cold electron-ion collisions that occur before ions are accelerated contribute less than ten percent of the final emittance. Our results are in qualitative agreement with the experiment, for which we present a refined analysis relating emittance to temperature, a better representative of the fundamental beam physics.