We analyze the time delay between emission of photoelectrons from the outer valence ns and np subshells in noble gas atoms following absorption of an attosecond extreme ultraviolet pulse. Various processes such as elastic scattering of the photoelectron on the parent ion and many-electron correlation affect the apparent "time zero" when the photoelectron leaves the atom. This qualitatively explains the time delay between photoemission from the 2s and 2p subshells of Ne as determined experimentally by attosecond streaking [Science 328, 1658 (2010)]. However, with our extensive numerical modeling, we were only able to account for less than half of the measured time delay of 21 ± 5 as. We argue that the extreme ultraviolet pulse alone cannot produce such a large time delay and it is the streaking IR field that is most likely responsible for this effect.