The thermal conductivity of free-standing silicon nanowires (SiNWs) with diameters from 1-3 nm has been studied by using the one-dimensional Boltzmann's transport equation. Our model explicitly accounts for the Umklapp scattering process and electron-phonon coupling effects in the calculation of the phonon scattering rates. The role of the electron-phonon coupling in the heat transport is relatively small for large silicon nanowires. It is found that the effect of the electron-phonon coupling on the thermal conduction is enhanced as the diameter of the silicon nanowires decreases. Electrons in the conduction band scatter low-energy phonons effectively where surface modes dominate, resulting in a smaller thermal conductivity. Neglecting the electron-phonon coupling leads to overestimation of the thermal transport for ultra-thin SiNWs. The detailed study of the phonon density of states from the surface atoms and central atoms shows a better understanding of the nontrivial size dependence of the heat transport in silicon nanowire.