We present a study of ultrafast carrier transfer from highly luminescent states inside the core of silicon nanocrystal (due to quasidirect transitions) to states on the nanocrystal-matrix interface. This transfer leads to a sub-picosecond luminescence decay, which is followed by a slower decay component induced by carrier relaxation to lower interface states. We investigate the luminescence dynamics for two different surface passivation types and we propose a general model describing spectral dependence of ultrafast carrier dynamics. Our results stress the crucial role of the energy distribution of the interface states on surface-related quenching of quasidirect luminescence in silicon nanocrystals. We discuss how to avoid this quenching in order to bring the attractive properties of the quasidirect recombination closer to exploitation.