The ultrafast fluorescence quenching of 2,6-sulfanyl-core-substituted naphthalenediimides was investigated by transient spectroscopy. We find a strong dependence of the relaxation on the chemical structure of the substituent. Direct linking of an aryl rest to the sulfur atom leads to a strong red shift of the fluorescence in 1 ps and the disappearance of the emission in 5-7 ps depending on the polarity and viscosity of the solvent. This complex behavior is interpreted with the help of quantum chemical calculations. The calculations suggest that the initial relaxation corresponds to a planarization of the substituents and an associated partial electron transfer. This is followed by a twisting of the phenylsulfanyl substituents out of the molecular plane that allows a complete localization of the electron-donating orbital on the aryl group. Finally the back transfer happens in another 5-7 ps. For an additional methylene spacer group between the sulfur and the aryl, this sequence of relaxation steps is not possible and a simple exponential decay, slower by about 1 order of magnitude, is found.