Interlayer vibrations with discrete quantized modes in two-dimensional (2D) materials can be excited by ultrafast light due to the inherent low dimensionality and van der Waals force as a restoring force. Controlling such interlayer vibrations in layered materials, which are closely related to fundamental nanomechanical interactions and thermal transport, in spatial- and time-domain provides an in-depth understanding of condensed matters and potential applications for advanced phononic and photonics devices. The manipulation of interlayer vibrational modes has been implemented in a spatial domain through material design to develop novel optoelectronic and phononic devices with various 2D materials, but such control in a time domain is still lacking. We present an all-optical method for controlling the interlayer vibrations in a highly precise manner with Bi2Se3 as a promising optoelectronic and thermoelasticity material in layered structures using a coherently controlled pump and probe scheme. The observed thickness-dependent fast interlayer breathing modes and substrate-induced slow interfacial modes can be exactly explained by a modified linear chain model including coupling effect with substrate. In addition, the results of coherent control experiments also agree with the simulation results based on the interference of interlayer vibrations. This investigation is universally applicable for diverse 2D materials and provides insight into the interlayer vibration-related dynamics and novel device implementation based on an ultrafast timescale interlayer-spacing modulation scheme.