Spontaneous episodic activity is a fundamental mode of operation of developing networks. Surprisingly, the duration of an episode of activity correlates with the length of the silent interval that precedes it, but not with the interval that follows. Here we use a modeling approach to explain this characteristic, but thus far unexplained, feature of developing networks. Because the correlation pattern is observed in networks with different structures and components, a satisfactory model needs to generate the right pattern of activity regardless of the details of network architecture or individual cell properties. We thus developed simple models incorporating excitatory coupling between heterogeneous neurons and activity-dependent synaptic depression. These models robustly generated episodic activity with the correct correlation pattern. The correlation pattern resulted from episodes being triggered at random levels of recovery from depression while they terminated around the same level of depression. To explain this fundamental difference between episode onset and termination, we used a mean field model, where only average activity and average level of recovery from synaptic depression are considered. In this model, episode onset is highly sensitive to inputs. Thus noise resulting from random coincidences in the spike times of individual neurons led to the high variability at episode onset and to the observed correlation pattern. This work further shows that networks with widely different architectures, different cell types, and different functions all operate according to the same general mechanism early in their development.