Computer-generated holography (CGH) is an advanced technology for three-dimensional (3D) displays. While the stochastic gradient descent (SGD) method is effective for holographic optimization, its application to holographic video displays is computationally expensive, as each frame requires separate optimization. To address this, we propose a novel, to the best of our knowledge, clustering optimization strategy to accelerate the SGD process for holographic video displays. Our method exploits the inherent similarities between video frames by jointly optimizing shared features first, followed by the frame-specific optimization of unique features, thereby minimizing redundant computations. The process involves clustering video frames based on common features and using the optimized holograms of the cluster center image, along with global scale factors, as initial conditions for each frame within the cluster. Numerical simulations and optical experiments demonstrate that the proposed method achieves approximately a twofold increase in computational efficiency, significantly enhancing the feasibility of holographic video displays for broader applications.