Three-dimensional (3D) display can provide more information than two-dimensional display, and real-time 3D reconstruction of the real-world environment has broad application prospects as a key technology in the field of meta-universe and Internet of Things. 3D holographic display is considered to be an ideal 3D display scheme, thus enhancing the computational speed and reconstruction quality of 3D holograms can offer substantial support for real-time 3D reconstruction. Here, we proposed a real-time 3D holographic photography for real-world scenarios driven by both physical model and artificial intelligence. The 3D information of the real scene was acquired by a depth camera and then divided into 30 layers using the layer-based method. Convolutional neural networks (CNN) were used to build the mapping of intensity and depth maps to computer-generated holograms (CGH). The differentiability of the angular spectrum algorithm was used to realize the self-supervised training of the network, while the composite loss function was employed to optimize network parameters by calculating the loss between reconstructed and target images. The trained network can generate a CGH with a resolution of 1024×1024 in 14.5 ms. The proposed system operates at 22 frames per second and successfully reconstructs 3D video of dynamic scene. The system exhibits significant potential for application in intelligent manufacturing, remote office work, distance education and other fields.