XctDiff: Reconstruction of CT Images with Consistent Anatomical Structures from a Single Radiographic Projection Image

Reconstructing CT images is equivalent to learning volume structure, which presents greater computational and pattern coverage challenges than 2D images. Inspired by latent generation models [5, 12], we adopt a similar compression design to accommodate a 3D format input. For the architecture aspect, a homogeneous spatial codebook and an extra super-resolution module [14] are incorporated to improve the reconstruction quality. For the loss function, we use 2D and 3D discriminators to optimize jointly the whole and slice.

Specifically, for given input CT $x\in\mathbb{R}^{H\times W\times D}$, it is compressed by encoder $E$ into a latent representation $z\in\mathbb{R}^{h\times w\times d\times n_{z}}$, where $h,w$, and $d$ denote the feature map size in the latent space, $n_{z}$ represents the dimension of the codebook entries. Each spatial representation is then quantized element-by-element into the closest codebook entity, which can be expressed as:

where $z_{n}$ and $n$ denote the codebook entity and the number of entities, respectively. $\mathcal{Z}$ is the vector quantization encoder.

To ensure perceptually rich variance space for generation, we adopt the heuristic strategy of vector quantization encoder [5, 21] and utilize z-score normalization to transform the variance space into a homogeneous space, which avoids the smoothing generated by the encoder against the codebook and thus improves the quality of reconstruction. The training objectives of the codebook can be summarized as follows:

where $\mathcal{N}$ represents the z-score normalization, $sg[\cdot]$ denotes the stop-gradient operation. We also introduce a super-resolution module at the end of decoder $G$ to improve the reconstruction performance. In addition, we apply a combination of slice discriminator $D_{2d}$ and volume discriminator $D_{3d}$ to optimize further jointly the autoencoder. The optimization objective of the discriminator can be formulated as:

where $x$ and $\hat{x}$ represent ground truth and prediction, respectively. $x_{k}$ denotes the $kth$ slice in the CT images. Finally, the optimization function of the autoencoder is summarized as follows:

where $\mathcal{L}_{rec}$ and $\mathcal{L}_{lpips}$ denote L1 loss and perceptual loss, $\lambda_{i}$ are weighted hyperparameters.

Reconstruction of a CT image from a single radiograph requires 3D prior information. We decouple the mapping relationship between the radiograph and the corresponding CT scan into two phases: shape mapping and anatomical structure mapping. As shown in Fig 2(a), we proposed a progressive encoder $PE$ to obtain robust 3D priors by simplifying the complex mapping relationship. Specifically, for given an input radiograph $p$, we use multilayer perceptron (MLP) to convert the multilayer stacked slices into the human 3D space. The MLP only works in one direction since frontal projection radiograph matches in geometry a coronal slice, making it easier to learn the shape of the organs from this direction, which can be expressed as follows:

where $v$ and $p$ denote the voxel of the CT image and the pixel of the radiograph, respectively. In the anatomical structure mapping stage, we globally optimize the extracted rough features by multiple successive 3D convolutional layers to obtain more accurate 3D anatomical priors. Finally, the extracted prior information is mapped into the homogeneous space by vector quantization encoder. Formally, this process can be summarized as:

The DM [6, 16, 17] uses UNet $\theta$ to predict the added noise $\varepsilon$ at each time step $t$ in the inverse process. To minimize the pattern coverage challenge, we apply convolution only in the image plane (i.e., kernel size of 3×3×1). For given anatomical priors $y$ extracted from radiograph, the denoising encoder $\varepsilon_{\theta}(x,t,y)$ controls the direction of generation based on the prior distribution via cross-attention. The optimization function of the conditional diffusion model can be written as:

where $x_{t}$ is the sampling of the diffusion process at the moment $t$, which is effected by noise schedule $\alpha_{t}$.

Similar to previous works [13, 18, 20], we used digitally reconstructed radiograph (DRR) technology to translate real CT scans into corresponding radiographs. However, as shown in Fig 2(b), a disparity exists between synthetic radiographs and real ones in terms of realism and style, which may hinder model generalization and evaluation. Thus, when inferring real radiographs, we utilized CycleGAN [22] to transform real radiographs into synthetic-style radiographs.

Experiments were conducted on four public datasets. The LIDC-IDRI dataset [2], comprising 1018 CT scan images, was employed for training and validation of our proposed framework, with voxel resolutions resampled to [2.5,2.5,2.5] and cropped to [128,128,128] cube regions. Among these, 916 cases were used for training and 102 cases were used for testing. In addition, the ChestXray2017 dataset [8], containing 5856 radiographic images, was used for style conversion between real and synthetic radiographic images. We randomly selected 1000 normal radiographs without lesions from this dataset for style migration and validation. For the BCV [9] and MSD Spleen [1] datasets, which are abdominal segmentation datasets, 25 cases were randomly selected using 5-fold cross-validation, with 5 and 7 cases were used for testing, respectively.

All experiments were conducted on 2 RTX3090 GPUs. The XctDiff framework comprises two main training phases. Initially, we trained a perceptual compression model with a batch size of 1, a learning rate of 2e-4, and 80,000 iterations. Subsequently, a progressive encoder was trained with a batch size of 16, a learning rate of 1e-4, and 50,000 iterations. In the second stage, a conditional DM in latent space was trained with a batch size of 8, a learning rate of 1e-4, and 100,000 iterations. For the self-supervised pre-training task, we utilized the code library provided by Swin UNETR [19]. Each model was cross-validated on the training dataset with 5 folds and trained for 50,000 iterations.

Reconstruction Result. The quantitative results of CT reconstruction from a single radiograph are evaluated in the metrics of Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS). As shown in Table 1, it is evident that XctDiff outperforms the second-best method, X2CTGAN, by +0.9 PSNR(dB) and +2.1 SSIM(%). Furthermore, as demonstrated through qualitative visualization in Fig 3, our approach produces high-quality CT images with precise anatomical structures, e.g., the heart, lungs, spine, and ribs can be easily identified, which is crucial for downstream tasks. Experimental results show that CT reconstructed from a single X-ray image has much lower reconstruction accuracy than traditional CT images. But it also demonstrates the potential for certain downstream tasks.

Ablation Study. Experiment results of image quality with different settings and improvements are presented in Tab. 2. We use two different settings as benchmarks. It can be observed that the utilization of the codebook is inefficient (only 1.8% and 1.5%, respectively), which suggests that CT images are always reconstructed based on a limited number of semantic features, resulting in a poor quality reconstruction. The third line demonstrates that the reconstructed quality is significantly improved by mapping the encoder output and codebook onto an isotropic Euclidean space. In the fourth line, the incorporation of a super-resolution module also improves the reconstruction quality. When these two components are combined, the reconstruction results surpass those achieved by the first baseline, exhibiting an improvement of +0.8 PSNR(dB), +3.7 SSIM(%) and -1.1 LPIPS(%), respectively. In addition, we conducted an analysis on the impact of varying codebook quantities and dimensions. The results show that the best performance was achieved with a dimension set of 8 and a codebook quantity of 8192, which we use as the default setting. As shown in the Tab. 3, we also performed ablation in the progressive semantic encoder and improved autoencoder, demonstrating their positive impact on enhancing the performance of XctDiff.

Self-supervised Pre-training. Reconstructed CT from a single radiograph can facilitate the self-supervised pre-training of medical images by circumventing challenges in data acquisition and privacy limitations. As shown in Fig. 4, we demonstrate the efficacy of the reconstructed CT from real radiographs for the self-supervised pre-training task using two models based on different paradigms (UNet [4] and Swin UNETR [19]) on both BCV dataset and MSD Spleen dataset. The pre-training performance on the Reconstructed CT dataset is slightly lower than training on the real CT dataset but still with a significant improvement of +3.1 and +1.8 Dice(%), respectively. For the MSD Spleen dataset, the UNet [4] model pretrained on reconstructed CT images obtained the highest 93.7 Dice(%), demonstrating a significant improvement of +1.7 and +0.4 compared to the baseline model and the model pretrained on real CT images, respectively. In addition, the Swin UNETR pre-trained on reconstructed CT dataset also exhibited competitive performance.