LIDIA: Precise Liver Tumor Diagnosis on Multi-Phase Contrast-Enhanced CT via Iterative Fusion and Asymmetric Contrastive Learning

Mask Transformer. Recently, mask transformers have been proposed for various segmentation tasks. Rather than performing per-pixel classification as in traditional semantic segmentation methods, they predict a set of binary masks and assigns a class label to each mask, enabling instance segmentation [4, 3, 22]. An example is Mask2Former [3], which uses a pixel encoder-decoder to generate multi-scale features and employs learnable embeddings as object queries. These queries interact with image features as well as themselves through a transformer decoder to segment objects and also identify their classes. Benefiting from the masked attention mechanism, this approach allows the transformer block to concentrate on specific local regions where tumors are situated, making Mask2Former particularly effective for the segmentation and diagnosis of lesions, which are often small in size.

Iterative Fusion Module (IFM). In clinical workflows, physicians often determine the type of lesions based on the differences in features of lesion regions of interest (ROIs) across various phases. Inspired by this practice, to effectively utilize the complementary information between different phases, we first propose an iterative multi-phase fusion module. Formally, the specific information extraction for each phase can be defined as a function mapping from the phase image to feature map:

where $\mathcal{F}_{p}$ represents the context-specific block for phase $p$, and $\bm{h}^{p}$ is the resultant feature map for that phase with $C$ channels. The feature extraction layer is comprised of two convolutional blocks, each employing 3D convolutions, instance normalization, and LeakyReLU activations to progressively extract information. Then, to effectively aggregate multi-phase information, we perform the fusion of information following the temporal sequence of the phases. Specifically, we fuse them in the order of non-contrast, arterial, venous, and delayed phases. The fusion process iteratively incorporates feature maps from each phase $p$ into a fused feature $\bm{h}^{fuse}$. The fusion operation is performed by concatenating the current fused $\bm{h}^{fuse}$ with the feature of the next phase $\bm{h}^{i}$ and applying the convolutional block $\mathcal{F}_{conv}$ to extract new features. This is mathematically represented by the following recursive equation:

where $\bm{h}^{fuse}_{1}=\bm{h}^{\text{NC}}$ and $\bm{h}^{fuse}_{k}$ represents the fused feature after incorporating the $k$-th subsequent phase’s feature. The final fused feature is $\bm{h}^{fuse}_{4}$ if delayed phase is present and $\bm{h}^{fuse}_{3}$ if not. This process provides a way to adaptively incorporate variable phase numbers. It is able to capture dynamic contrast changes among phases similar to ConvLSTM [17], while being more lightweight.

Asymmetric Contrastive Learning (ACL). To increase intra-class compactness and inter-class discriminability for liver tumors, we propose an asymmetric contrastive loss for the embedding of lesions, denoted as $\mathcal{L}_{acl}$. The liver contains numerous rare lesion types that are underrepresented (typically less than 10 samples in our dataset). It is impractical to assign each rare type as a separate class because the network could not effectively learn given such few samples, so we collectively assign as them as the “others” class. The ordinary supervised contrastive loss applies a uniform attraction within each class and repulsion between any two distinct classes, treating all classes equally. Due to the inherent heterogeneity of “others” class, clustering them using ordinary supervised contrastive loss would limit the model’s flexibility when dealing with unseen lesions or those with similar appearances. On the other hand, we differentiates between common and rare lesions: $\mathcal{L}_{acl}$ only performs attraction within each common class and repulsion among common classes; while for samples within the “others” class, we do not try to cluster them together, but instead only keep them distant from the common classes. The original supervised contrastive loss can be represented by the following equation:

where

Here, $z(x)$ is a non-linear projection function. $\mathcal{P}(x)$ is the set of positive samples with the same label as $x$. $\mathcal{A}(x)$ is the set of all contrastive samples with respect to $x$, and $\mathcal{I}$ is the set of training samples.

Training and Inference. Our task includes binary mask prediction and lesion type classification. For mask prediction, we employ a hybrid loss function, $\mathcal{L}_{seg}$, which combines Cross-Entropy (CE) loss with Dice loss. Similar to [16], we adopt a foreground-enhanced sampling strategy when computing $\mathcal{L}_{seg}$, which increases the ratio of foreground pixels in the loss calculation, improving the recall of small lesions.For lesion type classification, due to the class imbalance issue present among liver lesions, we employ focal loss $\mathcal{L}_{focal}$ [11] instead of the CE loss in [3] to enhance the learning focus on the underrepresented classes. Therefore, the final loss function is expressed as a combination of above loss, mathematically represented as:

Liver lesion classification faces the challenge of different tumors can present similar features while small lesions in segmentation tasks are easily missed due to their size. Therefore, when updating the model, we employ the sharpness aware minimization strategy [6], which seeks weights that demonstrate low sensitivity to input noises. This leads to more robust predictions, especially for subtle or ambiguous features, thereby improving the model’s performance on both classification and segmentation of lesions.

Effect of iterative fusion module. We used nn-UNet with three input phases (discarding the delayed phase) as the baseline method and compared the performance of various phase fusion approaches, where the AUC was calculated using the method described in [24]. The advantages of IFM are particularly evident across various lesions, including HCC, ICC, cysts, and “others”. All methods except baseline nnU-Net can accept variable number of input phases and thus take advantage of the complete phase information (including delayed phase), yet IFM achieves better performance.

Ablation study. As shown in Table 3, focal loss is beneficial for improving the underrepresented calc class samples. LiverMax effectively confines the foreground within the liver region, significantly reducing false positives and thereby increasing accuracy. Moreover, IFM efficiently exploits multi-phase information, substantially enhancing performance in heterogeneous categories such as HCC and others. Lastly, with the regularization and optimization methods, ACL and SAM, LIDIA achieves the highest average AUC of $93.6\%$. The findings from the ablation study clearly demonstrate the efficacy of our individual modules.