Recent advances in deep learning (DL) have greatly improved the performance of positron emission tomography (PET) denoising performance. However, DL model performance can vary a lot across subjects, due to the large variability of the count levels and spatial distributions. A generalizable DL model that mitigates the subject-wise variations is highly expected toward a reliable and trustworthy system for clinical application. In this work, we propose a contrastive adversarial learning framework for subject-wise domain generalization (DG). Specifically, we configure a contrastive discriminator in addition to the UNet-based denoising module to check the subject-related information in the bottleneck feature, while the denoising module is adversarially trained to enforce the extraction of subject-invariant features. The sampled low-count realizations from the list-mode data are used as anchor-positive pairs to be close to each other, while the other subjects are used as negative samples to be distributed far away. We evaluated on 97 18F-MK6240 tau PET studies, each having 20 noise realizations with 25% fractions of events. Training, validation, and testing were implemented using 1400, 120, and 420 pairs of 3D image volumes in a subject-independent manner. The proposed contrastive adversarial DG demonstrated superior denoising performance than conventional UNet without subject-wise DG and cross-entropy-based adversarial DG.