Joint Beamforming Design and User Clustering Algorithm in NOMA-Assisted ISAC Systems

To enhance the performance of non-orthogonal multiple access (NOMA)-assisted integrated sensing and communication (ISAC) systems in multi-user distributed scenarios, an improved Gaussian Mixture Model (GMM)-based user clustering algorithm is proposed. This algorithm is tailored for ISAC systems, significantly improving bandwidth reuse gains and reducing serial interference. First, using the Sum of Squared Errors (SSE), the algorithm reduces sensitivity to the initial cluster center locations, improving clustering accuracy. Then, direction weight factors are introduced based on the base station position and a penalty function involving users' Euclidean distances and sensing power. Modifications to the EM algorithm in calculating posterior probabilities and updating the covariance matrix help align user clusters with the characteristics of NOMAISAC systems. This improves users' interference resistance, lowers decoding difficulty, and optimizes the system's sensing capabilities. Finally, a fractional programming (FP) approach addresses the non-convex joint beamforming design problem, enhancing power and channel gains and achieving co-optimizing sensing and communication signals. The simulation results show that, under the improved GMM user clustering algorithm and FP optimization, the NOMA-ISAC system improves user spectral efficiency by 4.3% and base station beam intensity by 5.4% compared to traditional ISAC systems.