This paper presents an outlier immune 3D active shape models framework for robust volumetric segmentation of the carotid artery required for accurate plaque burden assessment. In the proposed technique, outlier handling is based on a shape metric that is invariant to scaling, rotation and translation by using the ratio of inter-landmark distances as a local shape dissimilarity measure. Tolerance intervals for each descriptor are calculated from the training samples and used to infer the validity of landmarks. The identified outliers are corrected prior to the model fitting using the ratios distributions and appearance information. To improve the feature point search, the method exploits the geometrical knowledge from the outlier analysis at the previous iteration to weight the gray level appearance based fitness measure. A combined intensity-phase feature point search is also introduced which significantly limits the presence of outliers and improves the overall search accuracy. Both numerical and in vivo assessments of the method involving volumetric segmentation of the carotid artery have shown that the outlier handling technique is capable of handling a significant presence of outliers independently of the amplitudes.