Faces-the most common and complex stimuli in our daily lives-contain multidimensional information used to infer social attributes that guide consequential behaviors, such as deciding who to trust. Decades of research illustrates that perceptual information from faces is processed holistically. An open question, however, is whether goals might impact this perceptual process, influencing the encoding and representation of the complex social information embedded in faces. If an individual were able to factorize information so that each dimension is separately represented, it might enable flexibility. Having a goal, for example, might mean that only goal-relevant dimensions are leveraged to inform behavior. Whether people are able to build such factorized representations remains unknown, largely due to natural correlations between social attributes. We overcome these confounds using a new statistical face model that orthogonalizes perceived facial attractiveness and trustworthiness. Across three experiments (N = 249), we observe that only in some contexts can humans successfully factorize multidimensional social information. When there is a clear goal of assessing another's trustworthiness, people successfully decompose these social attributes. The more an individual factorizes, the more they entrust money to others in a subsequent trust game. However, when the goal is to assess attractiveness, irrelevant information about trustworthiness is so potent that it biases how attractive someone is perceived-a trustworthiness "halo effect." In contrast, in goal-agnostic environments, we do not find any evidence of factorization; instead, people encode multidimensional social information in an entwined and holistic fashion that distorts their perceptions of social attributes. (PsycInfo Database Record (c) 2025 APA, all rights reserved).