Cell contacts provide spatial cues that polarize early embryos and epithelial cells. The homophilic adhesion protein E-cadherin is required for contact-induced polarity in many cells. However, it is debated whether E-cadherin functions instructively as a spatial cue, or permissively by ensuring adequate adhesion so that cells can sense other contact signals. In Caenorhabditis elegans, contacts polarize early embryonic cells by recruiting the RhoGAP PAC-1 to the adjacent cortex, inducing PAR protein asymmetry. Here we show that the E-cadherin HMR-1, which is dispensable for adhesion, functions together with the α-catenin HMP-1, the p120 catenin JAC-1, and the previously uncharacterized linker PICC-1 (human CCDC85A-C) to bind PAC-1 and recruit it to contacts. Mislocalizing the HMR-1 intracellular domain to contact-free surfaces draws PAC-1 to these sites and depolarizes cells, demonstrating an instructive role for HMR-1 in polarization. Our findings identify an E-cadherin-mediated pathway that translates cell contacts into cortical polarity by directly recruiting a symmetry-breaking factor to the adjacent cortex.