Mechanisms generating diverse cell types from multipotent progenitors are crucial for normal development. Neural crest cells (NCCs) are multipotent stem cells that give rise to numerous cell-types, including pigment cells. Medaka has four types of NCC-derived pigment cells (xanthophores, leucophores, melanophores and iridophores), making medaka pigment cell development an excellent model for studying the mechanisms controlling specification of distinct cell types from a multipotent progenitor. Medaka many leucophores-3 (ml-3) mutant embryos exhibit a unique phenotype characterized by excessive formation of leucophores and absence of xanthophores. We show that ml-3 encodes sox5, which is expressed in premigratory NCCs and differentiating xanthophores. Cell transplantation studies reveal a cell-autonomous role of sox5 in the xanthophore lineage. pax7a is expressed in NCCs and required for both xanthophore and leucophore lineages; we demonstrate that Sox5 functions downstream of Pax7a. We propose a model in which multipotent NCCs first give rise to pax7a-positive partially fate-restricted intermediate progenitors for xanthophores and leucophores; some of these progenitors then express sox5, and as a result of Sox5 action develop into xanthophores. Our results provide the first demonstration that Sox5 can function as a molecular switch driving specification of a specific cell-fate (xanthophore) from a partially-restricted, but still multipotent, progenitor (the shared xanthophore-leucophore progenitor).