In recognizing the host cellular receptor and mediating fusion of virus and cell membranes, the spike (S) glycoprotein of coronaviruses is the most critical viral protein for cross-species transmission and infection. Here we determined the cryo-EM structures of the spikes from bat (RaTG13) and pangolin (PCoV_GX) coronaviruses, which are closely related to SARS-CoV-2. All three receptor-binding domains (RBDs) of these two spike trimers are in the "down" conformation, indicating they are more prone to adopt the receptor-binding inactive state. However, we found that the PCoV_GX, but not the RaTG13, spike is comparable to the SARS-CoV-2 spike in binding the human ACE2 receptor and supporting pseudovirus cell entry. We further identified critical residues in the RBD underlying different activities of the RaTG13 and PCoV_GX/SARS-CoV-2 spikes. These results collectively indicate that tight RBD-ACE2 binding and efficient RBD conformational sampling are required for the evolution of SARS-CoV-2 to gain highly efficient infection.