Combining techniques of cavity quantum electrodynamics, quantum measurement, and quantum feedback, we have realized the heralded transfer of a polarization qubit from a photon onto a single atom with 39% efficiency and 86% fidelity. The reverse process, namely, qubit transfer from the atom onto a given photon, is demonstrated with 88% fidelity and an estimated efficiency of up to 69%. In contrast to previous work based on two-photon interference, our scheme is robust against photon arrival-time jitter and achieves much higher efficiencies. Thus, it constitutes a key step toward the implementation of a long-distance quantum network.