The magnetic properties of gold nanotubes encapsulated transition metal (TM, TM=Co and Mn) and monoatomic chains (TM@Au) are studied using first-principles density functional calculations. The TM chains are significantly stabilized by the gold nanotube coating. TM-TM distance-dependent ferromagnetic-antiferromagnetic phase transition in TM@Au is observed and can be understood by Ruderman-Kittel-Kasuya-Yosida (RKKY) model. The magnetocrystalline anisotropy energies of the TM@Au tubes are dramatically enhanced by one order of magnitude compared to those of free TM chains. Furthermore, the stronger interaction between Mn chain and gold nanotube even switches the easy magnetization axis along the tube.