The surface potential and the efficiency of interfacial charge transfer are extremely important for designing future semiconductor devices based on the emerging two-dimensional (2D) phosphorene. Here, we directly measured the strong layer-dependent surface potential of mono- and few-layered phosphorene on gold, which is consistent with the reported theoretical prediction. At the same time, we used an optical way photoluminescence (PL) spectroscopy to probe charge transfer in the phosphorene-gold hybrid system. We firstly observed highly anisotropic and layer-dependent PL quenching in the phosphorene-gold hybrid system, which is attributed to the highly anisotropic/layer-dependent interfacial charge transfer.