A fiber-optic toluene gas sensor based on reduced graphene oxide (rGO) is demonstrated and its sensing property is investigated experimentally and theoretically. The rGO film is deposited on a side polished fiber (SPF), allowing the strong interaction between rGO film and propagating field and making the SPF sensitive to toluene gas. It is found that the sensor has good linearity and reversibility and can work at room temperature with the response and the recovery time of 256 s and the detection limit of 79 ppm. Moreover, a theoretical model for the sensor is established to analyze the sensing mechanism. Theoretical analysis indicates this type of sensor could work in a wide range of toluene gas concentration and shows that a significant rise in its sensitivity can be expected by adjusting the doping level or chemical potential of graphene.