We introduce a setup of optical tweezers, capable of carrying out temperature-dependent rheological measurements of soft materials. In our setup, the particle displacement is detected by imaging a bright spot due to fluorescence emitted from a dye-labeled particle against a dark background onto a quadrant photodiode. This setup has a relatively wide space around the sample that allows us to further accessorize the optical tweezers by a temperature control unit. The applicability of the setup was examined on the basis of the rheological measurements using a typical viscoelastic system, namely a worm-like micelle solution. The temperature and frequency dependences of the local viscoelastic functions of the worm-like micelle solution obtained by this setup were in good accordance with those obtained by a conventional oscillatory rheometer, confirming the capability of the optical tweezers as a tool for the local rheological measurements of soft materials. Since the optical tweezers measurements only require a tiny amount of sample (~40 μL), the rheological measurements using our setup should be useful for soft materials of which the available amount is limited.