Laser speckle rheology (LSR) is an optical technique for assessing the viscoelastic properties of materials with several industrial, biological, and medical applications. In LSR, the viscoelastic modulus, G*(ω), of a material is quantified by analyzing the temporal fluctuations of speckle patterns. However, the size of scattering particles within the material also influences the rate of speckle fluctuations, independent of sample mechanical properties, and complicates the accurate estimation of G*(ω). Here, we demonstrate that the average particle size may be retrieved from the azimuth-angle dependence of time-averaged speckle intensities, permitting the accurate quantification of the viscoelastic moduli of materials with unknown particle size distribution using LSR.