We study the effect of a small density n_{v} of quenched nonmagnetic impurities, i.e., vacancy disorder, in gapped short-range resonating valence bond (RVB) spin liquid states and valence bond solid (VBS) states of quantum magnets. We argue that a large class of short-range RVB liquids are stable to vacancy disorder at small n_{v} on the kagome lattice, while the corresponding states on triangular, square, and honeycomb lattices are unstable to vacancy disorder at any nonzero n_{v} due to the presence of emergent vacancy-induced local moments. In contrast, VBS states are argued to be generically unstable (independent of lattice geometry) to vacancy disorder at any nonzero n_{v} due to such a local-moment instability. Our arguments rely in part on an analysis of the statistical mechanics of maximally packed dimer covers of the diluted lattice, and are fully supported by our computational results on O(N) symmetric designer Hamiltonians. These arguments also imply that short-range RVB spin liquid states are generically unstable to bond dilution on all these lattices.