We consider a class of proposed gravitational-wave detectors based on multiple atomic interferometers separated by large baselines and referenced by common laser systems. We compute the sensitivity limits of these detectors due to intrinsic phase noise of the light sources, noninertial motion of the light sources, and atomic shot noise and compare them to sensitivity limits for traditional light interferometers. We find that atom interferometers and light interferometers are limited in a nearly identical way by intrinsic phase noise and that both require similar mitigation strategies (e.g., multiple-arm instruments) to reach interesting sensitivities. The sensitivity limit from motion of the light sources is slightly different and, in principle, favors the atom interferometers in the low-frequency limit, although the limit in both cases is severe.