The planar scattering amplitudes of N=4 super-Yang-Mills theory display symmetries and structures which underlie their relatively simple analytic properties such as having only logarithmic singularities and no poles at infinity. Recent work shows in various nontrivial examples that the simple analytic properties of the planar sector survive into the nonplanar sector, but this has yet to be understood from underlying symmetries. Here, we explicitly show that for an infinite class of nonplanar integrals that covers all subleading-color contributions to the two-loop four- and five-point amplitudes of N=4 super-Yang-Mills theory, symmetries analogous to dual conformal invariance exist. A natural conjecture is that this continues to all amplitudes of the theory at any loop order.