We investigate the dynamics of spin glasses from the "rheological" point of view, in which aging is suppressed by the action of small, nonconservative forces. The different features can be expressed in terms of the scaling of relaxation times with the magnitude of the driving force, which plays the role of the critical parameter. Stated in these terms, ultrametricity loses much of its mystery and can be checked rather easily. This approach also seems a natural starting point to investigate what would be the real-space structures underlying the hierarchy of time scales. We study in detail the appearance of this many-scale behavior in a mean-field model, in which dynamic ultrametricity is clearly present. A similar analysis is performed on numerical results obtained for a three-dimensional spin glass: In that case, our results are compatible with either that dynamic ultrametricity is absent or that it develops so slowly that even in experimental time-windows it is still hardly observable.