Four-dimensional respiratory correlated computed tomography (4D RCCT) has been widely used for studying organ motion. Most current algorithms use binning techniques which introduce artifacts that can seriously hamper quantitative motion analysis. In this paper, we develop an algorithm for tracking organ motion which uses raw time-stamped data and simultaneously reconstructs images and estimates deformations in anatomy. This results in a reduction of artifacts and an increase in signal-to-noise ratio (SNR). In the case of CT, the increased SNR enables a reduction in dose to the patient during scanning. This framework also facilitates the incorporation of fundamental physical properties of organ motion, such as the conservation of local tissue volume. We show in this paper that this approach is accurate and robust against noise and irregular breathing for tracking organ motion. A detailed phantom study is presented, demonstrating accuracy and robustness of the algorithm. An example of applying this algorithm to real patient image data is also presented, demonstrating the utility of the algorithm in reducing artifacts.