This paper presents a general framework for analyzing structural and radiometric asymmetry in brain images. In a healthy brain, the left and right hemispheres are largely symmetric across the mid-sagittal plane. Brain tumors may belong to one or both of the following categories: mass-effect, in which the diseased tissue displaces healthy tissue; and infiltrating, in which healthy tissue has become diseased. Mass-effect brain tumors cause structural asymmetry by displacing healthy tissue, and may cause radiometric asymmetry in adjacent normal structures due to edema. Infiltrating tumors have a different radiometric response from healthy tissue. Thus, structural and radiometric asymmetries across the mid-sagittal plane in brain images provide important cues that tumors may be present. We have developed a framework that registers images with their reflections across the mid-sagittal plane. The registration process accounts for tissue displacement through large deformation image warping. Radiometric differences are taken into account through an additive intensity field. We present an efficient multi-scale algorithm for the joint estimation of structural and radiometric asymmetry. Results for nine MR images of patients with tumors and four normal control subjects are presented.