Log-euclidean polyaffine transforms have recently been introduced to characterize the local affine behavior of the deformation in principal anatomical structures. The elegant mathematical framework makes them a powerful tool for image registration. However, their application is limited to large structures since they require the pre-definition of affine regions. This paper extends the polyaffine registration to adaptively fit a log-euclidean polyaffine transform that captures deformations at smaller scales. The approach is based on the sparse selection of matching points in the images and the formulation of the problem as an expectation maximization iterative closest point problem. The efficiency of the algorithm is shown through experiments on inter-subject registration of brain MRI between a healthy subject and patients with multiple sclerosis.