The relativistic filamentation instability (RFI) of linearly polarized intense laser beams in plasmas near the critical density is investigated. It is found that the RFI is anisotropic to transverse perturbations in this case; a homogeneous laser beam evolves to a stratified structure parallel to the laser polarization direction, as demonstrated recently with three-dimensional particle-in-cell simulations by Nishihara et al. [Proc. SPIE 3886, 90 (2000)]. A weakly relativistic theory is developed for plasmas near the critical density. It shows that the anisotropy of the RFI results from a suppression of the instability in the laser polarization direction due to the electrostatic response. The anisotropic RFI is also analyzed based on an envelope equation for the laser beam. Finally, the envelope equation is solved numerically, and anisotropic filamentation and self-focusing are illustrated.