We study the Rayleigh-instability-type transformation of polystyrene (PS) thin films coated in the nanopores of anodic aluminum oxide (AAO) templates. The PS thin films are fabricated using a solution-wetting method, in which the wall thicknesses are controlled by the concentrations of the polymer solutions and the diameters of the nanopores. By thermal annealing, the surfaces of the thin films undulate and the morphologies transform from nanotubes to Rayleigh-instability-induced nanostructures (short nanorods) and long nanorods. To understand the mechanism of the morphology transformation further, we construct the morphology diagrams by annealing the PS thin films at different temperatures and times. We observe that the morphology diagrams of the PS thin films prepared by different concentrations are similar, indicating that the transformation kinetics are not affected by the film thicknesses. The values of the undulation wavelengths, however, are controlled by the film thicknesses and the diameters of the nanopores.