Many cellular processes require a polarization axis which generally initially emerges as an inhomogeneous distribution of molecular markers in the cell. We present a simple analytical model of a general mechanism of cell polarization taking into account the positive feedback due to the coupled dynamics of molecular markers and cytoskeleton filaments. We find that the geometry of the organization of cytoskeleton filaments, nucleated on the membrane (e.g., cortical actin) or from a center in the cytoplasm (e.g., microtubule asters), dictates whether the system is capable of spontaneous polarization or polarizes only in response to external asymmetric signals. Our model also captures the main features of recent experiments of cell polarization in two considerably different biological systems, namely, mating budding yeast and neuron growth cones.