Diffusion in the plasma membrane of living cells is often found to display anomalous dynamics. However, the mechanism underlying this diffusion pattern remains highly controversial. Here, we study the physical mechanism underlying Kv2.1 potassium channel anomalous dynamics using single-molecule tracking. Our analysis includes both time series of individual trajectories and ensemble averages. We show that an ergodic and a nonergodic process coexist in the plasma membrane. The ergodic process resembles a fractal structure with its origin in macromolecular crowding in the cell membrane. The nonergodic process is found to be regulated by transient binding to the actin cytoskeleton and can be accurately modeled by a continuous-time random walk. When the cell is treated with drugs that inhibit actin polymerization, the diffusion pattern of Kv2.1 channels recovers ergodicity. However, the fractal structure that induces anomalous diffusion remains unaltered. These results have direct implications on the regulation of membrane receptor trafficking and signaling.