The aspect ratio (Γ) dependence of the heat transfer (Nusselt number Nu in dimensionless form) in turbulent (two-dimensional) Rayleigh-Bénard convection is numerically studied in the regime 0.4≤Γ≤1.25 for Rayleigh numbers 10(7)≤Ra≤Ra(9) and Prandtl numbers Pr=0.7 (gas) and 4.3 (water). Nu(Γ) shows a very rich structure with sudden jumps and sharp transitions. We connect these structures to the way the flow organizes itself in the sample and explain why the aspect ratio dependence of Nu is more pronounced for small Pr. Even for fixed Γ different turbulent states (with different resulting Nu) can exist, between which the flow can or cannot switch. In the latter case the heat transfer thus depends on the initial conditions.