The use of bitumen as a construction material dates back to antiquity. The materials in use then were mostly naturally occurring in contrast to modern bitumens which have become highly technical artificial materials. This article reviews the current understanding of bitumen structure and the consequences in terms of properties, with a strong emphasis on the rheological properties. The links between chemistry, structure and mechanical properties are highlighted in the framework of an updated colloidal picture of bitumen. It shows that a simple solvation parameter allows quantifying the effect of the asphaltenes on the rheological properties of bitumen. This appears as a promising approach in order to understand more complex phenomena such as bitumen ageing or the diffusion of rejuvenating oils into an older bitumen. From this structural modelling, the effect of several modifiers, such as polymers, acids or mineral fillers, is explained using fundamental results from the mechanics of colloidal suspensions and multiphase materials through the Palierne model. Thus, relevant parameters describing polymer-bitumen or mineral fillers-bitumen interactions can be extracted, as detailed from literature data. In the case of mineral filler, volume fraction is the key parameter but particle size comes also into play when fine fillers are considered. In the case of polymer-modified bitumens, the swelling extent of the polymer controls all other parameters of importance: volume fraction of dispersed phase and mechanical properties of both dispersed and continuous phases. In addition, interesting rheological features due to droplet shape relaxations are described in polymer-modified bitumens. Although a general picture of bitumen structure is shown to emerge, the many fundamental points that remain to be addressed are discussed throughout the paper.