The ultrastructure of bone can be considered as a conjunction between the biology and the biomechanics of the tissue. It is the result of cellular and molecular activities of bone formation, and its organization dominates the mechanical behavior of bone. Following this perspective, the objective of this review is to provide a current understanding of bone ultrastructure and its relationships with the toughness of the tissue. Therefore, we first provide a discussion on the organization of bone constituents, namely collagen, mineral, and water. Then, we present evidence on how the toughness of bone relates to its ultrastructure through the formation of micro damage. In addition, attention is given to how damage accumulation serves as a toughening mechanism. Finally, we describe how changes in the ultrastructure-caused by osteogenesis imperfecta, gamma irradiation, fluoride treatment, and aging affect the toughness and competence of bone.