Linux Kernel Programming: A comprehensive guide to kernel internals, writing kernel modules, and kernel synchronization

Linux Kernel Programming

A comprehensive guide to kernel internals, writing kernel modules, and kernel synchronization

Kaiwan N Billimoria

BIRMINGHAM - MUMBAI

Linux Kernel Programming

Copyright © 2021 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Wilson D'souza

Publishing Product Manager: Vijin Boricha

Content Development Editor: Romy Dias

Senior Editor: Rahul D'souza

Technical Editor: Nithik Cheruvakodan

Copy Editor: Safis Editing

Project Coordinator: Neil Dmello

Proofreader: Safis Editing

Indexer: Manju Arasan

Production Designer: Joshua Misquitta

First published: March 2021

Production reference: 2170321

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-78995-343-5

www.packt.com

First, to my dear parents, Diana and Nadir Nads, for showing me how to live a happy and productive life. To my dear wife, Dilshad (an accomplished financial advisor herself), and our amazing kids, Sheroy and Danesh – thanks for all your love and patience.

– Kaiwan N Billimoria

Contributors

About the author

Kaiwan N Billimoria taught himself BASIC programming on his dad's IBM PC back in 1983. He was programming in C and Assembly on DOS until he discovered the joys of Unix, and by around 1997, Linux!

Kaiwan has worked on many aspects of the Linux system programming stack, including Bash scripting, system programming in C, kernel internals, device drivers, and embedded Linux work. He has actively worked on several commercial/FOSS projects. His contributions include drivers to the mainline Linux OS and many smaller projects hosted on GitHub. His Linux passion feeds well into his passion for teaching these topics to engineers, which he has done for well over two decades now. He's also the author of Hands-On System Programming with Linux. It doesn't hurt that he is a recreational ultrarunner too.

Writing this book took a long while; I'd like to thank the team from Packt for their patience and skill! Carlton Borges, Romy Dias, Vijin Boricha, Rohit Rajkumar, Vivek Anantharaman, Nithin Varghese, and all the others. It was indeed a pleasure working with you.

I owe a debt of gratitude to the very able technical reviewers – Donald Donnie Tevault and Anil Kumar. They caught a lot of my mistakes and omissions and greatly helped make this book better.

About the reviewers

Donald A. Tevault, but you can call him Donnie, got involved with Linux way back in 2006 and has been working with it ever since. He holds the Linux Professional Institute Level 3 Security certification, and the GIAC Incident Handler certification. Donnie is a professional Linux trainer, and thanks to the magic of the internet, teaches Linux classes literally the world over from the comfort of his living room. He's also a Linux security researcher for an IoT security company.

Anil Kumar is a Linux BSP and firmware developer at Intel. He has over 12 years of software development experience across many verticals, including IoT, mobile chipsets, laptops/Chromebooks, media encoders, and transcoders. He has a master's degree in electronics design from the Indian Institute of Science and a bachelor's degree in electronics and communication from BMS College of Engineering, India. He is an electronics enthusiast and blogger and loves tinkering to create fun DIY projects.

Table of Contents

Title Page

Copyright and Credits

Linux Kernel Programming

Dedication

Contributors

About the author

About the reviewers

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Get in touch

Bewertungen

Section 1: The Basics

Kernel Workspace Setup

Technical requirements

Running Linux as a guest VM

Installing a 64-bit Linux guest

Turn on your x86 system's virtualization extension support 

Allocate sufficient space to the disk

Install the Oracle VirtualBox Guest Additions

Experimenting with the Raspberry Pi

Setting up the software – distribution and packages

Installing software packages

Installing the Oracle VirtualBox guest additions

Installing required software packages

Installing a cross toolchain and QEMU

Installing a cross compiler

Important installation notes

Additional useful projects

Using the Linux man pages

The tldr variant

Locating and using the Linux kernel documentation

Generating the kernel documentation from source

Static analysis tools for the Linux kernel

Linux Trace Toolkit next generation

The procmap utility

Simple Embedded ARM Linux System FOSS project

Modern tracing and performance analysis with [e]BPF

The LDV - Linux Driver Verification - project

Summary

Questions

Further reading

Building the 5.x Linux Kernel from Source - Part 1

Technical requirements 

Preliminaries for the kernel build

Kernel release nomenclature

Kernel development workflow – the basics

Types of kernel source trees

Steps to build the kernel from source

Step 1 – obtaining a Linux kernel source tree

Downloading a specific kernel tree

Cloning a Git tree

Step 2 – extracting the kernel source tree

A brief tour of the kernel source tree

Step 3 – configuring the Linux kernel

Understanding the kbuild build system

Arriving at a default configuration

Obtaining a good starting point for kernel configuration

Kernel config for typical embedded Linux systems

Kernel config using distribution config as a starting point

Tuned kernel config via the localmodconfig approach

Getting started with the localmodconfig approach

Tuning our kernel configuration via the make menuconfig UI

Sample usage of the make menuconfig UI

More on kbuild

Looking up the differences in configuration

Customizing the kernel menu – adding our own menu item

The Kconfig* files

Creating a new menu item in the Kconfig file

A few details on the Kconfig language

Summary

Questions

Further reading

Building the 5.x Linux Kernel from Source - Part 2

Technical requirements

Step 4 – building the kernel image and modules

Step 5 – installing the kernel modules

Locating the kernel modules within the kernel source

Getting the kernel modules installed

Step 6 – generating the initramfs image and bootloader setup

Generating the initramfs image on Fedora 30 and above

Generating the initramfs image – under the hood

Understanding the initramfs framework

Why the initramfs framework?

Understanding the basics of the boot process on the x86

More on the initramfs framework

Step 7 – customizing the GRUB bootloader

Customizing GRUB – the basics

Selecting the default kernel to boot into

Booting our VM via the GNU GRUB bootloader

Experimenting with the GRUB prompt

Verifying our new kernel's configuration

Kernel build for the Raspberry Pi

Step 1 – cloning the kernel source tree

Step 2 – installing a cross-toolchain

First method – package install via apt

Second method – installation via the source repo

Step 3 – configuring and building the kernel

Miscellaneous tips on the kernel build

Minimum version requirements

Building a kernel for another site

Watching the kernel build run

A shortcut shell syntax to the build procedure 

Dealing with compiler switch issues

Dealing with missing OpenSSL development headers

Summary

Questions

Further reading

Writing Your First Kernel Module - LKMs Part 1

Technical requirements

Understanding kernel architecture – part 1

User space and kernel space

Library and system call APIs

Kernel space components

Exploring LKMs

The LKM framework

Kernel modules within the kernel source tree

Writing our very first kernel module

Introducing our Hello, world LKM C code

Breaking it down

Kernel headers

Module macros

Entry and exit points

Return values

The 0/-E return convention

The ERR_PTR and PTR_ERR macros

The __init and __exit keywords

Common operations on kernel modules

Building the kernel module

Running the kernel module

A quick first look at the kernel printk()

Listing the live kernel modules

Unloading the module from kernel memory

Our lkm convenience script

Understanding kernel logging and printk

Using the kernel memory ring buffer

Kernel logging and systemd's journalctl

Using printk log levels

The pr_ convenience macros

Wiring to the console

Writing output to the Raspberry Pi console

Enabling the pr_debug() kernel messages

Rate limiting the printk instances

Generating kernel messages from the user space

Standardizing printk output via the pr_fmt macro

Portability and the printk format specifiers

Understanding the basics of a kernel module Makefile

Summary 

Questions

Further reading

Writing Your First Kernel Module - LKMs Part 2

Technical requirements

A better Makefile template for your kernel modules

Configuring a debug kernel

Cross-compiling a kernel module

Setting up the system for cross-compilation

Attempt 1 – setting the special environment variables

Attempt 2 – pointing the Makefile to the correct kernel source tree for the target

Attempt 3 – cross-compiling our kernel module

Attempt 4  – cross-compiling our kernel module

Gathering minimal system information

Being a bit more security-aware

Licensing kernel modules

Emulating library-like features for kernel modules

Performing library emulation via multiple source files

Understanding function and variable scope in a kernel module

Understanding module stacking

Trying out module stacking

Passing parameters to a kernel module

Declaring and using module parameters

Getting/setting module parameters after insertion

Module parameter data types and validation

Validating kernel module parameters

Overriding the module parameter's name

Hardware-related kernel parameters

Floating point not allowed in the kernel

Auto-loading modules on system boot

Module auto-loading – additional details

Kernel modules and security – an overview

Proc filesystem tunables affecting the system log

The cryptographic signing of kernel modules

Disabling kernel modules altogether

Coding style guidelines for kernel developers

Contributing to the mainline kernel

Getting started with contributing to the kernel

Summary

Questions

Further reading

Section 2: Understanding and Working with the Kernel

Kernel Internals Essentials - Processes and Threads

Technical requirements

Understanding process and interrupt contexts

Understanding the basics of the process VAS

Organizing processes, threads, and their stacks – user and kernel space

User space organization

Kernel space organization

Summarizing the current situation

Viewing the user and kernel stacks

Traditional approach to viewing the stacks

Viewing the kernel space stack of a given thread or process

Viewing the user space stack of a given thread or process

[e]BPF – the modern approach to viewing both stacks

The 10,000-foot view of the process VAS

Understanding and accessing the kernel task structure

Looking into the task structure

Accessing the task structure with current

Determining the context

Working with the task structure via current

Built-in kernel helper methods and optimizations

Trying out the kernel module to print process context info

Seeing that the Linux OS is monolithic

Coding for security with printk

Iterating over the kernel's task lists

Iterating over the task list I – displaying all processes

Iterating over the task list II – displaying all threads

Differentiating between the process and thread – the TGID and the PID

Iterating over the task list III – the code

Summary

Questions

Further reading

Memory Management Internals - Essentials

Technical requirements

Understanding the VM split

Looking under the hood – the Hello, world C program

Going beyond the printf() API

VM split on 64-bit Linux systems

Virtual addressing and address translation

The process VAS – the full view

Examining the process VAS

Examining the user VAS in detail

Directly viewing the process memory map using procfs

Interpreting the /proc/PID/maps output

The vsyscall page

Frontends to view the process memory map

The procmap process VAS visualization utility

Understanding VMA basics

Examining the kernel segment

High memory on 32-bit systems

Writing a kernel module to show information about the kernel segment

Viewing the kernel segment on a Raspberry Pi via dmesg

Macros and variables describing the kernel segment layout

Trying it out – viewing kernel segment details

The kernel VAS via procmap

Trying it out – the user segment

The null trap page

Viewing kernel documentation on the memory layout

Randomizing the memory layout – KASLR

User-mode ASLR

KASLR

Querying/setting KASLR status with a script

Physical memory

Physical RAM organization

Nodes

Zones

Direct-mapped RAM and address translation

Summary

Questions

Further reading

Kernel Memory Allocation for Module Authors - Part 1

Technical requirements

Introducing kernel memory allocators

Understanding and using the kernel page allocator (or BSA)

The fundamental workings of the page allocator

Freelist organization

The workings of the page allocator

Working through a few scenarios

The simplest case

A more complex case

The downfall case

Page allocator internals – a few more details

Learning how to use the page allocator APIs

Dealing with the GFP flags

Freeing pages with the page allocator

Writing a kernel module to demo using the page allocator APIs

Deploying our lowlevel_mem_lkm kernel module

The page allocator and internal fragmentation

The exact page allocator APIs

The GFP flags – digging deeper

Never sleep in interrupt or atomic contexts

Understanding and using the kernel slab allocator

The object caching idea

Learning how to use the slab allocator APIs

Allocating slab memory

Freeing slab memory

Data structures – a few design tips

The actual slab caches in use for kmalloc

Writing a kernel module to use the basic slab APIs

Size limitations of the kmalloc API

Testing the limits – memory allocation with a single call

Checking via the /proc/buddyinfo pseudo-file

Slab allocator – a few additional details

Using the kernel's resource-managed memory allocation APIs

Additional slab helper APIs

Control groups and memory

Caveats when using the slab allocator

Background details and conclusions

Testing slab allocation with ksize() – case 1

Testing slab allocation with ksize() – case 2

Interpreting the output from case 2

Graphing it

Slab layer implementations within the kernel

Summary

Questions

Further reading

Kernel Memory Allocation for Module Authors - Part 2

Technical requirements

Creating a custom slab cache

Creating and using a custom slab cache within a kernel module

Creating a custom slab cache

Using the new slab cache's memory

Destroying the custom cache

Custom slab – a demo kernel module

Understanding slab shrinkers

The slab allocator – pros and cons – a summation

Debugging at the slab layer

Debugging through slab poisoning

Trying it out – triggering a UAF bug

SLUB debug options at boot and runtime

Understanding and using the kernel vmalloc() API

Learning to use the vmalloc family of APIs

A brief note on memory allocations and demand paging

Friends of vmalloc()

Specifying the memory protections

Testing it – a quick Proof of Concept

Why make memory read-only?

The kmalloc() and vmalloc() APIs – a quick comparison

Memory allocation in the kernel – which APIs to use when

Visualizing the kernel memory allocation API set

Selecting an appropriate API for kernel memory allocation

A word on DMA and CMA

Stayin' alive – the OOM killer

Reclaiming memory – a kernel housekeeping task and OOM

Deliberately invoking the OOM killer

Invoking the OOM killer via Magic SysRq

Invoking the OOM killer with a crazy allocator program

Understanding the rationale behind the OOM killer

Case 1 – vm.overcommit set to 2, overcommit turned off

Case 2 – vm.overcommit set to 0, overcommit on, the default

Demand paging and OOM

Understanding the OOM score

Summary

Questions

Further reading

The CPU Scheduler - Part 1

Technical requirements

Learning about the CPU scheduling internals – part 1 – essential background

What is the KSE on Linux?

The POSIX scheduling policies

Visualizing the flow

Using perf to visualize the flow

Visualizing the flow via alternate (CLI) approaches

Learning about the CPU scheduling internals – part 2

Understanding modular scheduling classes

Asking the scheduling class

A word on CFS and the vruntime value

Threads – which scheduling policy and priority

Learning about the CPU scheduling internals – part 3

Who runs the scheduler code?

When does the scheduler run?

The timer interrupt part

The process context part

Preemptible kernel

CPU scheduler entry points

The context switch

Summary

Questions

Further reading

The CPU Scheduler - Part 2

Technical requirements

Visualizing the flow with LTTng and trace-cmd

Visualization with LTTng and Trace Compass

Recording a kernel tracing session with LTTng

Reporting with a GUI – Trace Compass

Visualizing with trace-cmd

Recording a sample session with trace-cmd record

Reporting and interpretation with trace-cmd report (CLI)

Reporting and interpretation with a GUI frontend

Understanding, querying, and setting the CPU affinity mask

Querying and setting a thread's CPU affinity mask

Using taskset(1) to perform CPU affinity

Setting the CPU affinity mask on a kernel thread

Querying and setting a thread’s scheduling policy and priority

Within the kernel – on a kernel thread

CPU bandwidth control with cgroups

Looking up cgroups v2 on a Linux system

Trying it out – a cgroups v2 CPU controller

Converting mainline Linux into an RTOS

Building RTL for the mainline 5.x kernel (on x86_64)

Obtaining the RTL patches

Applying the RTL patch

Configuring and building the RTL kernel

Mainline and RTL – technical differences summarized

Latency and its measurement

Measuring scheduling latency with cyclictest

Getting and applying the RTL patchset

Installing cyclictest (and other required packages) on the device

Running the test cases

Viewing the results

Measuring scheduler latency via modern BPF tools

Summary

Questions

Further reading

Section 3: Delving Deeper

Kernel Synchronization - Part 1

Critical sections, exclusive execution, and atomicity

What is a critical section?

A classic case – the global i ++

Concepts – the lock

A summary of key points

Concurrency concerns within the Linux kernel

Multicore SMP systems and data races

Preemptible kernels, blocking I/O, and data races

Hardware interrupts and data races

Locking guidelines and deadlocks

Mutex or spinlock? Which to use when

Determining which lock to use – in theory

Determining which lock to use – in practice

Using the mutex lock

Initializing the mutex lock

Correctly using the mutex lock

Mutex lock and unlock APIs and their usage

Mutex lock – via [un]interruptible sleep?

Mutex locking – an example driver

The mutex lock – a few remaining points

Mutex lock API variants

The mutex trylock variant

The mutex interruptible and killable variants

The mutex io variant

The semaphore and the mutex

Priority inversion and the RT-mutex

Internal design

Using the spinlock

Spinlock – simple usage

Spinlock – an example driver

Test – sleep in an atomic context

Testing on a 5.4 debug kernel

Testing on a 5.4 non-debug distro kernel

Locking and interrupts

Using spinlocks – a quick summary

Summary

Questions

Further reading

Kernel Synchronization - Part 2

Using the atomic_t and refcount_t interfaces

The newer refcount_t versus older atomic_t interfaces

The simpler atomic_t and refcount_t interfaces

Examples of using refcount_t within the kernel code base

64-bit atomic integer operators

Using the RMW atomic operators

RMW atomic operations – operating on device registers

Using the RMW bitwise operators

Using bitwise atomic operators – an example

Efficiently searching a bitmask

Using the reader-writer spinlock

Reader-writer spinlock interfaces

A word of caution

The reader-writer semaphore

Cache effects and false sharing

Lock-free programming with per-CPU variables

Per-CPU variables

Working with per-CPU

Allocating, initialization, and freeing per-CPU variables

Performing I/O (reads and writes) on per-CPU variables

Per-CPU – an example kernel module

Per-CPU usage within the kernel

Lock debugging within the kernel

Configuring a debug kernel for lock debugging

The lock validator lockdep – catching locking issues early

Examples – catching deadlock bugs with lockdep

Example 1 – catching a self deadlock bug with lockdep

Fixing it

Example 2 – catching an AB-BA deadlock with lockdep

lockdep – annotations and issues

lockdep annotations

lockdep issues

Lock statistics

Viewing lock stats

Memory barriers – an introduction

An example of using memory barriers in a device driver

Summary

Questions

Further reading

About Packt

Why subscribe?

Other Books You May Enjoy

Leave a review - let other readers know what you think

Preface

This book has been explicitly written with a view to helping you learn Linux kernel development in a practical, hands-on fashion, along with the necessary theoretical background to give you a well-rounded view of this vast and interesting topic area. It deliberately focuses on kernel development via the powerful Loadable Kernel Module (LKM) framework; the vast majority of kernel projects and products, which includes device driver development, are done in this manner.

The focus is kept on both working hands-on with, and understanding at a sufficiently deep level, the internals of the Linux OS. In these regards, we cover everything from building the Linux kernel from source through understanding and working with complex topics such as synchronization within the kernel.

To guide you on this exciting journey, we divide this book into three sections. The first section covers the basics – setting up a workspace required for kernel development, building the kernel from source, and writing your first kernel module.

The next section, a key one, will help you understand important and essential kernel internals – the Linux kernel architecture, the task structure, and user and kernel-mode stacks. Memory management is a key and interesting topic – we devote three whole chapters to it (covering the internals to a sufficient extent, and importantly, how exactly to allocate any free kernel memory). The working and deeper details of CPU scheduling on Linux round off this section.

The last section of the book deals with the more advanced topic of kernel synchronization – a necessity for professional design and code on the Linux kernel. We devote two whole chapters to covering key topics within this.

The book uses the, at the time of writing, latest 5.4 Long Term Support (LTS) Linux kernel. It's a kernel that will be maintained (both bug and security fixes) from November 2019 right through December 2025! This is a key point, ensuring that this book's content remains current and valid for years to come!

We very much believe in a hands-on approach: over 20 kernel modules (besides several user apps and shell scripts) on this book's GitHub repository make the learning come alive, making it fun, interesting, and useful.

We highly recommend you also make use of this book's companion guide, Linux Kernel Programming (Part 2).

It's an excellent industry-aligned beginner's guide to writing misc character drivers, performing I/O on peripheral chip memory and handling hardware interrupts. You can get this book for free along with your copy, alternately you can also find this eBook in the GitHub repository at: https://github.com/PacktPublishing/Linux-Kernel-Programming/tree/master/Linux-Kernel-Programming-(Part-2).

We really hope you learn from and enjoy this book. Happy reading!

Who this book is for

This book is primarily for those of you beginning your journey in the vast arena of Linux kernel module development and, to some extent, Linux device driver development. It's also very much targeted at those of you who have already been working on Linux modules and/or drivers, who wish to gain a much deeper, well-structured understanding of Linux kernel architecture, memory management, and synchronization. This level of knowledge about the underlying OS, covered in a properly structured manner, will help you no end when you face difficult-to-debug real-world situations.

What this book covers

Chapter 1, Kernel Workspace Setup, guides you on setting up a full-fledged Linux kernel development workspace (typically, as a fully virtualized guest system). You will learn how to install all required software packages on it, including a cross toolchain. You will also learn about several other open source projects that will be useful on your journey to becoming a professional kernel/driver developer. Once this chapter is done, you will be ready to build a Linux kernel as well as to start writing and testing kernel code (via the loadable kernel module framework). In our view, it's very important for you to actually use this book in a hands-on fashion, trying out and experimenting with code. The best way to learn something is to do so empirically – not taking anyone's word on anything at all, but by trying it out and experiencing it for yourself.

Chapter 2, Building the 5.x Linux Kernel from Source – Part 1, is the first part of explaining how to build the modern Linux kernel from scratch with source code. In this part, you will be given necessary background information – the version nomenclature, the different source trees, the layout of the kernel source – on the kernel source tree. Next, you will be shown in detail how exactly to download a stable vanilla Linux kernel source tree onto the VM. We shall then learn a little regarding the layout of the kernel source code, getting, in effect, a 10,000-foot view of the kernel code base. The actual work of extracting and configuring the Linux kernel then follows. Creating and using a custom menu entry for kernel configuration is also shown.

Chapter 3, Building the 5.x Linux Kernel from Source – Part 2, is the second part on performing kernel builds from source code. In this part, you will continue from the previous chapter, now actually building the kernel, installing kernel modules, understanding what exactly initramfs (initrd) is and how to generate it, as well as setting up the bootloader (for x86). Also, as a valuable add-on, this chapter then explains how to cross-compile the kernel for a typical embedded ARM target (using the popular Raspberry Pi as a target device). Several tips and tricks on kernel builds, and even kernel security (hardening), are mentioned as well.

Chapter 4, Writing Your First Kernel Module – LKMs Part 1, is the first of two parts that cover a fundamental aspect of Linux kernel development – the LKM framework, and how it is to be understood and used by the module user, by you – the kernel module or device driver programmer. It covers the basics of the Linux kernel architecture and then, in great detail, every step involved in writing a simple Hello, world kernel module, compiling, inserting, checking, and removing it from the kernel space. We also cover kernel logging via the ubiquitous printk API in detail.

Chapter 5, Writing Your First Kernel Module – LKMs Part 2, is the second part that covers the LKM framework. Here, we begin with something critical – learning how to use a better Makefile, which will help you generate more robust code (having several code-checking, correction, static analysis targets, and so on). We then show in detail the steps to successfully cross-compile a kernel module for an alternate architecture, how to emulate library-like code in the kernel (via both the linking and the module-stacking approaches), defining and using passing parameters to your kernel module. Additional topics include the auto-loading of modules at boot, important security guidelines, and some information on the kernel documentation and how to access it. Several example kernel modules make the learning more interesting.

Chapter 6, Kernel Internals Essentials – Processes and Threads, delves into some essential kernel internals topics. We begin with what is meant by execution in process and interrupt contexts, and minimal but required coverage of the process user virtual address space (VAS) layout. This sets the stage for you; you'll then learn about Linux kernel architecture in more depth, focusing on the organization of process/thread task structures and their corresponding stacks – user- and kernel-mode. We then show you more on the kernel task structure (a root data structure), how to practically glean information from it, and even iterate over various (task) lists. Several kernel modules make the topic come alive.

Chapter 7, Memory Management Internals – Essentials, a key chapter, delves into essential internals of the Linux memory management subsystem, to the level of detail required for the typical module author or driver developer. This coverage is thus necessarily more theoretical in nature; nevertheless, the knowledge gained here is crucial to you, the kernel developer, both for deep understanding and usage of appropriate kernel memory APIs as well as for performing meaningful debugging at the level of the kernel. We cover the VM split (and how it is on various actual architectures), gaining deep insight into the user VAS (our procmap utility will be an eye-opener), as well as the kernel segment (or kernel VAS). We then briefly delve into the security technique of memory layout randomization ([K]ASLR), and end this chapter with a discussion on physical memory organization within Linux.

Chapter 8, Kernel Memory Allocation for Module Authors Part 1, gets our hands dirty with the kernel memory allocation (and obviously, deallocation) APIs. You will first learn about the two allocation layers within Linux – the slab allocator that's layered above the kernel memory allocation engine, and the page allocator (or BSA). We shall briefly learn about the underpinnings of the page allocator algorithm and its freelist data structure; this information is valuable when deciding which layer to use. Next, we dive straight into the hands-on work of learning about the usage of these key APIs. The ideas behind the slab allocator (or cache) and the primary kernel allocator APIs – the kzalloc/kfree – are covered. Importantly, the size limitations, downsides, and caveats when using these common APIs are covered in detail as well. Also, especially useful for driver authors, we cover the kernel's modern resource-managed memory allocation APIs (the devm_*() routines).

Chapter 9, Kernel Memory Allocation for Module Authors Part 2, goes further, in a logical fashion, from the previous chapter. Here, you will learn how to create custom slab caches (useful for high-frequency (de)allocations for, say, a custom driver), along with some help regarding debugging memory allocations at the slab layer. Next, you'll understand and use the vmalloc() API (and friends). Very importantly, having covered many APIs for kernel memory (de)allocation, you will now learn how to pick and choose an appropriate API given the real-world situation you find yourself in. This chapter is rounded off with important coverage of the kernel's Out Of Memory (OOM) killer framework. Understanding it will also lead to a much deeper understanding of how user space memory allocation really works, via the demand paging technique.

Chapter 10, The CPU Scheduler - Part 1, the first part of two chapters, covers a useful mix of theory and practice regarding CPU scheduling on the Linux OS. The minimal necessary theoretical background on the thread as the KSE and available kernel scheduling policies are topics initially covered. Next, sufficient kernel internal details on CPU scheduling are covered to have you understand how scheduling on the modern Linux OS works. Along the way, you will learn how to visualize PU scheduling with powerful tools such as perf; thread scheduling attributes (policy and real-time priority) are delved into as well. 

Chapter 11, The CPU Scheduler – Part 2, the second part on CPU scheduling, continues to cover the topic in more depth. Here, we cover further visualization tools for CPU scheduling (leveraging powerful software such as LTTng and the trace-cmd utility). Next, the CPU affinity mask and how to query/set it, controlling scheduling policy and priority on a per-thread basis – such a powerful feature! – are delved into. An overview of the meaning and importance of control groups (cgroups), along with an interesting example on CPU bandwidth allocation via cgroups v2 is seen. Can you run Linux as an RTOS? Indeed you can! The details on actually doing so are then shown. We round off this chapter with a discussion on (scheduling) latencies and how to measure them.

Chapter 12, Kernel Synchronization – Part 1, first covers the key concepts regarding critical sections, atomicity, what a lock conceptually achieves and, very importantly, the why of all this. We then cover concurrency concerns when working within the Linux kernel; this moves us naturally on to important locking guidelines, what deadlock means, and key approaches to preventing deadlock. Two of the most popular kernel locking technologies – the mutex lock and the spinlock – are then discussed in depth along with several (driver) code examples.

Chapter 13, Kernel Synchronization – Part 2, continues the journey on kernel synchronization. Here, you'll learn about key locking optimizations – using lightweight atomic and (the more recent) refcount operators to safely operate on integers, RMW bit operators to safely perform bit ops, and the usage of the reader-writer spinlock over the regular one. Inherent risks, such as cache false sharing are discussed as well. An overview of lock-free programming techniques (with an emphasis on per-CPU variables and their usage, along with examples) is then covered. A critical topic – lock debugging techniques, including the usage of the kernel's powerful lockdep lock validator, is then covered. The chapter is rounded off with a brief look at memory barriers (along with an example).

To get the most out of this book

To get the most out of this book, we expect you to have knowledge and experience in the following:

Know your way around a Linux system, on the command line (the shell).

The C programming language.

It's not mandatory but experience with Linux system programming concepts and technologies will greatly help.

The details on hardware and software requirements, as well as their installation, are covered completely and in depth in Chapter 1, Kernel Workspace Setup. It's critical that you read it in detail and follow the instructions therein.

Also, we have tested all the code in this book (it has its own GitHub repository as well) on these platforms:

x86_64 Ubuntu 18.04 LTS guest OS (running on Oracle VirtualBox 6.1)

x86_64 Ubuntu 20.04.1 LTS guest OS (running on Oracle VirtualBox 6.1)

x86_64 Ubuntu 20.04.1 LTS native OS

ARM Raspberry Pi 3B+ (running both its distro kernel as well as our custom 5.4 kernel); lightly tested

x86_64 CentOS 8 guest OS (running on Oracle VirtualBox 6.1); lightly tested

We assume that, when running Linux as a guest (VM), the host system is either Windows 10 or later (of course, even Windows 7 will work), or a recent Linux distribution (for example, Ubuntu or Fedora), or even macOS.

If you are using the digital version of this book, we advise you to type the code yourself or, better, access the code via the GitHub repository (link available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

I strongly recommend that you follow the empirical approach: not taking anyone's word on anything at all, but trying it out and experiencing it for yourself. Hence, this book gives you many hands-on experiments and kernel code examples that you can and must try out yourself; this will greatly aid you in making real progress and deeply learning and understanding various aspects of Linux kernel development.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Linux-Kernel-Programming. In case there's an update to the code, it will be updated on the existing GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: http://www.packtpub.com/sites/default/files/downloads/9781789953435_ColorImages.pdf.

Conventions used

There are a number of text conventions used throughout this book.

CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: The ioremap() API returns a KVA of the void * type (since it's an address location)

A block of code is set as follows:

static int __init miscdrv_init(void)

{

    int ret;

    struct device *dev;

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

#define pr_fmt(fmt) %s:%s(): fmt, KBUILD_MODNAME, __func__

[...]

#include

#include            

[...]

Any command-line input or output is written as follows:

pi@raspberrypi:~ $ sudo cat /proc/iomem

Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: Select System info from the Administration panel.

Warnings or important notes appear like this.

Tips and tricks appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [email protected].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Bewertungen

Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!

For more information about Packt, please visit packt.com.

Section 1: The Basics

Here, you will learn how to perform basic kernel development tasks. You will set up a kernel development workspace, build a Linux kernel from source, learn about the LKM framework, and write a Hello, world kernel module.

This section comprises the following chapters:

Chapter 1, Kernel Workspace Setup

Chapter 2, Building the 5.x Linux Kernel from Source, Part 1

Chapter 3, Building the 5.x Linux Kernel from Source, Part 2

Chapter 4, Writing Your First Kernel Module – LKMs Part 1

Chapter 5, Writing Your First Kernel Module – LKMs Part 2

We highly recommend you also make use of this book's companion guide, Linux Kernel Programming (Part 2).

It's an excellent industry-aligned beginner's guide to writing misc character drivers, performing I/O on peripheral chip memory and handling hardware interrupts. This book is primarily for Linux programmers beginning to find their way with device driver development. Linux device driver developers looking to overcome frequent and common kernel/driver development issues, as well as understand and learn to perform common driver tasks - the modern Linux Device Model (LDM) framework, user-kernel interfaces, performing peripheral I/O, handling hardware interrupts, dealing with concurrency and more - will benefit from this book. A basic understanding of Linux kernel internals (and common APIs), kernel module development and C programming is required.

You can get this book for free along with your copy, alternately you can also find this eBook in the GitHub repository: https://github.com/PacktPublishing/Linux-Kernel-Programming/tree/master/Linux-Kernel-Programming-(Part-2).

Kernel Workspace Setup

Hello, and welcome to this book on learning Linux kernel development. To get the most out of this book, it is very important that you first set up the workspace environment that we will be using throughout the book. This chapter will teach you exactly how to do this and get started.

We will install a recent Linux distribution, preferably as a Virtual Machine (VM), and set it up to include all the required software packages. We will also clone this book's code repository on GitHub, and learn about a few useful projects that will help along this journey.

The best way to learn something is to do so empirically – not taking anyone's word on anything at all, but trying it out and experiencing it for yourself. Hence, this book gives you many hands-on experiments and kernel code examples that you can and indeed must try out yourself; this will greatly aid in your making real progress and deeply learning and understanding various aspects of Linux kernel and driver development. So, let's begin!

This chapter will take us through the following topics, which will help us set up our environment:

Running Linux as a guest VM

Setting up the software – distribution and packages

A few additional useful projects

Technical requirements

You will need a modern desktop PC or laptop. Ubuntu Desktop specifies the following as recommended system requirements for the installation and usage of the distribution:

A 2 GHz dual core processor or better.

RAM:

Running on physical host: 2 GB or more system memory (more will certainly help).

Running as a guest VM: The host system should have at least 4 GB RAM (the more the better and the smoother the experience).

25 GB of free hard drive space (I suggest more, at least double this).

Either a DVD drive or a USB port for the installer media (not required when setting up Ubuntu as a guest VM).

Internet access is definitely helpful and required at times.

As performing tasks such as building a Linux kernel from source is a very memory- and CPU-intensive process, I highly recommend that you try it out on a powerful Linux system with plenty of RAM and disk space to spare as well. It should be pretty obvious – the more RAM and CPU power the host system has, the better!

Like any seasoned kernel contributor, I would say that working on a native Linux system is best. However, for the purposes of this book, we cannot assume that you will always have a dedicated native Linux box available to you. So, we will assume that you are working on a Linux guest. Working within a guest VM also adds an additional layer of isolation and thus safety. 

Cloning our code repository: The complete source code for this book is freely available on GitHub at https://github.com/PacktPublishing/Linux-Kernel-Programming. You can clone and work on it by cloning the git tree, like so:

git clone https://github.com/PacktPublishing/Linux-Kernel-Programming.git

The source code is organized chapter-wise. Each chapter is represented as a directory – for example, ch1/ has the source code for this chapter. The root of the source tree has some code that is common to all chapters, such as the source files convenient.h, klib_llkd.c, as well as others.

For efficient code browsing, I would strongly recommend that you always index the code base with ctags(1) and/or cscope(1). For example, to set up the ctags index, just cd to the root of the source tree and type ctags -R .

Unless noted otherwise, the code output we show in the book is the output as seen on an x86-64 Ubuntu 18.04.3 LTS guest VM (running under Oracle VirtualBox 6.1). You should realize that due to (usually minor) distribution – and even within the same distributions but differing versions – differences, the output shown here may not perfectly match what you see on your Linux system.

Running Linux as a guest VM

As discussed previously, a practical and convenient alternative to using a native Linux system is to install and use the Linux distribution as a guest OS on a VM. It's key that you install a recent Linux distribution, preferably as a VM to be safe and avoid unpleasant data loss or other surprises. The fact is when working at the level of the kernel, abruptly crashing the system (and the data loss risks that arise thereof) is actually a commonplace occurrence. I recommend using Oracle VirtualBox 6.x (or the latest stable version) or other virtualization software, such as VMware Workstation.

Both of these are freely available. It's just that the code for this book has been tested on VirtualBox 6.1. Oracle VirtualBox is considered Open Source Software (OSS) and is licensed under the GPL v2 (the same as the Linux kernel). You can download it from https://www.virtualbox.org/wiki/Downloads. Its documentation can be found here: https://www.virtualbox.org/wiki/End-user_documentation.

The host system should be either MS Windows 10 or later (of course, even Windows 7 will work), a recent Linux distribution (for example, Ubuntu or Fedora), or macOS. So, let's get started by installing our Linux guest.

Installing a 64-bit Linux guest

Here, I won't delve into the minutiae of installing Linux as a guest on Oracle VirtualBox, the reason being that this installation is not directly related to Linux kernel development. There are many ways to set up a Linux VM; we really don't want to get into the details and the pros and cons of each of them here.

But if you are not familiar with this, don't worry. For your convenience, here are some excellent resources that will help you out:

A very clearly written tutorial entitled Install Linux Inside Windows Using VirtualBox by Abhishek Prakash (It's FOSS!, August 2019): https://itsfoss.com/install-linux-in-virtualbox/.

An alternate, similarly excellent resource is Install Ubuntu on Oracle VirtualBox:https://brb.nci.nih.gov/seqtools/installUbuntu.html.

Also, you can look up useful resources for installing a Linux guest on VirtualBox in the Further reading section at the end of this chapter. 

While you install the Linux VM, keep the following things in mind.

Turn on your x86 system's virtualization extension support 

Installing a 64-bit Linux guest requires that CPU virtualization extension support (Intel VT-x or AMD-SV) be turned on within the host system's basic input/output system (BIOS) settings. Let's see how to do this:

Our first step is to ensure that our CPU supports virtualization:

There are two broad ways to check this while on a Windows host:

One, run the Task Manager app and switch to the Performance tab. Below the CPU graph, you will see, among several other things, Virtualization, with Enabled or Disabled following it.

A second way to check on Windows systems is to open a Command window (cmd). In Command Prompt, type systeminfo and press Enter. Among the output seen will be the Virtualization Enabled in firmware line. It will be followed by either Yes or No.

To check this while on a Linux host, from Terminal, issue the following commands (processor virtualization extension support: vmx is the check for Intel processors, smv is the check for AMD processors):

egrep --color vmx|svm /proc/cpuinfo

For Intel CPUs, the vmx flag will show up (in color) if virtualization is supported. In the case of AMD CPUs, svm will show up (in color). With this, we know that our CPU supports virtualization. But in order to use it, we need to enable it in the computer BIOS.

Enter the BIOS by pressing DelorF12while booting (the precise key to press varies with the BIOS). Please refer to your system's manual to see which key to use. Search for terms such as Virtualization or Virtualization Technology (VT-x). Here is an example for Award BIOS:

Figure 1.1 – Setting the BIOS Virtualization option to the Enabled state

If you are using an Asus EFI-BIOS, you will have to set the entry to [Enabled] if it is not set by default. Visit https://superuser.com/questions/367290/how-to-enable-hardware-virtualization-on-asus-motherboard/375351#375351.

Now, choose to use hardware virtualization in VirtualBox's Settings menu for your VM. To do this, click on System and then Acceleration. After that, check the boxes, as shown in the following screenshot:

Figure 1.2 – Enabling hardware virtualization options within the VirtualBox VM settings

This is how we enable our host processor's hardware virtualization features for optimal performance.

Allocate sufficient space to the disk

For most desktop/laptop systems, allocating a gigabyte of RAM and two CPUs to the guest VM should be sufficient.

However, when allocating space for the guest's disk, please be generous. Instead of the usual/default 8 GB suggested, I strongly recommend you make it 50 GB or even more. Of course, this implies that the host system has more disk space than this available! Also, you can specify this amount to be dynamically allocated or allocated on-demand. The hypervisor will grow the virtual disk optimally, not giving it the entire space to begin with.

Install the Oracle VirtualBox Guest Additions

For best performance, it's important to install the Oracle VirtualBox Guest Additions as well within the guest VM. These are essentially para-virtualization accelerator software, which greatly helps with optimal performance. Let's see how to do this on an Ubuntu guest session:

First, update your Ubuntu guest OS's software packages. You can do so using the following command:

sudo apt update

sudo apt upgrade

On completion, reboot your Ubuntu guest OS and then install the required packages using the following command:

sudo apt install build-essential dkms linux-headers-$(uname -r)

Now, from the VMmenu bar, go toDevices | Insert Guest Additions CD image.... This will mount theGuest Additions ISOfile inside your VM. The following screenshot shows what it looks like doing this: 

Figure 1.3 – VirtualBox | Devices | Insert Guest Additions CD image

Now, a dialog window will pop up that will prompt you to runthe installer in order to launch it. Select Run.

The Guest Additions installation will now take place in a Terminal window that shows up. Once complete, hit the Enter key to close the window. Then, power off yourUbuntuguest OS in order to change some settings from the VirtualBox manager, as explained next.

Now, to enableShared ClipboardandDrag'n'Dropfunctionalities between the guest and host machines, go toGeneral | Advancedand enable the two options (Shared ClipboardandDrag'n'Drop) as you wish with the dropdowns:

Figure 1.4 – VirtualBox: enabling functionality between the host and guest

Then, clickOKto save the settings. Now boot into your guest system, log in, and test that everything is working fine.

As of the time of writing, Fedora 29 has an issue with the installation of the vboxsf kernel module required for the Shared Folders feature. I refer you to the following resource to attempt to rectify the situation: Bug 1576832 - virtualbox-guest-additions does not mount shared folder (https://bugzilla.redhat.com/show_bug.cgi?id=1576832).

If this refuses to work, you can simply transfer files between your host and guest VM over SSH (using scp(1)); to do so, install and start up the SSH daemon with the following commands:

sudo yum install openssh-server

sudo systemctl start sshd

Remember to update the guest VM regularly and when prompted. This is an essential security requirement. You can do so manually by using the following: 

sudo /usr/bin/update-manager

Finally, to be safe, please do not keep any important data on the guest VM. We will be working on kernel development. Crashing the guest kernel is actually a commonplace occurrence. While this usually does not cause data loss, you can never tell! To be safe, always back up any important data. This applies to Fedora as well. To learn how to install Fedora as a VirtualBox guest, visit https://fedoramagazine.org/install-fedora-virtualbox-guest/.

Sometimes, especially when the overhead of the X Window System (or Wayland) GUI is too high, it's preferable to simply work in console mode. You can do so by appending 3 (the run level) to the kernel command line via the bootloader. However, working in console mode within VirtualBox may not be that pleasant an experience (for one, the clipboard is unavailable, and the screen size and fonts are less than desirable). Thus, simply doing a remote login (via ssh, putty, or equivalent) into the VM from the host system can be a great way to work.

Experimenting with the Raspberry Pi

The Raspberry Pi is a popular credit card-sized Single-Board Computer (SBC), much like a small-factor PC that has USB ports, a microSD card, HDMI, audio, Ethernet, GPIO, and more. The System on Chip (SoC) that powers it is from Broadcom, and in it is an ARM core or cluster of cores. Though not mandatory, of course, in this book, we strive to also test and run our code on a Raspberry Pi 3 Model B+ target. Running your code on different target architectures is always a good eye-opener to possible defects and helps with testing. I encourage you to do the same:

Figure 1.5 – The Raspberry Pi with a USB-to-serial adapter cable attached to its GPIO pins

You can work on the Raspberry Pi target either using a digital monitor/TV via HDMI as the output device and a traditional keyboard/mouse over its USB ports or, more commonly for developers, over a remote shell via ssh(1). However, the SSH approach does not cut it in all circumstances. Having a serial console on the Raspberry Pi helps, especially when doing kernel debugging.

I would recommend that you check out the following article, which will help you set up a USB-to-serial connection, thus getting a console login to the Raspberry Pi from a PC/laptop: WORKING ON THE CONSOLE WITH THE RASPBERRY PI, kaiwanTECH: https://kaiwantech.wordpress.com/2018/12/16/working-on-the-console-with-the-raspberry-pi/.

To set up your Raspberry Pi, please refer to the official documentation: https://www.raspberrypi.org/documentation/. Our Raspberry Pi system runs the official Raspbian (Debian for Raspberry Pi) Linux OS with a recent (as of the time of writing) 4.14 Linux kernel. On the console of the Raspberry Pi, we run the following commands:

rpi $ lsb_release -a

No LSB modules are available.

Distributor ID: Raspbian

Description:    Raspbian GNU/Linux 9.6 (stretch)

Release:        9.6

Codename:      stretch

rpi $ uname -a

Linux raspberrypi 4.14.79-v7+ #1159 SMP Sun Nov 4 17:50:20 GMT 2018 armv7l GNU/Linux

rpi $

What if you don't have a Raspberry Pi, or it's not handy? Well, there's always a way – emulation! Though not as good as having the real thing, emulating the Raspberry Pi with the powerful Free and Open Source Software (FOSS) emulator called QEMU or Quick Emulator is a nice way to get started, at least.

As the details of setting up the emulated Raspberry Pi via QEMU go beyond the scope of this book, we will not be covering it. However, you can check out the following links to find out more: Emulating Raspberry Pi on Linux: http://embedonix.com/articles/linux/emulating-raspberry-pi-on-linux/ and qemu-rpi-kernel, GitHub: https://github.com/dhruvvyas90/qemu-rpi-kernel/wiki.

Also, of course, you do not have to confine yourself to the Raspberry Pi family; there are several other excellent prototyping boards available. One that springs to mind is the popular BeagleBone Black (BBB) board.

In fact, for professional development and product work, the Raspberry Pi is really not the best choice, for several reasons... a bit of googling will help you understand this. Having said that, as a learning and basic prototyping environment it's hard to beat, with the strong community (and tech hobbyist) support it enjoys.

Several modern choices of microprocessors for embedded Linux (and much more) are discussed and contrasted in this excellent in-depth article: SO YOU WANT TO BUILD AN EMBEDDED LINUX SYSTEM?, Jay Carlson, Oct 2020 : https://jaycarlson.net/embedded-linux/; do check it out.

By now, I expect that you have set up Linux as a guest machine (or are using a native test Linux box) and have cloned the book's GitHub code repository. So far, we have covered some information regarding setting up Linux as a guest VM (as well as optionally using boards such as the Raspberry Pi or the BeagleBone). Let's now move on to a key step: actually installing software components on our Linux guest system so that we can learn and write Linux kernel code on the system!

Setting up the software – distribution and packages

It is recommended to use one of the following or later stable version Linux distributions. As mentioned in the previous section, they can always be installed as a guest OS on a Windows or Linux host system, with the clear first choice being Ubuntu Linux 18.04 LTS Desktop. The following screenshot shows you the recommended version and the user interface:

Figure 1.6 – Oracle VirtualBox 6.1 running Ubuntu 18.04.4 LTS as a guest VM

The preceding version – Ubuntu 18.04 LTS Desktop – is the version of choice for this book, at least.  The two primary reasons for this are straightforward:

Ubuntu Linux is one of the, if not the, most popular Linux (kernel) development workstation environments in industry use today.

We cannot always, for lack of space and clarity, show the code/build output of multiple environments in this book. Hence, we have chosen to show the output as seen on Ubuntu 18.04 LTS Desktop.

Ubuntu 16.04 LTS Desktop is a good choice too (it has Long-Term Support (LTS) as well), and everything should work. To download it, visit https://www.ubuntu.com/download/desktop.

Some other Linux distributions that can also be considered include the following:

CentOS 8 Linux (not CentOS Stream): CentOS Linux is a distribution that's essentially a clone of the popular enterprise server distribution from RedHat (RHEL 8, in our case). You can download it from here: https://www.centos.org/download/.

Fedora Workstation: Fedora is a very well-known FOSS Linux distribution as well. You can think of it as being a kind of test-bed for projects and code that will eventually land up within RedHat's enterprise products. Download it from https://getfedora.org/ (download the Fedora Workstation image).

Raspberry Pi as a target: It's really best to refer to the official documentation to set up your Raspberry Pi (Raspberry Pi documentation: https://www.raspberrypi.org/documentation/). It's perhaps worth noting that Raspberry Pi kits are widely available that come completely pre-installed and with some hardware accessories as well. 

If you want to learn how to install a Raspberry Pi OS image on an SD card, visit https://www.raspberrypi.org/documentation/installation/installing-images/.

BeagleBone Black as a target: The BBB is, like the Raspberry Pi, an extremely popular embedded ARM SBC for hobbyists and pros. You can get started here: https://beagleboard.org/black. The System Reference Manual for the BBB can be found here: https://cdn.sparkfun.com/datasheets/Dev/Beagle/BBB_SRM_C.pdf. Though we don't present examples running on the BBB, nevertheless, it's a valid embedded Linux system that, once properly set up, you can run this book's code on.

Before we conclude our discussion on selecting our software distribution for the book, here are a few more points to note:

These distributions are, in their default form, FOSS and non-proprietary, and free to use as an end user.

Though our aim is to be Linux distribution-neutral, the code has only been tested on Ubuntu 18.04 LTS and lightly tested on CentOS 8, and a Raspberry Pi 3 Model B+ running the Raspbian GNU/Linux 9.9 (stretch) Debian-based Linux OS.

We will, as far as is possible, use the very latest (as of the time of writing) stable LTS

Linux kernel version 5.4 for our kernel build and code runs. Being an LTS kernel, the 5.4 kernel is an excellent choice to run on and learn with.

It is interesting to know that the 5.4 LTS kernel will indeed have a long lifespan; from November 2019 right up to December 2025! This is good news: this book's content remains current and valid for years to come!

For this book, we'll log in as the user account named llkd.

It's important to realize, for maximized security (with the latest defenses and fixes), that you must run the most recent Long Term Support (LTS) kernel possible for your project or product.

Now that we