Vous êtes sur la page 1sur 208

Constructing an Embedded Linux System

Embedded Linux from Scratch

Bill Gatliff
bgat@billgatliff.com

Freelance Embedded Systems Developer

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 1 / 59


Overview

Build a Linux system:


• Fundamental concepts
• Source code
• Build environment
• Toolchain
• Linux kernel
• Target runtime environment

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 2 / 59


Fundamental concepts

Why do this?
• Provides best control over system configuration
• Not all setups are created equal
• Upgrade on your schedule
• Custom configuration tuned to your environment
• Better insight into system operation

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 3 / 59


Fundamental concepts

Why NOT to do this?


• Process is tedious and time consuming
• Not all source code combinations work out-of-the-box
• Consume lots of time building, rebuilding, rebuilding...

If you don’t need to, don’t bother!


• Purchase a baseline setup from a reputable vendor

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 4 / 59


Fundamental concepts

General procedure:
• Obtain all the required source code
• Set up a GNU cross-development toolchain
• Build a Linux kernel
• Build a Linux runtime environment
• Install the kernel, runtime on the target
• Boot!

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 5 / 59


Fundamental concepts

The configure script:


• Sets up a source tree for building
• Generally the first step in the build process
• Produces Makefile, config.h and others

• Not all GNU-compatible software uses configure

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 6 / 59


Fundamental concepts

Target specification:
<arch>-<variant>-<os>-<runtime>

i.e.,
--target=arm-xscale-linux-gnu
--target=arm-unknown-linux-gnu
--target=arm-linux

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 7 / 59


Fundamental concepts

Common target specifications:

arm-elf arm-linux
h8300-elf h8300-linux
mips-elf mips-linux
mipsel-elf mipsel-linux
ppc-elf ppc-linux
sh-elf sh-linux

This is not an exhaustive list!

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 8 / 59


Fundamental concepts

Target specification has to be consistent!


$ export TARGET=arm-linux

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 9 / 59


Fundamental concepts

Build specification:
<arch>-<variant>-<os>-<runtime>

i.e.,
--build=i686-pc-linux-gnu

• Often detected automatically during configuration

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 10 / 59


Fundamental concepts

Host specification:
<arch>-<variant>-<os>-<runtime>

i.e.,
--host=i686-pc-linux-gnu

• Defaults to identity of build machine

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 11 / 59


Fundamental concepts

Pay careful attention to host vs. target!


• For code generators, the host is the workstation
• For code generators, the target is the embedded system

• For libraries and code, the host is the embedded system

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 12 / 59


Fundamental concepts

Prefix specification:
• Defines the installation location for build products
• Must be an absolute path

i.e.,
--prefix=/opt/bgat/H-i686-pc-linux-gnu

• Defaults to /usr/local in most systems

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 13 / 59


Fundamental concepts

Prefix specification must be consistent!


$ export BUILD_PREFIX=\
/opt/bgat/H-i686-pc-linux-gnu

$ export ROOTDIR_PREFIX=\
/opt/bgat/H-arm-linux-gnu

$ export TARGET=arm-linux

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 14 / 59


Fundamental concepts

Prefix specification must be consistent!


$ .../configure --target=$TARGET \
--prefix=$BUILD_PREFIX

$ .../configure --host=$TARGET \
--prefix=$ROOTDIR_PREFIX

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 15 / 59


Fundamental concepts

Patch files:
• Bug fixes, added functionality
• Usually associated with bleeding-edge or archaic work

$ gzip -d fixes-something.patch.gz
$ patch -p 1 < fixes-something.patch

$ gzip -dc fixes-something.patch.gz | patch -p 1

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 16 / 59


Fundamental concepts

--- linux-2.6.13/drivers/net/arm/at91_ether.c.orig
+++ linux-2.6.13/drivers/net/arm/at91_ether.c
@@ -807,10 +807,11 @@
/*
* Initialize the ethernet interface
*/
-int at91ether_setup(int phy_type, struct at91_eth_data *board_data)
+int at91ether_setup(int phy_type, struct device *class_dev)
{
struct net_device *dev;
struct at91_private *lp;
+ struct at91_eth_data *board_data = class_dev->platform_data;
AT91PS_EMAC regs;
unsigned int val;
@@ -843,7 +844,7 @@
lp->board_data = *board_data;

spin_lock_init(&lp->lock);
-
+
ether_setup(dev);

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 17 / 59


What do we need?

Native toolchain:

• Debian, Fedora, RHEL, i.e. any Linux distribution


• NetBSD, FreeBSD, OpenBSD, et al.
• Solaris, OSX
• Cygwin, MinGW

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 18 / 59


What do we need?

Disk space on the workstation:


• 640MB ought be enough for anybody

Target memory:
• Target filesystem occupies 2MB and up
• Linux kernel needs 1MB-4MB

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 19 / 59


What do we need?

Patience!

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 20 / 59


What else do we need?

http://gnu.org
• binutils-2.16.1.tar.bz2
• gcc-4.1.1.tar.bz2
• gdb-6.6.tar.bz2
http://uclibc.org
• uclibc-0.9.29.tar.bz2
http://busybox.net
• busybox-1.7.0.tar.bz2

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 21 / 59


What else do we need?

Your Linux kernel’s source code:


• linux-2.6.23

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 22 / 59


Roadmap

Let’s get started!


• Build binutils
• Build a bootstrap gcc
• Build uClibc
• Build gcc
• Build gdb

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 23 / 59


Roadmap

Let’s get started!


• Build a Linux kernel for the target system
• Build a minimal target runtime environment
• Install the kernel and runtime environment

• Boot Linux on the target

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 24 / 59


Roadmap

Let’s get started!


• Celebrate!! :)

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 25 / 59


Warning!

Naïve tinkering can break your system!


• Don’t overwrite your native toolchain
• Don’t overwrite your dynamic linker

• Don’t EVER work as the root user!


• Don’t EVER work as the root user!
(Ok, sometimes it’s unavoidable)

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 26 / 59


Setup

A few handy environment variables:


• Saves typing
• Minimizes risk of obscure errors

$ export TARGET=arm-linux
$ export PREFIX=/opt/bgat/H-i686-pc-linux-gnu
$ export PATH=${PREFIX}/bin:${PATH}
$ export ROOTDIR_PREFIX=/opt/bgat/H-arm-linux-uclibc

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 27 / 59


Binutils

Provides:
• Assembler and linker
• Object and file management utilities

$ tar xjf binutils-x.y.z.tar.bz2


$ mkdir build-binutils && cd build-binutils

$ ../binutils-x.y.z/configure \
--target=$TARGET --prefix=$PREFIX \
--disable-nls 2>&1 | tee configure-log

$ make all install 2>&1 | tee make-log

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 28 / 59


Bootstrap GCC

Provides:
• Pure cross compiler

$ tar xjf gcc-x.y.z.tar.bz2


$ mkdir build-bootgcc && cd build-bootgcc

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 29 / 59


Build bootstrap compiler

Provides:
• Pure cross compiler

$ ../gcc-x.y.z/configure \
--target=$TARGET --prefix=$PREFIX \
--with-gnu-as --with-gnu-ld \
--with-local-prefix=$PREFIX/$TARGET \
--disable-multilib --disable-nls \
--enable-threads=no \
--enable-symvers=gnu --enable-__cxa_atexit \
--disable-shared --enable-languages=c \
2>&1 | tee configure-log
$ make all-gcc install-gcc 2>&1 | tee make-log

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 30 / 59


Kernel Headers

Provides:
• Linux-specific header files

$ tar xjf linux-x.y.z.tar.bz2


$ cd linux-x.y.z

$ make ARCH=arm csb726_defconfig


$ make ARCH=arm CROSS_COMPILE=${TARGET}- clean zImage
$ make ARCH=arm INSTALL_HDR_PATH=${PREFIX}/${TARGET} \
headers_install

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 31 / 59


uClibc

Provides:
• C runtime library

$ tar xjf uClibc-x.y.z.tar.bz2


$ cd uClibc-x.y.z

$ make menuconfig

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 32 / 59


Build uClibc

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 33 / 59


Build uClibc

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 34 / 59


Build uClibc

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 35 / 59


Build uClibc

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 36 / 59


Build uClibc

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 37 / 59


Build uClibc

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 38 / 59


Build uClibc

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 39 / 59


Build uClibc

Provides:
• C runtime library

$ make menuconfig
...
$ make

$ make PREFIX=${PREFIX}/${TARGET} install_dev

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 40 / 59


Cross Compiler

Provides:
• C/C++ compiler

$ mkdir build-gcc && cd build-gcc


$ ../gcc-x.y.z/configure \
--target=$TARGET --prefix=$PREFIX \
--with-gnu-as --with-gnu-ld \
--with-local-prefix=$PREFIX/$TARGET \
--disable-nls --enable-threads=posix \
--enable-symvers=gnu --enable-__cxa_atexit \
--enable-languages=c,c++ \
2>&1 | tee configure-log
$ make all install 2>&1 | tee make-log

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 41 / 59


Debugger

Provides:
• Cross debugger
• Instruction set simulator

$ tar xjf gdb-x.y.z.tar.bz2


$ mkdir build-gdb && cd build-gdb

$ ../gdb-x.y.z/configure \
--target=$TARGET --prefix=$PREFIX \
2>&1 | tee configure-log

$ make all install 2>&1 | tee make-log

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 42 / 59


“Hello, world!”

Provides:
• Test of the toolchain and runtime environment
• Remote debugging demonstration

$ ${TARGET}-gcc -g -static -o hello hello.c

$ file hello
hello: ELF 32-bit LSB executable, ARM, version 1 (ARM),
statically linked, not stripped

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 43 / 59


Build Linux kernel

Provides:
• Linux kernel
• Note: kernels < 2.6.14 might not build with gcc-4

$ cd linux-x.y.z
$ make ARCH=arm CROSS_COMPILE=${TARGET}- clean zImage

$ cp arch/arm/boot/zImage /tftpboot

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 44 / 59


Build a root filesystem tree

Provides:
• Location to install root filesystem components

$ mkdir -p $ROOTDIR_PREFIX
$ mkdir -p $ROOTDIR_PREFIX/dev

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 45 / 59


Create device nodes

Provides:
• Means for applications to connect to devices
• Essential for user applications

NFS, JFFS2, ext2 filesystems:


$ mknod $ROOTDIR_PREFIX/dev/tty c 5 0
$ mknod $ROOTDIR_PREFIX/dev/console c 5 1
$ mknod $ROOTDIR_PREFIX/dev/ttyS0 c 4 64
$ mknod $ROOTDIR_PREFIX/dev/null c 1 3

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 46 / 59


Create device nodes

Provides:
• Means for applications to connect to devices
• Essential for user applications

Romfs:
$ touch $ROOTDIR_PREFIX/dev/@tty,c,5,0
$ touch $ROOTDIR_PREFIX/dev/@console,c,5,1
$ touch $ROOTDIR_PREFIX/dev/@ttyS0,c,4,64
$ touch $ROOTDIR_PREFIX/dev/@null,c,1,3

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 47 / 59


Build and install Busybox

Provides:
• Basic command shell and utilities

$ tar xjf busybox-x.y.z.tar.bz2


$ cd busybox-x.y.z

$ make menuconfig

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 48 / 59


Build and install Busybox

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 49 / 59


Build and install Busybox

Provides:
• Basic command shell and utilities

$ make menuconfig
...
$ make CROSS_COMPILE=${TARGET}-
$ make CROSS_COMPILE=${TARGET}- \
CONFIG_PREFIX=${ROOTDIR_PREFIX} install

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 50 / 59


Install uClibc linker and libraries

Provides:
• Dynamic linker
• Shared libraries

$ cd uClibc-x.y.z
...
$ make PREFIX=${ROOTDIR_PREFIX} install_runtime

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 51 / 59


Filesystem image

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 52 / 59


Build ROMFS filesystem image

Provides:
• A source of Linux applications

$ genromfs -d $ROOTDIR_PREFIX \
-f /tftpboot/arm-linux-uclibc.img

Alternative:
• Export the filesystem via NFS

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 53 / 59


Install the filesystem image

Provides:
• User applications, runtime libraries
• Boot scripts, etc.

uMON> tftp -F romfs.img 192.168.4.254 get arm-linux-uclibc.img


Retrieving romfs-test.img from 192.168.4.254...
TFTP transfer complete.
Rcvd 812032 bytes
Adding romfs.img (size=812032) to TFS...

uMON> tfs ls
Name Size Location Flags Info
monrc 163 0x000412dc e envsetup
romfs.img 812032 0x000413dc
startlinux 4633 0x0004005c e

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 54 / 59


Install the Linux kernel

Provides:
• Linux!

uMON> tftp -F zImage 192.168.4.254 get zImage


TFTP transfer complete.
Rcvd 1530692 bytes
Adding zImage (size=1530692) to TFS...

uMON> tfs ls
Name Size Location Flags Info
monrc 163 0x000412dc e envsetup
romfs.img 812032 0x000f183c
startlinux 4633 0x0004005c e
zImage 1530692 0x001b7c9c

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 55 / 59


Launch the Linux kernel

Provides:
• Pure joy

uMON> tfs cp zImage $APPRAMBASE


uMON> call $APPRAMBASE 0 718
Linux 2.6.22-csb726.......................
Linux version 2.6.22-csb726 (bgat@mercury)
(gcc version 4.1.1) #19 Fri Aug 24 11:39:37 CST 2007
CPU: XScale-PXA255 [69052d06] revision 6 (ARMv5TE)
Machine: Cogent CSB726 Development Platform
Memory policy: ECC disabled, Data cache writeback
Memory clock: 99.53MHz (*27)
Run Mode clock: 199.07MHz (*2)
Turbo Mode clock: 398.13MHz (*2.0, active)
...

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 56 / 59


Launch the Linux kernel

...
Built 1 zonelists
Kernel command line: console=ttyS0,38400
ip=192.168.4.20:192.168.4.254:192.168.4.254:255.255.255.0::eth0:off
mtdparts=flash00:812032@989244(root)ro video=pxafb:mode:640x480-16
PID hash table entries: 512 (order: 9, 8192 bytes)
...
IP-Config: Complete:
device=eth0, addr=192.168.4.20, mask=255.255.255.0,
gw=192.168.4.254, host=192.168.4.20, domain=, nis-domain=(none),
bootserver=192.168.4.254, rootserver=192.168.4.254, rootpath=
...
VFS: Mounted root (romfs filesystem) readonly.
Freeing init memory: 112K

Please press Enter to activate this console.

BusyBox v1.01 (2006.01.06-18:15+0000) Built-in shell (ash)


Enter ’help’ for a list of built-in commands.

/ #

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 57 / 59


Resources

LXR cross-reference tool Crosstool


• http://lxr.linux.no • http://kegel.com/crosstool

ARM Linux website


Linux kernel mailing list
• http://arm.linux.org.uk
• http://lkml.org
Crossgcc mailing list
• http://sourceware.org/ml Linux From Scratch
My website • http://linuxfromscratch.org

• http://billgatliff.com
Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 58 / 59
Constructing an Embedded Linux System
Embedded Linux from Scratch

Bill Gatliff
bgat@billgatliff.com

Freelance Embedded Systems Developer

Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 59 / 59


Linux Hardware I/O via Mmap(2)
Introduction to User-Space Hardware I/O

Bill Gatliff
bgat@billgatliff.com

Freelance Embedded Systems Developer

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 1 / 40


Overview

Roadmap:
• Quick overview of protected memory
• “User mode” vs. “kernel mode”
• The const and volatile keywords
• The mmap(2) system call
• Security implications
• Code examples

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 2 / 40


What is “Protected Memory”?

Many Linux device drivers:


• Run from kernel memory
• Have no direct access to or from user memory

An interface:
• Bridges the gap between user and kernel memory
• Under Linux, implementation relies on a device node
• (We’ll come back to this in detail later)

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 3 / 40


What is “Protected Memory”?

Protected memory :
• The MMU keeps memory spaces separated
• A exception occurs if you go out-of-bounds

When an exception occurs:


• The kernel kills the user process, and/or,
• The kernel generates an OOPS message

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 4 / 40


What is “Protected Memory”?

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 5 / 40


What is “Protected Memory”?

User mode device drivers:


• User programs that access hardware

“But how can that be?!”


• Stay tuned! :)

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 6 / 40


Advantages and Disadvantages

Why run a driver in a user application?


• No kernel code required
• Potentially less risk to system stability
• Easier to manage driver development

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 7 / 40


Advantages and Disadvantages

Potentially less risk?


• You can still command the hardware to do something stoopid
• Interrupt hangs will (probably)(not) hang the kernel
• Linux will protect you from wild pointer dereferencing

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 8 / 40


Advantages and Disadvantages

Doesn’t promote “mainlining”:


• User applications don’t go in kernel source code!

No support for interrupt handlers:


• Requires a small amount of kernel code
• The majority can be in user memory, however

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 9 / 40


Advantages and Disadvantages

Performance:
• Depends on specifics of implementation
• May be better or worse than kernel code

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 10 / 40


Security Implications

“Is this a security risk?”


• No.
• Only privileged users will get access to device memory
• Others will be denied access per Linux security models
• Your system is no more or less secure than with alternatives

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 11 / 40


Two Approaches

/dev/mem and memmap(2)


• Ye olde skool way
• Straightforward, works well
• Supported by even archaic kernel versions
• Perfect for polled, slow devices

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 12 / 40


Two Approaches

#include<linux/uio_driver.h>
• Sysfs API for device-related information
• Builds on the memmap() approach
• Ideal for pluggable devices, esp. PCI
• Relatively new addition to the kernel (2.6.18-ish)

(We’ll come back to this later)

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 13 / 40


Two Approaches

“Which one do I use?”


• Long-lived drivers probably need uio_driver API
• It’s overkill for simple polling, however

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 14 / 40


The /dev/mem Device

The /dev/mem device:


• A device node
• Allows users to mmap() a physical address
• The read() and write() methods don’t work

void *mmap(void *addr, size_t length, int prot


int flags, int fd, off_t offset);

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 15 / 40


The /dev/mem Device

To control hardware:
• Call open(“/dev/mem”, ...);
• Use mmap(2) to map the device’s control registers
• Use pointer dereferencing to drive the device as always

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 16 / 40


The /dev/mem Device

int main (int argc, char *argv[])


{
/* mknod /dev/mem c 1 1 */
/* crw-r--r-- 1 root root 1, 1 2006-02-27 10:49 /dev/mem */
int fd = open("/dev/mem", O_RDWR | O_SYNC);

if (!fd)
{
perror("fd");
return -1;
}

unsigned long gpio


= (unsigned long)mmap(0, 0x1000, PROT_READ | PROT_WRITE,
MAP_SHARED, fd, GPIO);

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 17 / 40


The /dev/mem Device

unsigned long gpiod = gpio + GPIOD;


printf("gpiod mapped to %lx\n", gpiod);

volatile unsigned int* perd = PIO_PER(gpiod);


volatile unsigned int* psrd = PIO_PSR(gpiod);
volatile unsigned int* oerd = PIO_OER(gpiod);
volatile unsigned int* osrd = PIO_OSR(gpiod);
volatile unsigned int* sodrd = PIO_SODR(gpiod);
volatile unsigned int* codrd = PIO_CODR(gpiod);
volatile unsigned int* pdsrd = PIO_PDSR(gpiod);

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 18 / 40


The /dev/mem Device

#define LED (1 << 4) /* PD4 */

*perd = LED;
*oerd = LED;
printf("psrd: %x osrd: %x\n", *psrd, *osrd);

*codrd = LED;
*sodrd = LED;

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 19 / 40


Manpage

MMAP(2) Linux Programmer’s Manual MMAP(2)

NAME
mmap, munmap - map or unmap files or devices into memory

SYNOPSIS
#include <sys/mman.h>

void *mmap(void *start, size_t length, int prot,


int flags, int fd, off_t offset);
int munmap(void *start, size_t length);

DESCRIPTION
The mmap() function creates a new mapping in the virtual address
space of the calling process. The starting address for the new
mapping is specified in addr. The length argument specifies the
length of the mapping...

RETURN VALUE
On success, mmap() returns a pointer to the mapped area. On
error, the value MAP_FAILED (that is, (void*) -1) is returned...

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 20 / 40


Parameters

PROT_READ
• Request read permissions to the mapped memory

PROT_WRITE
• Request write permissions to the mapped memory

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 21 / 40


Parameters

MAP_SHARED
• Request a “shared” mapping
• Updates are visible to other processes
• Updates are carried through to the underlying device
• (No other option works for device memory)

unsigned int *p;


p = mmap(0, 0x1000, PROT_READ | PROT_WRITE,
MAP_SHARED, fd, 0xfffff000);

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 22 / 40


Parameters

O_SYNC
• Use in open(1)
• Indicates nocache, which is probably what you want

int fd;
fd = open("/dev/mem", O_RDWR | O_SYNC);

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 23 / 40


Reading and Writing the Hardware

Use normal C-style pointers:


• Dereference to read or write
• Model banks of registers as arrays
• Don’t forget the volatile and const keywords!

int volatile const * volatile p;

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 24 / 40


Reading and Writing the Hardware

Watch out for endianness and alignment!


• Know the alignment restrictions of your host processor
• Know the alignment restrictions of the target device
• Be vigilant for data representation assumptions

• Scrutinize the compiler’s assembly language carefully!

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 25 / 40


mmap-gpio-csb737.c

1 #define GPIO 0xfffff000UL


2 #define GPIOA 0x200UL
3 #define GPIOD 0x800UL

1 #define PIO_PER(p) ((volatile unsigned int*)((unsigned long)(p) + 0))


2 #define PIO_PDR(p) ((volatile unsigned int*)((unsigned long)(p) + 4))
3 #define PIO_PSR(p) ((volatile unsigned int*)((unsigned long)(p) + 8))
4 #define PIO_OER(p) ((volatile unsigned int*)((unsigned long)(p) + 0x10))
5 #define PIO_ODR(p) ((volatile unsigned int*)((unsigned long)(p) + 0x14))
6 #define PIO_OSR(p) ((volatile unsigned int*)((unsigned long)(p) + 0x18))
7 #define PIO_SODR(p) ((volatile unsigned int*)((unsigned long)(p) + 0x30))
8 #define PIO_CODR(p) ((volatile unsigned int*)((unsigned long)(p) + 0x34))
9 #define PIO_PDSR(p) ((volatile unsigned int*)((unsigned long)(p) + 0x3c))

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 26 / 40


mmap-gpio-csb737.c

1 int main (int argc, char *argv[])


2 {
3 /* mknod /dev/mem c 1 1 */
4 /* crw-r--r-- 1 root root 1, 1 2006-02-27 10:49 /dev/mem */
5 int fd = open("/dev/mem", O_RDWR | O_SYNC);
6
7 if (!fd)
8 {
9 perror("fd");
10 return -1;
11 }
12
13 unsigned long gpio
14 = (unsigned long)mmap(0, 0x1000, PROT_READ | PROT_WRITE,
15 MAP_SHARED, fd, GPIO);

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 27 / 40


mmap-gpio-csb737.c

1 unsigned long gpiod = gpio + GPIOD;


2 printf("gpiod mapped to %lx\n", gpiod);
3
4 volatile unsigned int* perd = PIO_PER(gpiod);
5 volatile unsigned int* psrd = PIO_PSR(gpiod);
6 volatile unsigned int* oerd = PIO_OER(gpiod);
7 volatile unsigned int* osrd = PIO_OSR(gpiod);
8 volatile unsigned int* sodrd = PIO_SODR(gpiod);
9 volatile unsigned int* codrd = PIO_CODR(gpiod);
10 volatile unsigned int* pdsrd = PIO_PDSR(gpiod);

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 28 / 40


mmap-gpio-csb737.c

1 #define LED (1 << 4) /* PD4 */


2
3 *perd = LED;
4 *oerd = LED;
5 printf("psrd: %x osrd: %x\n", *psrd, *osrd);
6
7 *codrd = LED;

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 29 / 40


mmap-gpio-csb737.c

1 munmap((void*)gpio, 0x1000);
2 close(fd);
3
4 return 0;

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 30 / 40


mmap-gpio-csb737.c

# gcc -g -Wall -o csb737 mmap-gpio-csb737.c

# ./csb737
psrd: 1ea503b7 osrd: 1ea503b7
...

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 31 / 40


Caveat

The previous example is a Bad Idea:


• Other drivers are using GPIO
• We risk concurrent access issues

Well, not entirely true:


• The hardware naturally prevents some problems
• (Important details are left as an exercise)

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 32 / 40


Recap

Protected memory:
• Separates kernel and user memory spaces
• Prevents direct access to hardware
• “User mode” vs. “kernel mode”

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 33 / 40


Recap

The mmap(2) system call:


• Returns a pointer to physical memory
• Must be a privileged user process
• Use the volatile keyword

Code example:
• CSB737 GPIO controller

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 34 / 40


Recap

Advantages:
• “Device drivers” are ordinary applications
• Probably easier to develop and debug
• Potentially better application integration

Disadvantages:
• Modest performance hit

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 35 / 40


Linux Hardware I/O via Mmap(2)
Introduction to User-Space Hardware I/O

Bill Gatliff
bgat@billgatliff.com

Freelance Embedded Systems Developer


Demonstration

CSB737 LED:
• Blink from a user application
• (Using mmap(2), not gpiolib)
• See mmap-gpio-csb737.c for code

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 37 / 40


Assignment

Questions:
• How does the AT91SAM9263 GPIO controller work?
• What features minimize the risks of concurrent access?

Questions:
• How do you configure pins as inputs, outputs?
• How do you prevent glitches during configuration?

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 38 / 40


Assignment

Repeat the demonstration:


• Build, run mmap-gpio-csb737.c
• Verify that it works as expected
• Review, understand the code

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 39 / 40


Assignment

Read the pushbutton:


• Use it to vary the blink rate of the LED

Copyright © Bill Gatliff, 2010 Linux Hardware I/O via Mmap(2) 40 / 40


Linux Kernel Modules
An Overview for Embedded Systems

Bill Gatliff
bgat@billgatliff.com

Freelance Embedded Systems Developer

Copyright © Bill Gatliff, 2010 Kernel Modules 1 / 37


Overview

Roadmap:
• What is a “kernel module”?
• Kernel modules are not device drivers!
• Entry and exit codes
• Error handling
• Building and testing

Copyright © Bill Gatliff, 2010 Kernel Modules 2 / 37


What is a “kernel module”?

Linux can load object files into a running kernel:


• These object files are called “kernel modules”

Yes, ordinary object files:


• Code, data
• ‘Exportable” symbols
• Unresolved references to external symbols

Copyright © Bill Gatliff, 2010 Kernel Modules 3 / 37


What is a “kernel module”?

Once loaded, a kernel module becomes kernel code:


• Loaded into kernel memory space
• Can access kernel services
• Can implement new kernel services
• Can offer interfaces for user applications

Copyright © Bill Gatliff, 2010 Kernel Modules 4 / 37


What is a “kernel module”?

And, like all kernel code:


• Cannot use an FPU, if present
• Cannot touch user memory directly

Copyright © Bill Gatliff, 2010 Kernel Modules 5 / 37


What is a “kernel module”?

“Kernel modules are device drivers, right?”


• No!

Device drivers are code:


• ... and a kernel module might contain such code
• ... but might not
• (Linux device drivers aren’t what you think they are)

Copyright © Bill Gatliff, 2010 Kernel Modules 6 / 37


What is a “kernel module”?

It’s more accurate to say:


• Device drivers are often implemented in kernel modules
• A kernel module might contain a device driver, but might not

Copyright © Bill Gatliff, 2010 Kernel Modules 7 / 37


Minimal Example

1 #include <linux/module.h>
2
3 #define MODULE_NAME "skeleton"
4
5 int __init example_init (void)
6 {
7 printk(KERN_ERR "%s: %s()\n", MODULE_NAME, __FUNCTION__);
8 return 0;
9 }
10
11 void __exit example_exit (void)
12 {
13 printk(KERN_ERR "%s: %s()\n", MODULE_NAME, __FUNCTION__);
14 }
15
16 module_init(example_init);
17 module_exit(example_exit);

Copyright © Bill Gatliff, 2010 Kernel Modules 8 / 37


Entry and Exit Functions

module_init()
• Macro that refers to module entry code
• Invoked when module is loaded
• Module “expunged” on nonzero return value
• Limit one per module

Copyright © Bill Gatliff, 2010 Kernel Modules 9 / 37


Entry and Exit Functions

module_exit()
• Macro that refers to module exit code
• Invoked when module is unloaded
• No return value
• Module “expunged” from memory at exit
• Limit one per module

Copyright © Bill Gatliff, 2010 Kernel Modules 10 / 37


Entry and Exit Functions

Things to do on entry:
• Allocate module-global memory
• Register device drivers
• Register interfaces
• ...

On exit:
• Undo all the above!

Copyright © Bill Gatliff, 2010 Kernel Modules 11 / 37


The __init and __exit Macros

__init
• Code gets placed into the init.text section
• After module initialization, init.text memory is discarded
• After initialization, the code no longer exists!

Freeing unused kernel memory: 68k freed

Copyright © Bill Gatliff, 2010 Kernel Modules 12 / 37


The __init and __exit Macros

__exit
• Marks code needed only during exit
• Kernel knows it can set this code aside until later

Copyright © Bill Gatliff, 2010 Kernel Modules 13 / 37


Handling Errors

int __init my_init_function(void)


{
int err;
/* registration takes a pointer and a name */
err = register_this(ptr1, "skull");
if (err) goto fail_this;
err = register_that(ptr2, "skull");
if (err) goto fail_that;
err = register_those(ptr3, "skull");
if (err) goto fail_those;
return 0; /* success */
fail_those: unregister_that(ptr2, "skull");
fail_that: unregister_this(ptr1, "skull");
fail_this: return err; /* propagate the error */
}

Source: Chapter 2, Linux Device Drivers, 3rd. ed.

Copyright © Bill Gatliff, 2010 Kernel Modules 14 / 37


Handling Errors

The general idea:


• Break initialization down into atomic steps
• Provide labels for error-recovery code
• Stack up recovery steps in reverse order

On an error:
• Enter at the right place via goto
• Fall through recovery steps to completion

Copyright © Bill Gatliff, 2010 Kernel Modules 15 / 37


Handling Errors

Initialization:
int __init my_init_function(void)
{
int err;

/* step 1 */
err = init_step_1(...);
if (err)
goto step_1_failed;

/* step 2 */
err = init_step_2(...);
if (err)
goto step_2_failed;

...

Copyright © Bill Gatliff, 2010 Kernel Modules 16 / 37


Handling Errors

... and the recovery:


...

/* success! */
return 0;

step_3_failed:
/* undo step 2 */
...;

step_2_failed:
/* undo step 1 */
...;

step_1_failed:
/* (nothing to do) */

return err;
}

Copyright © Bill Gatliff, 2010 Kernel Modules 17 / 37


Handling Errors

int __init my_init_function(void)


{
struct a *p_a;
struct b *p_b;

/* step 1 */
p_a = kmalloc(sizeof(struct a), GFP_KERNEL);
if (!p_a)
goto a_alloc_failed;

/* step 2 */
p_b = kmalloc(sizeof(struct b), GFP_KERNEL);
if (!p_b)
goto b_alloc_failed;

...

Copyright © Bill Gatliff, 2010 Kernel Modules 18 / 37


Handling Errors

...

/* success! */
return 0;

b_alloc_failed:
kfree(p_a);

a_alloc_failed:
/* (nothing to kfree() */

return err;
}

Copyright © Bill Gatliff, 2010 Kernel Modules 19 / 37


Handling Errors

Each step must be atomic:


• It must completely succeed, or completely fail
• Subdivide into smaller steps as necessary
• Refactor code into subroutines

Copyright © Bill Gatliff, 2010 Kernel Modules 20 / 37


Module Information Macros

MODULE_LICENSE
• Communicates the module’s distribution license to the kernel
• Proper use prevents “tainting” the kernel
• Some kernel symbols are only exported to GPL-licensed code

MODULE_LICENSE(‘‘GPL’’);
MODULE_LICENSE(‘‘Copyright (c) 2007...’’);

Copyright © Bill Gatliff, 2010 Kernel Modules 21 / 37


Module Information Macros

Recognized license descriptions:


“GPL”

“GPL v2”
“GPL and additional rights”

“Dual BSD/GPL”
“Proprietary”

Copyright © Bill Gatliff, 2010 Kernel Modules 22 / 37


The tainted flag

“Tainting the kernel?!”


• Proprietary modules, when loaded, set the tainted flag
• Flag shows up in OOPS messages and elsewhere
• Requires reboot to reset
• /proc/sys/kernel/tainted

Copyright © Bill Gatliff, 2010 Kernel Modules 23 / 37


The tainted flag

Unable to handle kernel NULL pointer dereference at virtual address 0


pgd = c0004000
[00000000] *pgd=00000000
Internal error: Oops: 8f5
Modules linked in:
CPU: 0
PC is at do_initcalls+0x28/0xe0
LR is at init+0x34/0xf0
pc : [<c0008a0c>] lr : [<c002409c>] Not tainted
sp : c037ffc4 ip : c037ffe4 fp : c037ffe0
r10: 00000000 r9 : 00000000 r8 : 00000000
r7 : c001e8c4 r6 : 00000000 r5 : c037e000 r4 : c001e684
r3 : 00000000 r2 : c036dee4 r1 : 00000000 r0 : c036a4e0
Flags: Nzcv IRQs on FIQs on Mode SVC\_32 Segment kernel
Control: 397F Table: A0004000 DAC: 00000017
Process swapper (pid: 1, stack limit = 0xc037e194)
...

Copyright © Bill Gatliff, 2010 Kernel Modules 24 / 37


Module Information Macros

Introduced in 2.4.10 by Alan Cox:


“I get so many bug reports caused by the nvidia
modules...”
[http://lwn.net/2001/0906/a/ac-license.php3]

• Some developers ignore posts of tainted OOPS messages

Copyright © Bill Gatliff, 2010 Kernel Modules 25 / 37


Module Information Macros

“So why don’t I just take that flag out?”


• You could, but the kernel would still be “tainted”

“... since its arguably digital rights management


[you] might face five years in jail in the USA for
doing so 8)”
[http://lwn.net/2001/0906/a/ac-tainted.php3]

Copyright © Bill Gatliff, 2010 Kernel Modules 26 / 37


Module Information Macros

MODULE_AUTHOR
• Identifies the module’s author

MODULE_AUTHOR(‘‘Bill Gatliff <bgat@billgatliff.com>’’);

Copyright © Bill Gatliff, 2010 Kernel Modules 27 / 37


Module Information Macros

MODULE_DESCRIPTION
• Describes what the module is or does

MODULE_DESCRIPTION(‘‘TSC2003 touch screen driver...’’);

Copyright © Bill Gatliff, 2010 Kernel Modules 28 / 37


Module Information Macros

MODULE_VERSION
• Module version information

MODULE_VERSION(‘‘1.23-rc4’’);

Copyright © Bill Gatliff, 2010 Kernel Modules 29 / 37


skeleton_module.c

1 #include <linux/module.h>
2
3 #define MODULE_NAME "skeleton"
4
5 int __init example_init (void)
6 {
7 printk(KERN_ERR "%s: %s()\n", MODULE_NAME, __FUNCTION__);
8 return 0;
9 }
10
11 void __exit example_exit (void)
12 {
13 printk(KERN_ERR "%s: %s()\n", MODULE_NAME, __FUNCTION__);
14 }
15
16 module_init(example_init);
17 module_exit(example_exit);
18
19 MODULE_LICENSE("GPL");
20 MODULE_VERSION("1.2-rc3");
21 MODULE_AUTHOR("Bill Gatliff <bgat@billgatliff.com>");
22 MODULE_DESCRIPTION("A do-nothing example");

Copyright © Bill Gatliff, 2010 Kernel Modules 30 / 37


Building and Loading a Module

The kernel’s build machinery is called “Kbuild”


• Makefiles, configurators, configuration files
• Distributed with the Linux kernel’s source code
• Also used by other projects, e.g. Busybox

$ make ARCH=arm menuconfig

Copyright © Bill Gatliff, 2010 Kernel Modules 31 / 37


Building and Loading a Module

Even proprietary modules need Kbuild:


• Selects the right compiler, flags, command scripts
• Matches up kernel, module version information

... but proprietary modules:


• Cannot mix with GPL kernel code
• Must maintain related kernel sources separately

Copyright © Bill Gatliff, 2010 Kernel Modules 32 / 37


Building and Loading a Module

Create a Makefile in your working directory:

1 obj-m += example_module.o
2
3 all:
4 make -C $(KERNELSRC) M=$(PWD) modules
5
6 clean:
7 make -C $(KERNELSRC) M=$(PWD) clean

Copyright © Bill Gatliff, 2010 Kernel Modules 33 / 37


Building and Loading a Module

Build!

$ export KERNELSRC=.../path/to/kernel/source
$ make

$ modinfo example_module.ko
filename: example_module.ko
license: Copyright (c) 2007...
author: Bill Gatliff <bgat@billgatliff.com>

$ make clean

Copyright © Bill Gatliff, 2010 Kernel Modules 34 / 37


Building and Loading a Module

Test!

$ insmod example_module.ko
$ dmesg
...
example_module: example_init called
$ rmmod example_module
$ dmesg
...
example_module: example_init called
example_module: example_exit called

Copyright © Bill Gatliff, 2010 Kernel Modules 35 / 37


Building and Loading a Module

Actually:
• ... including cross-specific stuff
• ... omitting redundancy in template Makefile
• (Yes, you could eliminate the Makefile altogether!)

$ make KERNELRC=/path/to/kernel/source
obj-m=skeleton.o ARCH=arm CROSS_COMPILE=${TARGET}-

Copyright © Bill Gatliff, 2010 Kernel Modules 36 / 37


Linux Kernel Modules
An Overview for Embedded Systems

Bill Gatliff
bgat@billgatliff.com

Freelance Embedded Systems Developer


Decoding Linux OOPS Messages
Techniques for Debugging Linux Systems

Bill Gatliff
bgat@billgatliff.com

Freelance Embedded Systems Developer

Copyright © Bill Gatliff, 2010 OOPS Messages 1 / 28


Overview

OOPS!
• What is an OOPS message?
• Understanding an OOPS
• Reading the message
• Finding the bug

Copyright © Bill Gatliff, 2010 OOPS Messages 2 / 28


Overview

Related technologies:
• printk()
• strace
• objdump

Copyright © Bill Gatliff, 2010 OOPS Messages 3 / 28


OOPS Messages

“I don’t like debuggers. Never have, probably never


will.”
-- Linus Torvalds. Posted on LKML in September, 2000.

Copyright © Bill Gatliff, 2010 OOPS Messages 4 / 28


OOPS Messages

Initiated when the kernel detects an error:


• Usually serious
• Not always fatal
• Frequent (undesirable) initiator is buggy drivers

Copyright © Bill Gatliff, 2010 OOPS Messages 5 / 28


OOPS Messages

Unable to handle kernel NULL pointer dereference at virtual address 00000000


pgd = c0004000
[00000000] *pgd=00000000
Internal error: Oops: 8f5 [#1]
Modules linked in:
CPU: 0
PC is at do_initcalls+0x28/0xe0
LR is at init+0x34/0xf0
pc : [<c0008a0c>] lr : [<c002409c>] Not tainted
sp : c037ffc4 ip : c037ffe4 fp : c037ffe0
r10: 00000000 r9 : 00000000 r8 : 00000000
r7 : c001e8c4 r6 : 00000000 r5 : c037e000 r4 : c001e684
r3 : 00000000 r2 : c036dee4 r1 : 00000000 r0 : c036a4e0
Flags: Nzcv IRQs on FIQs on Mode SVC_32 Segment kernel
Control: 397F Table: A0004000 DAC: 00000017
Process swapper (pid: 1, stack limit = 0xc037e194)
...

Copyright © Bill Gatliff, 2010 OOPS Messages 6 / 28


OOPS Messages

...
Stack: (0xc037ffc4 to 0xc0380000)
ffc0: c026099c 00000000 00000000 00000000 c037fff4 c037ffe4 c002409c
ffe0: c00089f0 00000000 00000000 c037fff8 c003c84c c0024074 e59f012c ebf9883f
Backtrace:
[<c00089e4>] (do_initcalls+0x0/0xe0) from [<c002409c>] (init+0x34/0xf0)
r7 = 00000000 r6 = 00000000 r5 = 00000000 r4 = C026099C
[<c0024068>] (init+0x0/0xf0) from [<c003c84c>] (do_exit+0x0/0x360)
r4 = 00000000
Code: e3c5503f e3a03000 e1540007 e5956004 (e5833000)
<0>Kernel panic - not syncing: Attempted to kill init!

Copyright © Bill Gatliff, 2010 OOPS Messages 7 / 28


Reading an OOPS Message

From this:
Stack: (0xc037ffc4 to 0xc0380000)
ffc0: c026099c 00000000 00000000 00000000
ffe0: c00089f0 00000000 00000000 c037fff8 c003c84c
...
Code: e3c5503f e3a03000 e1540007 e5956004 (e5833000)
<0>Kernel panic - not syncing: Attempted to kill init!

To this:
*(unsigned long*)0 = 0; // TODO: force an OOPS

Copyright © Bill Gatliff, 2010 OOPS Messages 8 / 28


Reading an OOPS Message

What you need:


• vmlinux
• System.map
• objdump
• Kernel source code

Copyright © Bill Gatliff, 2010 OOPS Messages 9 / 28


Reading an OOPS Message

The first line:


• “Why I’m sending this message”
• Read carefully!
• (The format varies by architecture)

Unable to handle kernel NULL pointer


dereference at virtual address 00000000

Copyright © Bill Gatliff, 2010 OOPS Messages 10 / 28


Reading an OOPS Message

Consider this:
Unable to handle kernel NULL pointer
dereference at virtual address 00000000

Against this:
Process swapper (pid: 1, stack limit=0xc037e194)

Copyright © Bill Gatliff, 2010 OOPS Messages 11 / 28


Reading an OOPS Message

Page Global Directory:


• Address of critical MM structures
• Each process has its own PGD
• (If there was a process at all)
• pgd_t

pgd = c0004000
[00000000] *pgd=00000000

Copyright © Bill Gatliff, 2010 OOPS Messages 12 / 28


Reading an OOPS Message

Architecture-specific:
• Detailed exception information

Internal error: Oops: 8f5 [#1]

Modules, if any:
Modules linked in:

Copyright © Bill Gatliff, 2010 OOPS Messages 13 / 28


Reading an OOPS Message

Which CPU:
• (SMP-only)

CPU: 0

Copyright © Bill Gatliff, 2010 OOPS Messages 14 / 28


Reading an OOPS Message

Where were we?


• Program Counter, Link Register
• Details require kernel symbols

PC is at do_initcalls+0x28/0xe0
LR is at init+0x34/0xf0
pc : [<c0008a0c>] lr : [<c002409c>] Not tainted

Copyright © Bill Gatliff, 2010 OOPS Messages 15 / 28


Reading an OOPS Message

Careful:
• The OOPS is the failure effect
• Failure mode might be several instructions back

Especially:
• Hardware problems
• Memory corruption

Copyright © Bill Gatliff, 2010 OOPS Messages 16 / 28


Reading an OOPS Message

Register values:
pc : [<c0008a0c>] lr : [<c002409c>]
sp : c037ffc4 ip : c037ffe4 fp : c037ffe0
r10: 00000000 r9 : 00000000 r8 : 00000000
r7 : c001e8c4 r6 : 00000000 r5 : c037e000
r4 : c001e684 r3 : 00000000 r2 : c036dee4
r1 : 00000000 r0 : c036a4e0

Copyright © Bill Gatliff, 2010 OOPS Messages 17 / 28


Reading an OOPS Message

Status register flags:


• SREG
• Coprocessors

Flags: Nzcv IRQs on FIQs on Mode SVC_32


Control: 397F Table: A0004000 DAC: 00000017

Copyright © Bill Gatliff, 2010 OOPS Messages 18 / 28


Reading an OOPS Message

Stack dump:
Stack: (0xc037ffc4 to 0xc0380000)
ffc0: c026099c 00000000 00000000
ffd0: 00000000 c037fff4 c037ffe4 c002409c
ffe0: c00089f0 00000000 00000000 c037fff8
fff0: c003c84c c0024074 e59f012c ebf9883f

Copyright © Bill Gatliff, 2010 OOPS Messages 19 / 28


Reading an OOPS Message

Stack dump:
...
Backtrace:
[<c00089e4>] (do_initcalls+0x0/0xe0)
from [<c002409c>] (init+0x34/0xf0)
r7 = 00000000 r6 = 00000000 r5 = 00000000
[<c0024068>] (init+0x0/0xf0)
from [<c003c84c>] (do_exit+0x0/0x360)
r4 = 00000000

Copyright © Bill Gatliff, 2010 OOPS Messages 20 / 28


Reading an OOPS Message

Instruction stream:
• Ideal sanity check!

Code: e3c5503f e3a03000 e1540007


e5956004 (e5833000)

Copyright © Bill Gatliff, 2010 OOPS Messages 21 / 28


Reading an OOPS Message

And finally:
<0>Kernel panic - not syncing:
Attempted to kill init!

Copyright © Bill Gatliff, 2010 OOPS Messages 22 / 28


Finding the Offending Instruction

Where were we?


• Program Counter, Link Register

PC is at do_initcalls+0x28/0xe0
LR is at init+0x34/0xf0
pc : [<c0008a0c>] lr : [<c002409c>] Not tainted

Copyright © Bill Gatliff, 2010 OOPS Messages 23 / 28


Finding the Offending Instruction

Use objdump:

$ objdump --disassemble vmlinux | grep -C 16 "c0008a0c:"


...
c00089e4 <do_initcalls>:
c00089e4: e1a0c00d mov ip, sp
c00089e8: e92dd8f0 stmdb sp!, r4, r5, r6, r7, fp, ip, lr, pc
c00089ec: e24cb004 sub fp, ip, #4 ; 0x4
c00089f0: e59f40a8 ldr r4, [pc, #168] ; c0008aa0 <.init+0xaa0>
c00089f4: e59f70a8 ldr r7, [pc, #168] ; c0008aa4 <.init+0xaa4>
c00089f8: e3cd5d7f bic r5, sp, #8128 ; 0x1fc0
c00089fc: e3c5503f bic r5, r5, #63 ; 0x3f
c0008a00: e3a03000 mov r3, #0 ; 0x0
c0008a04: e1540007 cmp r4, r7
c0008a08: e5956004 ldr r6, [r5, #4]
c0008a0c: e5833000 str r3, [r3]
c0008a10: 2a000020 bcs c0008a98 <do_initcalls+0xb4>

Copyright © Bill Gatliff, 2010 OOPS Messages 24 / 28


Finding the Offending Instruction

static void __init do_initcalls(void)


{
initcall_t *call;
int count = preempt_count();
*(unsigned long*)0 = 0; // TODO: force an OOPS
for (call = __initcall_start; call < __initcall_end; call++) {
char *msg;
if (initcall_debug) {
printk(KERN_DEBUG "Calling initcall 0x%p", *call);
...

Copyright © Bill Gatliff, 2010 OOPS Messages 25 / 28


Triggering an OOPS message

void dump_stack(void)
• Generates OOPS output
• Does not halt the system
• See also: panic() and die()

Copyright © Bill Gatliff, 2010 OOPS Messages 26 / 28


Recap

Decoding an OOPS message:


• Understand what it says
• Use objdump to find the code

Scrutinize carefully:
• Understand the message itself
• Recognize why it might be wrong
• “Cascades” of failures

Copyright © Bill Gatliff, 2010 OOPS Messages 27 / 28


Decoding Linux OOPS Messages
Techniques for Debugging Linux Systems

Bill Gatliff
bgat@billgatliff.com

Freelance Embedded Systems Developer


POSIX.1b: Real-Time POSIX.1 Extensions
Scheduling and Prioritization

Bill Gatliff
bgat@billgatliff.com

Freelance Embedded Systems Developer

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 1 / 17


POSIX.1b Scheduling

“The official scheduling of the real world is


preemptive, priority scheduling.”
-- Bill Gallmeister, POSIX.4: Programming for the Real World

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 2 / 17


POSIX.1b Scheduling

Priority scheduling:
• Highest-priority tasks are scheduled first
• Same-priority tasks are predictably scheduled

Preemptive scheduling:
• Higher-priority tasks will interrupt lower-priority ones

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 3 / 17


POSIX.1b Scheduling

NICE(1) User Commands NICE(1)

NAME
nice - run a program with modified scheduling priority

SYNOPSIS
nice [OPTION] [COMMAND [ARG]...]

DESCRIPTION
Run COMMAND with an adjusted niceness, which affects process scheduling.
With no COMMAND, print the current niceness. Nicenesses range from -20
(most favorable scheduling) to 19 (least favorable).

-n, --adjustment=N
add integer N to the niceness (default 10)

NOTE: your shell may have its own version of nice, which usually
supersedes the version described here. Please refer to your shell’s
documentation for details about the options it supports...

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 4 / 17


nice(1)

Controls the “niceness” of a process:


• Adjusts the time-sharing scheduler
• Statistically, might make a program run faster

Ineffective for real-time needs!


• ... because it wasn’t designed for that

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 5 / 17


nice(1)

Need new interfaces to:


• Adjust wholesale scheduling policy
• Adjust scheduling parameters
• Yield the CPU in a predictable way
• Make things happen on time

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 6 / 17


sched_setscheduler()

Adjust scheduling policy:


sched_setscheduler(pid_t pid, int policy,
const struct sched_param *p);

Policies:
• SCHED_FIFO — Preemptive, priority-based
• SCHED_RR — Preemptive, priority-based with quanta
• SCHED_OTHER — Implementation-defined

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 7 / 17


sched_setscheduler()

struct sched_param
• Used by both SCHED_FIFO and SCHED_RR policies
• Implementations may add other fields as needed

struct sched_param {
...
int sched_priority;
...
};

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 8 / 17


sched_setscheduler()

SCHED_FIFO — First-In, First-Out


• Processes at lower priority levels are interrupted
• Same-priority processes are scheduled FIFO
• Highest-priority process must explicitly yield or block

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 9 / 17


sched_setscheduler()

Can you spot the problem?


struct sched_param scheduling_parameters;
scheduling_parameters.sched_priority =
sched_get_priority_max(SCHED_FIFO);

sched_setscheduler(0, SCHED_FIFO,
&scheduling_parameters);
...
while(1) {...}
...

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 10 / 17


sched_setscheduler()

SCHED_RR is just like SCHED_FIFO, except:


• Same-priority processes get only a quanta of time
• Processes go to the end of the priority queue

Notes:
• Quanta is implementation-defined
• Use sched_rr_get_interval(2) to determine quanta

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 11 / 17


sched_setscheduler()

#include <sched.h>

int sched_rr_get_interval(pid_t pid,


struct timespec *tp);

struct timespec {
time_t tv_sec; /* seconds */
long tv_nsec; /* nanoseconds */
};

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 12 / 17


On Child Processes

Scheduling parameters are inherited from the parent:


atprio(const char *file, *const argv[])
{
struct sched_param cmd_sched_params;
cmd_sched_params.sched_priority = atoi(argv[1]);

sched_setscheduler(0, SCHED_FIFO, &cmd_sched_params);


execvp(file, argv);
}

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 13 / 17


On Child Processes

This is tempting:
# atprio 127 rt_data_acquire | rt_data_process |
tee rt_file | rt_data_output
> /dev/output < /dev/input
This is what you want:
# atprio 127 sh -c ‘‘rt_data_acquire |
rt_data_process | tee rt_file |
rt_data_output > /dev/output < /dev/input’’

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 14 / 17


On Child Processes

Processes may set their own parameters:


#include <sched.h>

struct sched_param scheduling_parameters;


int i;

...
scheduling_parameters.sched_priority =
sched_get_priority_max(SCHED_FIFO);
i = sched_setscheduler(getpid(), SCHED_FIFO,
&scheduling_parameters);
...

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 15 / 17


Yields

Yielding to another process:


• Moves the process to the end of the priority queue
• Does not guarantee that other processes are ready to run
• Does not guarantee that there are same-priority processes

void sched_yield(void);

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 16 / 17


POSIX.1b: Real-Time POSIX.1 Extensions
Scheduling and Prioritization

Bill Gatliff
bgat@billgatliff.com

Freelance Embedded Systems Developer


POSIX.1b: Real-Time POSIX.1 Extensions
Timers

Bill Gatliff
bgat@billgatliff.com

Freelance Embedded Systems Developer

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 1 / 27


Basic Unix Timing Facilities

The problem:
• No guarantees for process scheduling
• What about background, periodic tasks?

while (1) {
/* read from the device */
read(fd, buf, nbytes);
/* do something ... */
}

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 2 / 27


Basic Unix Timing Facilities

Seconds since the Epoch:

#include <time.h>
time_t time(time_t *tp);

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 3 / 27


Basic Unix Timing Facilities

Milliseconds since the Epoch:


• (but granularity is often much less than 1 msec!)

#include <sys/time.h>
int gettimeofday(struct timeval *tv);

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 4 / 27


Basic Unix Timing Facilities

One approach to periodicity:


• Poor accuracy
• Subject to drift
• Poor granularity

while(1) {
sleep(sec);
/* do something ... */
}

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 5 / 27


Basic Unix Timing Facilities

An attempt at improvement:
• Isn’t likely to work!

while(1) {
sleep(sec - s);
/* do something ... */
}

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 6 / 27


Basic Unix Timing Facilities

alarm (int secs)


• Sends a SIGALRM when the timer expires
• One-shot timer
• Cancel alarm with alarm(0)
• Poor granularity

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 7 / 27


Basic Unix Timing Facilities

Setting a timer:
• No, you usually don’t get microsecond resolution!
• Still subject to drift, due to poor granularity

struct itimerval interval;


...
interval.it_value.tv_sec = secs;
interval.it_value.tv_usec = usecs;
setitimer(ITIMER_REAL, &interval, NULL);

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 8 / 27


POSIX.1b Timing Facilities

Getting and setting the time of day:


• Note: most hardware doesn’t support nanoseconds!

#include <time.h>
struct timespec {
time_t tv_sec; /* seconds */
long tv_nsec; /* nanoseconds */
}
int clock_gettime(clockid_t clk_id,
struct timespec *tp);
int clock_settime(clockid_t clk_id,
const struct timespec *tp);

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 9 / 27


POSIX.1b Timing Facilities

Getting and setting the time of day:


• Note: most hardware doesn’t support nanoseconds!

#include <time.h>
struct timespec current_time;
clock_gettime(CLOCK_REALTIME, &current_time);
clock_settime(CLOCK_REALTIME, &current_time);

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 10 / 27


POSIX.1b Timing Facilities

Available clocks:
• CLOCK_REALTIME — System-wide, real-time clock
• CLOCK_MONOTONIC — Time since an unspecified point

Not always supported:


• CLOCK_PROCESS_CPUTIME_ID — Per-process timer
• CLOCK_THREAD_CPUTIME_ID — Per-thread timer

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 11 / 27


POSIX.1b Timing Facilities

Determining clock resolution:


• Returns the smallest distinguishable time value

int clock_getres(clockid_t clk_id,


struct timespec *res);

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 12 / 27


POSIX.1b Timing Facilities

Short, imprecise delays:


• Sleeps for requested number of nanoseconds
• Exits early on receiving a signal
• If interrupted, returns time remaining

int nanosleep(const struct timespec *req,


struct timespec *rem);

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 13 / 27


POSIX.1b Timing Facilities

struct timespec sleep_time;


struct timespec leftover_time;
int ret;

ret = nanosleep(&sleep_time, &leftover_time);


if (EINTR == ret) {
/* we were interrupted */
/* ... */
}

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 14 / 27


POSIX.1b Timing Facilities

Interval timers:
• POSIX.1b timers are dynamically created:

struct itimerspec {
struct timespec it_value;
struct timespec it_interval;
};

timer_t created_timer;
timer_create(CLOCK_REALTIME, NULL, &created_timer);

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 15 / 27


POSIX.1b Timing Facilities

One-shot alarm:

struct itimerspec when;

/* two seconds from now */


when.it_value.tv_sec = 2;
when.it_value.tv_nsec = 0;
when.it_interval.tv_sec = 0;
when.it_interval.tv_nsec = 0;

timer_settime(created_timer, 0, &when, NULL);

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 16 / 27


POSIX.1b Timing Facilities

Interval timer:

struct itimerspec when;

/* two seconds from now, every second thereafter */


when.it_value.tv_sec = 2;
when.it_value.tv_nsec = 0;
when.it_interval.tv_sec = 1;
when.it_interval.tv_nsec = 0;

timer_settime(created_timer, 0, &when, NULL);

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 17 / 27


POSIX.1b Timing Facilities

Send a different signal:


• Default is SIGALRM

/* signal handler */
void timer_expired(int signo) {...}

/* bind to SIGUSR1 */
struct sigaction sa;
sa.sa_handler = timer_expired;
sigemptyset(&sa.sa_mask);
sa.sa_flags = 0;
sigaction(SIGUSR1, &sa, NULL);

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 18 / 27


POSIX.1b Timing Facilities

Send a different signal:


• Default is SIGALRM

struct sigevent timer_event;


timer_t created_timer;
struct itimerspec expiration_time;

/* create a timer that sends SIGUSR1 */


timer_event.sigev_signo = SIGUSR1;
timer_create(CLOCK_REALTIME,
&timer_event, &created_timer);

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 19 / 27


POSIX.1b Timing Facilities

Absolute time alarms:


• Use clock_gettime() or mktime() for seconds-since-Epoch

when.it_value.tv_sec = secs_since_epoch;
itimer_settime(created_timer,
TIMER_ABSTIME, &when, NULL);

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 20 / 27


POSIX.1b Timing Facilities

Absolute time alarms:


• Use clock_gettime() or mktime() for seconds-since-Epoch

struct tm absolute;
struct timespec abs_time;
absolute.tm_hour = 23; /* 12:xx pm */
absolute.tm_min = 30; /* xx:30 */
absolute.tm_mon = 8; /* September */
...
abs_time.tv_sec = mktime(&absolute);
abs_time.tv_nsec = 0;
timer_settime(created_timer, 0, &abs_time, NULL);

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 21 / 27


POSIX.1b Timing Facilities

The time left on an interrupted timer:

struct itimerspec time_remaining;


timer_gettime(created_timer, &time_remaining);

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 22 / 27


POSIX.1b Timing Facilities

Timer overrun signals aren’t queued:


• Rather, their occurrence is counted
• Overrun count is reset on each timer_getoverrun()

int n_overruns;
n_overruns = timer_getoverrun(created_timer);

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 23 / 27


POSIX.1b Timing Facilities

Timers can carry additional information:


• But only to POSIX.1b signal information handlers!

void posix1b_timer_handler(int sig,


siginfo_t *extra, void *cruft)
{
int n = extra->si_value.sival_int;
... or ...
void *p = extra->si_value.sival_ptr;
...
}

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 24 / 27


POSIX.1b Timing Facilities

Timers can carry additional information:

struct sigaction sa;


/* hook a POSIX.1b handler to SIGALRM */
sigemptyset(&sa.sa_mask);
sa.sa_flags = SA_SIGINFO;
sa.sa_sigaction = posix1b_timer_handler;
if (sigaction(SIGALRM, &sa, NULL))
{
perror("sigaction");
exit(1);
}

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 25 / 27


POSIX.1b Timing Facilities

Timers can carry additional information:

struct sigevent timer_event;


timer_t created_timer;
time_event.sigev_notify = SIGEV_SIGNAL;
time_event.sgev_signo = SIGALRM;
time_event.sigev_value.sival_int = 0xdeadbeef;
timer_create(CLOCK_REALTIME, &timer_event,
&created_timer);

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 26 / 27


POSIX.1b Timing Facilities

Timers can carry additional information:

union sigval {
int sival_int;
void *sival_ptr;
};

Copyright © Bill Gatliff, 2010 POSIX.1b: Real-Time POSIX.1 Extensions 27 / 27

Vous aimerez peut-être aussi