Académique Documents
Professionnel Documents
Culture Documents
Bill Gatliff
bgat@billgatliff.com
Why do this?
• Provides best control over system configuration
• Not all setups are created equal
• Upgrade on your schedule
• Custom configuration tuned to your environment
• Better insight into system operation
General procedure:
• Obtain all the required source code
• Set up a GNU cross-development toolchain
• Build a Linux kernel
• Build a Linux runtime environment
• Install the kernel, runtime on the target
• Boot!
Target specification:
<arch>-<variant>-<os>-<runtime>
i.e.,
--target=arm-xscale-linux-gnu
--target=arm-unknown-linux-gnu
--target=arm-linux
arm-elf arm-linux
h8300-elf h8300-linux
mips-elf mips-linux
mipsel-elf mipsel-linux
ppc-elf ppc-linux
sh-elf sh-linux
Build specification:
<arch>-<variant>-<os>-<runtime>
i.e.,
--build=i686-pc-linux-gnu
Host specification:
<arch>-<variant>-<os>-<runtime>
i.e.,
--host=i686-pc-linux-gnu
Prefix specification:
• Defines the installation location for build products
• Must be an absolute path
i.e.,
--prefix=/opt/bgat/H-i686-pc-linux-gnu
$ export ROOTDIR_PREFIX=\
/opt/bgat/H-arm-linux-gnu
$ export TARGET=arm-linux
$ .../configure --host=$TARGET \
--prefix=$ROOTDIR_PREFIX
Patch files:
• Bug fixes, added functionality
• Usually associated with bleeding-edge or archaic work
$ gzip -d fixes-something.patch.gz
$ patch -p 1 < fixes-something.patch
--- linux-2.6.13/drivers/net/arm/at91_ether.c.orig
+++ linux-2.6.13/drivers/net/arm/at91_ether.c
@@ -807,10 +807,11 @@
/*
* Initialize the ethernet interface
*/
-int at91ether_setup(int phy_type, struct at91_eth_data *board_data)
+int at91ether_setup(int phy_type, struct device *class_dev)
{
struct net_device *dev;
struct at91_private *lp;
+ struct at91_eth_data *board_data = class_dev->platform_data;
AT91PS_EMAC regs;
unsigned int val;
@@ -843,7 +844,7 @@
lp->board_data = *board_data;
spin_lock_init(&lp->lock);
-
+
ether_setup(dev);
Native toolchain:
Target memory:
• Target filesystem occupies 2MB and up
• Linux kernel needs 1MB-4MB
Patience!
http://gnu.org
• binutils-2.16.1.tar.bz2
• gcc-4.1.1.tar.bz2
• gdb-6.6.tar.bz2
http://uclibc.org
• uclibc-0.9.29.tar.bz2
http://busybox.net
• busybox-1.7.0.tar.bz2
$ export TARGET=arm-linux
$ export PREFIX=/opt/bgat/H-i686-pc-linux-gnu
$ export PATH=${PREFIX}/bin:${PATH}
$ export ROOTDIR_PREFIX=/opt/bgat/H-arm-linux-uclibc
Provides:
• Assembler and linker
• Object and file management utilities
$ ../binutils-x.y.z/configure \
--target=$TARGET --prefix=$PREFIX \
--disable-nls 2>&1 | tee configure-log
Provides:
• Pure cross compiler
Provides:
• Pure cross compiler
$ ../gcc-x.y.z/configure \
--target=$TARGET --prefix=$PREFIX \
--with-gnu-as --with-gnu-ld \
--with-local-prefix=$PREFIX/$TARGET \
--disable-multilib --disable-nls \
--enable-threads=no \
--enable-symvers=gnu --enable-__cxa_atexit \
--disable-shared --enable-languages=c \
2>&1 | tee configure-log
$ make all-gcc install-gcc 2>&1 | tee make-log
Provides:
• Linux-specific header files
Provides:
• C runtime library
$ make menuconfig
Provides:
• C runtime library
$ make menuconfig
...
$ make
Provides:
• C/C++ compiler
Provides:
• Cross debugger
• Instruction set simulator
$ ../gdb-x.y.z/configure \
--target=$TARGET --prefix=$PREFIX \
2>&1 | tee configure-log
Provides:
• Test of the toolchain and runtime environment
• Remote debugging demonstration
$ file hello
hello: ELF 32-bit LSB executable, ARM, version 1 (ARM),
statically linked, not stripped
Provides:
• Linux kernel
• Note: kernels < 2.6.14 might not build with gcc-4
$ cd linux-x.y.z
$ make ARCH=arm CROSS_COMPILE=${TARGET}- clean zImage
$ cp arch/arm/boot/zImage /tftpboot
Provides:
• Location to install root filesystem components
$ mkdir -p $ROOTDIR_PREFIX
$ mkdir -p $ROOTDIR_PREFIX/dev
Provides:
• Means for applications to connect to devices
• Essential for user applications
Provides:
• Means for applications to connect to devices
• Essential for user applications
Romfs:
$ touch $ROOTDIR_PREFIX/dev/@tty,c,5,0
$ touch $ROOTDIR_PREFIX/dev/@console,c,5,1
$ touch $ROOTDIR_PREFIX/dev/@ttyS0,c,4,64
$ touch $ROOTDIR_PREFIX/dev/@null,c,1,3
Provides:
• Basic command shell and utilities
$ make menuconfig
Provides:
• Basic command shell and utilities
$ make menuconfig
...
$ make CROSS_COMPILE=${TARGET}-
$ make CROSS_COMPILE=${TARGET}- \
CONFIG_PREFIX=${ROOTDIR_PREFIX} install
Provides:
• Dynamic linker
• Shared libraries
$ cd uClibc-x.y.z
...
$ make PREFIX=${ROOTDIR_PREFIX} install_runtime
Provides:
• A source of Linux applications
$ genromfs -d $ROOTDIR_PREFIX \
-f /tftpboot/arm-linux-uclibc.img
Alternative:
• Export the filesystem via NFS
Provides:
• User applications, runtime libraries
• Boot scripts, etc.
uMON> tfs ls
Name Size Location Flags Info
monrc 163 0x000412dc e envsetup
romfs.img 812032 0x000413dc
startlinux 4633 0x0004005c e
Provides:
• Linux!
uMON> tfs ls
Name Size Location Flags Info
monrc 163 0x000412dc e envsetup
romfs.img 812032 0x000f183c
startlinux 4633 0x0004005c e
zImage 1530692 0x001b7c9c
Provides:
• Pure joy
...
Built 1 zonelists
Kernel command line: console=ttyS0,38400
ip=192.168.4.20:192.168.4.254:192.168.4.254:255.255.255.0::eth0:off
mtdparts=flash00:812032@989244(root)ro video=pxafb:mode:640x480-16
PID hash table entries: 512 (order: 9, 8192 bytes)
...
IP-Config: Complete:
device=eth0, addr=192.168.4.20, mask=255.255.255.0,
gw=192.168.4.254, host=192.168.4.20, domain=, nis-domain=(none),
bootserver=192.168.4.254, rootserver=192.168.4.254, rootpath=
...
VFS: Mounted root (romfs filesystem) readonly.
Freeing init memory: 112K
/ #
• http://billgatliff.com
Copyright © Bill Gatliff, 2010 Constructing an Embedded Linux System 58 / 59
Constructing an Embedded Linux System
Embedded Linux from Scratch
Bill Gatliff
bgat@billgatliff.com
Bill Gatliff
bgat@billgatliff.com
Roadmap:
• Quick overview of protected memory
• “User mode” vs. “kernel mode”
• The const and volatile keywords
• The mmap(2) system call
• Security implications
• Code examples
An interface:
• Bridges the gap between user and kernel memory
• Under Linux, implementation relies on a device node
• (We’ll come back to this in detail later)
Protected memory :
• The MMU keeps memory spaces separated
• A exception occurs if you go out-of-bounds
Performance:
• Depends on specifics of implementation
• May be better or worse than kernel code
#include<linux/uio_driver.h>
• Sysfs API for device-related information
• Builds on the memmap() approach
• Ideal for pluggable devices, esp. PCI
• Relatively new addition to the kernel (2.6.18-ish)
To control hardware:
• Call open(“/dev/mem”, ...);
• Use mmap(2) to map the device’s control registers
• Use pointer dereferencing to drive the device as always
if (!fd)
{
perror("fd");
return -1;
}
*perd = LED;
*oerd = LED;
printf("psrd: %x osrd: %x\n", *psrd, *osrd);
*codrd = LED;
*sodrd = LED;
NAME
mmap, munmap - map or unmap files or devices into memory
SYNOPSIS
#include <sys/mman.h>
DESCRIPTION
The mmap() function creates a new mapping in the virtual address
space of the calling process. The starting address for the new
mapping is specified in addr. The length argument specifies the
length of the mapping...
RETURN VALUE
On success, mmap() returns a pointer to the mapped area. On
error, the value MAP_FAILED (that is, (void*) -1) is returned...
PROT_READ
• Request read permissions to the mapped memory
PROT_WRITE
• Request write permissions to the mapped memory
MAP_SHARED
• Request a “shared” mapping
• Updates are visible to other processes
• Updates are carried through to the underlying device
• (No other option works for device memory)
O_SYNC
• Use in open(1)
• Indicates nocache, which is probably what you want
int fd;
fd = open("/dev/mem", O_RDWR | O_SYNC);
1 munmap((void*)gpio, 0x1000);
2 close(fd);
3
4 return 0;
# ./csb737
psrd: 1ea503b7 osrd: 1ea503b7
...
Protected memory:
• Separates kernel and user memory spaces
• Prevents direct access to hardware
• “User mode” vs. “kernel mode”
Code example:
• CSB737 GPIO controller
Advantages:
• “Device drivers” are ordinary applications
• Probably easier to develop and debug
• Potentially better application integration
Disadvantages:
• Modest performance hit
Bill Gatliff
bgat@billgatliff.com
CSB737 LED:
• Blink from a user application
• (Using mmap(2), not gpiolib)
• See mmap-gpio-csb737.c for code
Questions:
• How does the AT91SAM9263 GPIO controller work?
• What features minimize the risks of concurrent access?
Questions:
• How do you configure pins as inputs, outputs?
• How do you prevent glitches during configuration?
Bill Gatliff
bgat@billgatliff.com
Roadmap:
• What is a “kernel module”?
• Kernel modules are not device drivers!
• Entry and exit codes
• Error handling
• Building and testing
1 #include <linux/module.h>
2
3 #define MODULE_NAME "skeleton"
4
5 int __init example_init (void)
6 {
7 printk(KERN_ERR "%s: %s()\n", MODULE_NAME, __FUNCTION__);
8 return 0;
9 }
10
11 void __exit example_exit (void)
12 {
13 printk(KERN_ERR "%s: %s()\n", MODULE_NAME, __FUNCTION__);
14 }
15
16 module_init(example_init);
17 module_exit(example_exit);
module_init()
• Macro that refers to module entry code
• Invoked when module is loaded
• Module “expunged” on nonzero return value
• Limit one per module
module_exit()
• Macro that refers to module exit code
• Invoked when module is unloaded
• No return value
• Module “expunged” from memory at exit
• Limit one per module
Things to do on entry:
• Allocate module-global memory
• Register device drivers
• Register interfaces
• ...
On exit:
• Undo all the above!
__init
• Code gets placed into the init.text section
• After module initialization, init.text memory is discarded
• After initialization, the code no longer exists!
__exit
• Marks code needed only during exit
• Kernel knows it can set this code aside until later
On an error:
• Enter at the right place via goto
• Fall through recovery steps to completion
Initialization:
int __init my_init_function(void)
{
int err;
/* step 1 */
err = init_step_1(...);
if (err)
goto step_1_failed;
/* step 2 */
err = init_step_2(...);
if (err)
goto step_2_failed;
...
/* success! */
return 0;
step_3_failed:
/* undo step 2 */
...;
step_2_failed:
/* undo step 1 */
...;
step_1_failed:
/* (nothing to do) */
return err;
}
/* step 1 */
p_a = kmalloc(sizeof(struct a), GFP_KERNEL);
if (!p_a)
goto a_alloc_failed;
/* step 2 */
p_b = kmalloc(sizeof(struct b), GFP_KERNEL);
if (!p_b)
goto b_alloc_failed;
...
...
/* success! */
return 0;
b_alloc_failed:
kfree(p_a);
a_alloc_failed:
/* (nothing to kfree() */
return err;
}
MODULE_LICENSE
• Communicates the module’s distribution license to the kernel
• Proper use prevents “tainting” the kernel
• Some kernel symbols are only exported to GPL-licensed code
MODULE_LICENSE(‘‘GPL’’);
MODULE_LICENSE(‘‘Copyright (c) 2007...’’);
“GPL v2”
“GPL and additional rights”
“Dual BSD/GPL”
“Proprietary”
MODULE_AUTHOR
• Identifies the module’s author
MODULE_DESCRIPTION
• Describes what the module is or does
MODULE_VERSION
• Module version information
MODULE_VERSION(‘‘1.23-rc4’’);
1 #include <linux/module.h>
2
3 #define MODULE_NAME "skeleton"
4
5 int __init example_init (void)
6 {
7 printk(KERN_ERR "%s: %s()\n", MODULE_NAME, __FUNCTION__);
8 return 0;
9 }
10
11 void __exit example_exit (void)
12 {
13 printk(KERN_ERR "%s: %s()\n", MODULE_NAME, __FUNCTION__);
14 }
15
16 module_init(example_init);
17 module_exit(example_exit);
18
19 MODULE_LICENSE("GPL");
20 MODULE_VERSION("1.2-rc3");
21 MODULE_AUTHOR("Bill Gatliff <bgat@billgatliff.com>");
22 MODULE_DESCRIPTION("A do-nothing example");
1 obj-m += example_module.o
2
3 all:
4 make -C $(KERNELSRC) M=$(PWD) modules
5
6 clean:
7 make -C $(KERNELSRC) M=$(PWD) clean
Build!
$ export KERNELSRC=.../path/to/kernel/source
$ make
$ modinfo example_module.ko
filename: example_module.ko
license: Copyright (c) 2007...
author: Bill Gatliff <bgat@billgatliff.com>
$ make clean
Test!
$ insmod example_module.ko
$ dmesg
...
example_module: example_init called
$ rmmod example_module
$ dmesg
...
example_module: example_init called
example_module: example_exit called
Actually:
• ... including cross-specific stuff
• ... omitting redundancy in template Makefile
• (Yes, you could eliminate the Makefile altogether!)
$ make KERNELRC=/path/to/kernel/source
obj-m=skeleton.o ARCH=arm CROSS_COMPILE=${TARGET}-
Bill Gatliff
bgat@billgatliff.com
Bill Gatliff
bgat@billgatliff.com
OOPS!
• What is an OOPS message?
• Understanding an OOPS
• Reading the message
• Finding the bug
Related technologies:
• printk()
• strace
• objdump
...
Stack: (0xc037ffc4 to 0xc0380000)
ffc0: c026099c 00000000 00000000 00000000 c037fff4 c037ffe4 c002409c
ffe0: c00089f0 00000000 00000000 c037fff8 c003c84c c0024074 e59f012c ebf9883f
Backtrace:
[<c00089e4>] (do_initcalls+0x0/0xe0) from [<c002409c>] (init+0x34/0xf0)
r7 = 00000000 r6 = 00000000 r5 = 00000000 r4 = C026099C
[<c0024068>] (init+0x0/0xf0) from [<c003c84c>] (do_exit+0x0/0x360)
r4 = 00000000
Code: e3c5503f e3a03000 e1540007 e5956004 (e5833000)
<0>Kernel panic - not syncing: Attempted to kill init!
From this:
Stack: (0xc037ffc4 to 0xc0380000)
ffc0: c026099c 00000000 00000000 00000000
ffe0: c00089f0 00000000 00000000 c037fff8 c003c84c
...
Code: e3c5503f e3a03000 e1540007 e5956004 (e5833000)
<0>Kernel panic - not syncing: Attempted to kill init!
To this:
*(unsigned long*)0 = 0; // TODO: force an OOPS
Consider this:
Unable to handle kernel NULL pointer
dereference at virtual address 00000000
Against this:
Process swapper (pid: 1, stack limit=0xc037e194)
pgd = c0004000
[00000000] *pgd=00000000
Architecture-specific:
• Detailed exception information
Modules, if any:
Modules linked in:
Which CPU:
• (SMP-only)
CPU: 0
PC is at do_initcalls+0x28/0xe0
LR is at init+0x34/0xf0
pc : [<c0008a0c>] lr : [<c002409c>] Not tainted
Careful:
• The OOPS is the failure effect
• Failure mode might be several instructions back
Especially:
• Hardware problems
• Memory corruption
Register values:
pc : [<c0008a0c>] lr : [<c002409c>]
sp : c037ffc4 ip : c037ffe4 fp : c037ffe0
r10: 00000000 r9 : 00000000 r8 : 00000000
r7 : c001e8c4 r6 : 00000000 r5 : c037e000
r4 : c001e684 r3 : 00000000 r2 : c036dee4
r1 : 00000000 r0 : c036a4e0
Stack dump:
Stack: (0xc037ffc4 to 0xc0380000)
ffc0: c026099c 00000000 00000000
ffd0: 00000000 c037fff4 c037ffe4 c002409c
ffe0: c00089f0 00000000 00000000 c037fff8
fff0: c003c84c c0024074 e59f012c ebf9883f
Stack dump:
...
Backtrace:
[<c00089e4>] (do_initcalls+0x0/0xe0)
from [<c002409c>] (init+0x34/0xf0)
r7 = 00000000 r6 = 00000000 r5 = 00000000
[<c0024068>] (init+0x0/0xf0)
from [<c003c84c>] (do_exit+0x0/0x360)
r4 = 00000000
Instruction stream:
• Ideal sanity check!
And finally:
<0>Kernel panic - not syncing:
Attempted to kill init!
PC is at do_initcalls+0x28/0xe0
LR is at init+0x34/0xf0
pc : [<c0008a0c>] lr : [<c002409c>] Not tainted
Use objdump:
void dump_stack(void)
• Generates OOPS output
• Does not halt the system
• See also: panic() and die()
Scrutinize carefully:
• Understand the message itself
• Recognize why it might be wrong
• “Cascades” of failures
Bill Gatliff
bgat@billgatliff.com
Bill Gatliff
bgat@billgatliff.com
Priority scheduling:
• Highest-priority tasks are scheduled first
• Same-priority tasks are predictably scheduled
Preemptive scheduling:
• Higher-priority tasks will interrupt lower-priority ones
NAME
nice - run a program with modified scheduling priority
SYNOPSIS
nice [OPTION] [COMMAND [ARG]...]
DESCRIPTION
Run COMMAND with an adjusted niceness, which affects process scheduling.
With no COMMAND, print the current niceness. Nicenesses range from -20
(most favorable scheduling) to 19 (least favorable).
-n, --adjustment=N
add integer N to the niceness (default 10)
NOTE: your shell may have its own version of nice, which usually
supersedes the version described here. Please refer to your shell’s
documentation for details about the options it supports...
Policies:
• SCHED_FIFO — Preemptive, priority-based
• SCHED_RR — Preemptive, priority-based with quanta
• SCHED_OTHER — Implementation-defined
struct sched_param
• Used by both SCHED_FIFO and SCHED_RR policies
• Implementations may add other fields as needed
struct sched_param {
...
int sched_priority;
...
};
sched_setscheduler(0, SCHED_FIFO,
&scheduling_parameters);
...
while(1) {...}
...
Notes:
• Quanta is implementation-defined
• Use sched_rr_get_interval(2) to determine quanta
#include <sched.h>
struct timespec {
time_t tv_sec; /* seconds */
long tv_nsec; /* nanoseconds */
};
This is tempting:
# atprio 127 rt_data_acquire | rt_data_process |
tee rt_file | rt_data_output
> /dev/output < /dev/input
This is what you want:
# atprio 127 sh -c ‘‘rt_data_acquire |
rt_data_process | tee rt_file |
rt_data_output > /dev/output < /dev/input’’
...
scheduling_parameters.sched_priority =
sched_get_priority_max(SCHED_FIFO);
i = sched_setscheduler(getpid(), SCHED_FIFO,
&scheduling_parameters);
...
void sched_yield(void);
Bill Gatliff
bgat@billgatliff.com
Bill Gatliff
bgat@billgatliff.com
The problem:
• No guarantees for process scheduling
• What about background, periodic tasks?
while (1) {
/* read from the device */
read(fd, buf, nbytes);
/* do something ... */
}
#include <time.h>
time_t time(time_t *tp);
#include <sys/time.h>
int gettimeofday(struct timeval *tv);
while(1) {
sleep(sec);
/* do something ... */
}
An attempt at improvement:
• Isn’t likely to work!
while(1) {
sleep(sec - s);
/* do something ... */
}
Setting a timer:
• No, you usually don’t get microsecond resolution!
• Still subject to drift, due to poor granularity
#include <time.h>
struct timespec {
time_t tv_sec; /* seconds */
long tv_nsec; /* nanoseconds */
}
int clock_gettime(clockid_t clk_id,
struct timespec *tp);
int clock_settime(clockid_t clk_id,
const struct timespec *tp);
#include <time.h>
struct timespec current_time;
clock_gettime(CLOCK_REALTIME, ¤t_time);
clock_settime(CLOCK_REALTIME, ¤t_time);
Available clocks:
• CLOCK_REALTIME — System-wide, real-time clock
• CLOCK_MONOTONIC — Time since an unspecified point
Interval timers:
• POSIX.1b timers are dynamically created:
struct itimerspec {
struct timespec it_value;
struct timespec it_interval;
};
timer_t created_timer;
timer_create(CLOCK_REALTIME, NULL, &created_timer);
One-shot alarm:
Interval timer:
/* signal handler */
void timer_expired(int signo) {...}
/* bind to SIGUSR1 */
struct sigaction sa;
sa.sa_handler = timer_expired;
sigemptyset(&sa.sa_mask);
sa.sa_flags = 0;
sigaction(SIGUSR1, &sa, NULL);
when.it_value.tv_sec = secs_since_epoch;
itimer_settime(created_timer,
TIMER_ABSTIME, &when, NULL);
struct tm absolute;
struct timespec abs_time;
absolute.tm_hour = 23; /* 12:xx pm */
absolute.tm_min = 30; /* xx:30 */
absolute.tm_mon = 8; /* September */
...
abs_time.tv_sec = mktime(&absolute);
abs_time.tv_nsec = 0;
timer_settime(created_timer, 0, &abs_time, NULL);
int n_overruns;
n_overruns = timer_getoverrun(created_timer);
union sigval {
int sival_int;
void *sival_ptr;
};