directCell: H.

Penner
U. Bacher
Hybrid systems J. Kunigk
C. Rund
with tightly H. J. Schick

coupled
accelerators
The Cell Broadband Enginet (Cell/B.E.) processor is a hybrid
IBM PowerPCt processor. In blade servers and PCI Expresst
card systems, it has been used primarily in a server context, with
Linuxt as the operating system. Because neither Linux as an
operating system nor a PowerPC processor-based architecture is
the preferred choice for all applications, some installations use the
Cell/B.E. processor in a coupled hybrid environment, which has
implications for the complexity of systems management, the
programming model, and performance. In the directCell approach,
we use the Cell/B.E. processor as a processing device connected to
a host via a PCI Express link using direct memory access and
memory-mapped I/O (input/output). The Cell/B.E. processor
functions as a processor and is perceived by the host like a device
while maintaining the native Cell/B.E. processor programming
approach. We describe the problems with the current practice that
led us to the directCell approach. We explain the challenge in
programming, execution control, and operation on the accelerators
that were faced during the design and implementation of a
prototype and present solutions to overcome them. We also provide
an outlook on where the directCell approach promises to better
solve customer problems.

Motivation and overview
Multicore technology is a predominant trend in the
information technology (IT) industry: Many-core and
heterogeneous multicore systems are becoming common.
This is true not only for the Cell Broadband Engine***
(Cell/B.E.) processor, which initiated this trend, but also
for concepts such as NVIDIA CUDA** [1] and Intel
Larrabee [2].
The Cell/B.E. processor and its successor, the IBM
PowerXCell* 8i processor, are known for their superior
computing performance and power efficiency. They
provide more flexibility and easier programmability
compared with graphics processing devices or field
programmable arrays, alleviating the challenge of
adapting software to accelerator technology. Yet, it is
still difficult in some scenarios to fully leverage these
performance capabilities with Cell/B.E. processor chips
alone. The main inhibitors are lack of applications for the
IBM Power Architecture* and the performance
characteristics of the IBM PowerPC* core
implementation in the Cell/B.E. processor. For mixed
workloads in environments for which the Cell/B.E.
processor is not suitable, such as Microsoft Windows**,
various hybrid topologies consisting of Cell/B.E.
processors and non-Cell/B.E. processor-based
components can be appropriate, depending on the
workload. Adequate system structures such as the
integration of accelerators into other architectures have
to be employed to reduce management and software
overhead. The most significant factors to be considered in
the design of Cell/B.E. processor-based system complexes
are the memory and communication architectures. The
following sections describe the properties of the
Cell/B.E. Architecture (CBEA) and their consequences
for system design.

Copyright 2009 by International Business Machines Corporation. Copying in printed form for private use is permitted without payment of royalty provided that (1) each
reproduction is done without alteration and (2) the Journal reference and IBM copyright notice are included on the first page. The title and abstract, but no other portions, of this
paper may be copied by any means or distributed royalty free without further permission by computer-based and other information-service systems. Permission to republish any other
portion of this paper must be obtained from the Editor.
0018-8646/09/$5.00 ª 2009 IBM

IBM J. RES. & DEV. VOL. 53 NO. 5 PAPER 2 2009 H. PENNER ET AL. 2:1
Memory hierarchy of the CBEA
The CBEA [3] contains various types of processor cores
on one chip: The IBM PowerPC processor element (PPE)
is compatible with the POWER* instruction set
architecture (ISA) but employs a lightweight
implementation with limited branch prediction and in-
order execution. The eight synergistic processor elements
(SPEs) are independent cores with a different ISA
optimized for computation-intensive workloads. The SPE
implementation on the chip targets efficiency, using in-
order execution and no dynamic branch prediction. Code
running on the PPE can access main memory through the
on-chip memory controller. In contrast, the SPE
instruction set allows access only to a very fast but small
memory area, the local store (LS), that is local to the core.
The LSs on the SPEs are not coherent to any other entity
on the chip or in the system but can request atomicity of
memory operations with respect to concurrent access to
the same memory location. For communication and
dataflow, the SPEs are able to access main memory and
I/O (input/output) memory using direct memory access
(DMA).
Loosely coupled systems
Loosely coupled systems cannot access the memories of
other nodes. Most clusters in the area of high-
performance computing (HPC) follow this system design,
including the first petaflops supercomputer [4]. If the
application permits, these systems scale very well.
Interconnectivity is usually attained by a high-speed
network infrastructure such as InfiniBand**. The data
distribution across the nodes is coarse grained to keep the
network overhead for synchronization and data exchange
low compared with the data processing tasks. The
predominant programming model for loosely coupled
systems is Message Passing Interface (MPI) [5],
potentially enriched by the hybrid IBM Accelerator
Library Framework (ALF) [6] and hybrid Data
Communication and Synchronization (DaCS) [7] libraries
to enable the Cell/B.E. processor in mixed-architecture
nodes. Another model based on remote procedure calls
is IBM Dynamic Application Virtualization [8].
Tightly coupled systems
Tightly coupled systems are usually smaller-scale systems
consisting of few computing entities. In our context, a
single host component, such as an x86 processor-based
system, exploits one or several accelerator devices.
Accelerators can range from field-programmable gate
arrays (FPGAs) over graphics processing units (GPUs) to
Cell/B.E. processor-based accelerator boards, each
offering different performance and programmability
attributes. Accelerators can be connected in a memory-
coherent or noncoherent fashion. Interconnects for
2:2 H. PENNER ET AL.
memory-coherent complexes include Elastic Interface or
Common Systems Interconnect. HyperTransport** can
be used in memory coherent and PCI Express** (PCIe**)
can be used in noncoherent memory setups. It is a
common feature of tightly coupled systems that they are
able to access the single main memory.
Applications for tightly coupled systems usually run on
one host part of the system (e.g., an x86 processor), while
the other parts of the system (accelerators such as
PowerXCell 8i processor-based entities or GPUs) will
perform computation-intensive tasks on data in main
memory. The tightly coupled approach makes
deployment easier because only one host system with
accelerator add-ons has to be maintained rather than a
cluster of nodes. Depending on the type of the tightly
coupled accelerator, the approach can be applied well to
non-HPC applications and allows for efficient and fine-
grained dataflow between components, as latency and
bandwidth are better than in loosely coupled systems.
Programming models include OpenMP** [9], ALF [6],
and DaCS [7] libraries, CUDA, the Apple OpenCL**
stack [10], and FPGA programming kits. Compared with
FPGA and GPU approaches, Cell/B.E. processor
accelerators, with their fast and large memories, offer the
highest flexibility and generality while maintaining an
easy-to-use programming model.
High-level system description
This section describes the directCell architecture. It
reflects on properties of acceleration cores and gives
insight on the design principles of efficiently attaching the
accelerator node.
General-purpose processor compared with SPEs on
the Cell/B.E. processor
The remarkable performance of the Cell/B.E. processor is
achieved mainly through the SPEs and their design
principles. The lack of a supervisor mode, interrupts, and
virtual memory addressing keeps the cores small,
resulting in a high number of cores per chip. High
computational performance is achieved through single-
instruction, multiple-data (SIMD) capabilities. In
directCell, the SPE cores are available for application
code only, while the PPE is not used for user applications.
This effectively makes the directCell accelerator an SPE
multicore entity. Unlike limited acceleration concepts
such as GPUs or FPGAs, the SPE cores are still able to
execute general-purpose code, for example, that
generated by an ordinary C compiler.
In standalone Cell/B.E. processor-based systems, OS
(operating system)-related tasks are handled on the PPE.
The task of setting up SPEs and starting program codes
on them is controlled from outside, using a set of
memory-mapped registers. All of these registers are
IBM J. RES. & DEV. VOL. 53 NO. 5 PAPER 2 2009
accessible through memory accesses and no special
instructions have to be used on the PowerPC core of the
Cell/B.E processor. Therefore, control tasks can be
performed by any entity having memory access to the
Cell/B.E. processor-side memory bus.
Coupling as device and not as system
In most cases, porting Windows applications to the
Power Architecture is not a viable option. Instead,
identifying computational hotspots and porting these to
the Cell/B.E. processor-based platform offers a way to
speed up the application. As the SPE cores are used
for acceleration, the port has to target the SPE
instruction set.
Cell/B.E. processor-based systems use PCIe as a high-
speed I/O link. PCIe is designed for point-to-point short-
distance links. This communication model does not scale
to many endpoints. However, it provides best-of-breed
bandwidth and latency characteristics and supports
memory-mapped I/O (MMIO) and DMA operations.
Because the Cell/B.E. processor resources are accessible
via MMIO, it is possible to directly integrate it into the
address space of other systems through PCIe following
the tightly coupled accelerator concept.
Communication overhead in this PCIe-based setup can
be kept minimal. The directCell approach takes
advantage of the low latencies of PCIe and makes the
SPEs available for existing applications as an acceleration
device instead of a separate system. To export a Cell/B.E.
processor-based system as an acceleration device on the
PCIe level, it must be configured as a PCIe endpoint
device, an approach currently used by all hybrid setups
involving the Cell/B.E. processor, including the first
petaflops computer [4]. The directCell solution evolves
from the endpoint mode and uses the Cell/B.E. processor
as a device at the host system software level. We propose
to relocate all control from the PPE to the host system
processor except for the bare minimum needed for
accelerator operation.
For directCell, we further propose to shift all
application code executed on the PPE in a standalone
configuration over to the host processor. This includes
scheduling of SPE threads and SPE memory management
and also applies to the PPE application code. Thus, the
host controls all program flow on the accelerator device.
None of the Cell/B.E. processor resources are hidden in
our model. More importantly, nothing on the accelerator
device obstructs the host from controlling it. In contrast
to a distributed hybrid system approach that relies on an
accelerator-resident OS to control all of its memory, we
consider an accelerator-resident OS superfluous. The host
OS runs the main application and is, therefore, in charge
of all of the hardware resources that are needed for its
execution.
IBM J. RES. & DEV. VOL. 53 NO. 5 PAPER 2 2009
In distributed hybrid systems, the host OS needs to
coordinate data-transfer operations with the accelerator
OS. MPI [5], which was designed for distributed systems,
features a sophisticated protocol to establish zero-copy
remote DMA (RDMA) transfers that are vital to hybrid
system performance. Without using an OS on the
accelerator, the host can act independently, rendering
much more flexibility to the host system programmer and
saving additional runtime overhead.
In order to support this execution model from the host
system software, the accelerator hardware should provide
the following mechanisms:
 Endpoint functionality.
 MMIO control of processor cores, including next
instruction pointer, translation lookaside buffer
(TLB), machine state register (MSR), and interrupt
acknowledgment.
 Interrupt facilities,
 Optionally PCIe coherency.
Providing these mechanisms in a processor design
yields enormous rewards with regard to system
configurations, especially in hybrid configurations.
The directCell architecture applies to Cell/B.E.
processor-based blades such as the IBM BladeCenter*
QS22 server [11] with specific firmware code to operate
in PCIe endpoint mode and the IBM PowerXCell
accelerator board (PXCAB) offering [12].
Applications and programming
Performance and flexibility are two essential and
generally opposing criteria when evaluating computer
architectures. On the performance side, specially designed
circuits, digital signal processors, and FPGAs provide
extremely high computing performance but are lacking
generality and programmability. Regarding flexibility,
general-purpose processors allow for a wide application
spectrum but lack the performance of highly optimized
systems. Cell/B.E. processors follow a tradeoff and
provide high computing performance while still offering a
good degree of flexibility and ease of programming.
The tightly coupled hybrid approach proposed with
directCell in this paper promises to capture the
advantages of both extremes by combining Cell/B.E.
processor-based accelerators for performance and x86
systems for wide applicability. Two aspects apply to
programmability for these hybrid systems: first, the
complexity to program the accelerator (i.e., the SPEs),
and second, the integration of the tightly coupled
accelerator attachment into the main application running
on the host system (i.e., the model to exploit the SPEs
from the x86 system).
H. PENNER ET AL. 2:3
Standard programming framework of the Cell/B.E.
processor
The SPE Runtime Management Library (libspe2) [13],
developed by IBM, Sony, and Toshiba, defines an
OS-independent interface to run code on the SPEs. It
provides basic functions to control dataflow and
communication from the PPE. Cell/B.E. processor
applications use this library to start the execution of code
on the SPEs using a threading model while the PPE runs
control and I/O code. The code on the PPE is usually
responsible for shuffling data in and out of the system and
for controlling execution on the SPEs. In most cases, code
on the SPEs pulls data from main memory into the LSs so
that the SPEs can work on this data. Synchronization
between the SPEs and between the PPE and SPEs can
take place through a mailbox mechanism.
Programming tightly coupled accelerators
The programming model and software framework for
tightly coupled accelerators such as the Cell/B.E.
processor can be structured in several ways. One way,
which resembles the loosely coupled systems approach
[14], is to add a communication layer between the two
standalone systems, the host and Cell/B.E. processor-
based system. These additional layers increase the burden
on software developers who have to deal with an explicit
communication model instead of a programming library.
A different model is used for directCell that replaces
parts of the PPE functionality with host processor code.
The PPE does not run application-specific code; instead,
code on the host processor assumes the functions that
were previously performed by the PPE. This allows
applications to fully leverage the advantages of a tightly
coupled accelerator model. In our implementation of
directCell, the libspe2 library is ported to the x86
architecture, which allows control of SPE code execution
and synchronization between an x86 system and SPE
entities. This reduces the complexity of programming the
directCell system to the level of complexity required to
natively program the Cell/B.E. processor. Existing
Cell/B.E. processor-aware code can continue to run
unchanged or with limited changes in this environment,
tapping a large number of kernels running on SPEs for
x86 environments. Furthermore, existing Windows or
Linux** applications can be extended by libspe2 calls to
leverage SPE acceleration, opening up the x86
environment and other application environments to
Cell/B.E. processor acceleration.
Implications of the tightly coupled accelerator
structure
The directCell approach has several implications on
programming and applications:
2:4 H. PENNER ET AL.
 Most of the application code runs on x86. Any code
that previously ran on the PPE now has to be built for
and run on the x86 platform.
 The libspe2 application programming interface (API)
to control SPEs is moved to x86, i.e., libspe2 is an x86
library.
 SPEs can perform DMA transfers between the SPE
LS and the x86 main memory as used by the main
application. DMA to the northbound memory that is
local to the Cell/B.E. processor accelerator systems is
still possible for SPEs. For transfers between the x86
memory and SPEs, the different endianness
characteristics of the architectures need to be
considered. (Endianness refers to whether bytes are
stored in the order of most significant to the least
significant or the reverse.) SIMD shuffle byte
instructions can be used for efficient data conversion
on the SPEs and may have to be added as a
conversion layer, depending on the type of data
exchanged between host and accelerator.
 The PCIe interface between the Cell/B.E. processor
and the x86 components does not permit atomic
DMA operations, a restriction that applies to all
accelerators attached through PCIe. Atomic DMA
operations on the Cell/B.E. processor local memory
are still possible. This constraint requires code
changes to existing applications that use atomic DMA
operations for synchronization.
 Memory local to the Cell/B.E. processors can be used
by SPE code for buffering; because of its proximity to
the Cell/B.E. processor chips, it has better
performance characteristics than the remote x86
memory. However, adding this mid-layer of memory
between the x86 main memory and the SPE LSs adds
to application complexity. A good way of using the
memory local to the Cell/B.E. processors might be to
use it to prefetch and cache large chunks of host
system data. The complexity of prefetching and
loading data on-demand from the Cell/B.E. processor
local memory can typically be encapsulated in a few
functions.
Structure of directCell
The directCell model can be applied to Cell/B.E.
processor-based systems only by means of software.
Several software components are needed on the host and
accelerator system side. In the following sections, we
introduce the components and how they interface with
each other.
Libspe2.dll
The basic interface for controlling SPE applications in
standalone Cell/B.E. processor-based systems is libspe2
IBM J. RES. & DEV. VOL. 53 NO. 5 PAPER 2 2009
[13]. To conform to this widely adopted programming
interface, and to be able to reuse a large amount of
existing code, directCell provides a Windows port via a
Windows DLL (dynamic linked library) that provides
such functions as spu_context_create() and
spu_context_run(). The libspe2 routines create a
directCell device instance and send I/O control
commands processed by the device driver.
The libspe2 used in standalone system configurations
interfaces with the SPE hardware resources through the
synergistic processor unit (SPU) file system (spufs) [15] in
the Linux kernel. Spufs offers two system calls and a
virtual file system for creating SPE threads that the Linux
kernel schedules to run on the SPE. In the Windows
implementation, the spe_context_create() libspe2 call
allocates a user-space representation of the SPE program
and also a kernel-space device instance that is accessed
by a device handle in user space. The function
spe_context_run() invokes a corresponding I/O control
(ioctl) that passes the device handle and a pointer to
the user-space buffer as parameters.
SPE binaries are embedded in a given Windows
application as executable and linking format (ELF) files.
In our Windows implementation, the ELF files are
integrated into accelerated applications as ELF resources.
The Windows API provides standard resource-handling
functions that are used to locate and extract a previously
embedded file at runtime. The libspe2 port also
implements ELF loading functionality for processing
the embedded SPU program to pass it to the
spe_context_run() call.
Device driver
Our implementation of directCell features a sophisticated
device driver that provides a software representation of
all of the Cell/B.E. processor facilities on a host system.
Accelerated application code is loaded, executed, and
unloaded by operating the device driver interfaces. From
a conceptual level, this closely corresponds to the well-
established spufs. The idea to span the spufs interface
across a high-speed interconnect has been pursued by
Heinig et al. [16] as rspufs [16]. Regarding the user-space
representation, spufs is based on the UNIX** file concept
by representing SPE hardware through a virtual file
system. In the case of Windows, the representation of
hardware in user space is dominated by a different model.
In Windows, files, volumes, and devices are represented
by file objects, but the concept of imposing a directory
structure for device control is not widely supported.
Windows allows the creation of device interface classes
that are associated with a globally unique identifier
(GUID) and a set of system services that it implements.
A device interface class is instantiated by creating a file
handle with the GUID. This handle is used to invoke
IBM J. RES. & DEV. VOL. 53 NO. 5 PAPER 2 2009
system services. The granularity of this scheme centers
around the idea of a single device node as found in a
Linux /dev tree. The fine-grain control gained by
subdividing the device details into a file system hierarchy
such as spufs is opposed by a large variety of system
services, each tailored for a specific purpose on a distinct
device class. Device drivers that support I/O operations
implement a relatively small set of I/O functions via an
I/O queue. The driver I/O queue registers callbacks, most
importantly read, write, I/O-control, stop, and resume.
The actual communication between a user-space
application and a device takes place through an
appropriate device service that invokes the Windows I/O
manager, which forms an I/O request package, puts it
into the I/O queue of the correct driver, and notifies the
driver via the previously registered callbacks.
The device driver can satisfy some of the function
requirements of higher level libraries such as libspe2 by
using existing system services. The command
spe_context_create() generates an SPE context by
invoking the CreateFile() system service using the
Windows Cell/B.E. processor device handle. This will
return a handle to a kernel representation of an SPE
context that is similar to the representation returned by
an spu_create() system call in spufs. The spu_run()
function is implemented as an ioctl that operates on the
previously created SPE handle. The ioctl is nonblocking,
although upper-level software should wait for completion
via designated wait routines for asynchronous I/O. The
ioctl initiates a direct DMA transfer of the SPE context
from host user-space memory to the LS of the SPU.
All data transfers in directCell follow this zero-copy
paradigm.
Regarding the mechanisms to access the Cell/B.E.
processor hardware facilities, the device driver must
perform additional steps compared to spufs. These
additional steps are due mainly to the conversion of the
Cell/B.E. processor address space and the host system
address space, as described in the following sections.
Cell/B.E. processor firmware layer
The standard product firmware of a Cell/B.E. processor-
based system is supplemented by a minimal software layer
that takes care of additional initialization tasks after
the endpoint-aware standard firmware has finished the
basic boot. Most importantly, this layer provides
accelerator runtime management and control. The
initialization tasks comprise the setup of the address
translation mechanism of the SPEs, the reporting of
available SPEs to the device driver before accelerator
usage, and the setup of the I/O memory management unit
(I/O MMU). The runtime management and control
functionality centers around handling of SPE events such
as page faults and completion signals. This handling is
H. PENNER ET AL. 2:5
 4

Host system Cell/B.E. processor-based system

Host processor Host SPE PPE

memory
SPU
Core 4 Core
Execution
Local store Firmware
Device driver units
Control
block
1 3,6 MFC
MMIO L2
L2 cache DMA cache
registers
SPU 6
context

3 2,5
2,5
Element
Host interconnect
southbridge bus Memory
Issue Southbridge chipset
MMIO PCIe link
DMA

Figure 1
In directCell, an SPE program is started on the host processor by directly programming an SPE via the PCIe link between the host processor
southbridge and that of the Cell/B.E. processor. Shown are the most important directCell software components and their lifecycle in the
directCell hybrid system configuration.

vital for the execution of SPE binaries. Additional
runtime debug and monitoring facilities for the SPEs
complete the firmware layer functionality. The event
handler and the debug monitor each run on separate PPE
hardware threads. In our current implementation, the
debug monitor provides a simple facility for probing
memory locations, memory-mapped resources, and
interfaces via the serial console. In future
implementations, the debug console I/O could be
redirected via PCIe to a console application that runs
on the host.
Accelerator operation
While directCell maintains compatibility with the
common programming models for the CBEA [17], it
introduces a new runtime model. Figure 1 shows and the
following paragraphs describe in detail how directCell
delivers efficient workload acceleration and makes
additional overhead imposed by a Cell/B.E. processor-
resident OS superfluous. The numbers in parentheses
below refer to the numbers in red in the figure.
Consider a user-space application on x86 that needs to
run an SPE program via a PCIe-attached directCell
device. Through our libspe2 implementation, it interfaces
with the device driver by using the standard Windows
API functions to allocate an SPE context (representation
of LS, general-purpose registers, and problem or
privileged state). The SPE binary is retrieved from the
executable main application and loaded into the SPE
context. The device driver implements a standard
Windows function to support opening a device
representation of the directCell system and running
the previously created SPE context (1). The
spe_context_run() method is implemented as an ioctl.
Upon invocation, the driver picks a free physical SPE on
the remote system and orchestrates the transfer of the
SPE context to that system via MMIO operations (2). For
the actual transfer of data, it is crucial that only DMA is
used to maximize throughput and minimize latency
because the data has to traverse the PCIe bus. MMIO is
used only as a control channel of the host system into the
Cell/B.E. processor-based system. This concept of
loading comprises a small bootstrap loader as described

2:6 H. PENNER ET AL. IBM J. RES. & DEV. VOL. 53 NO. 5 PAPER 2 2009
in Reference [18] to achieve this. The loader is transferred
to the SPE in a single 16-KB DMA operation that also
receives and contains as parameters the origin address of
the target SPE binary and its size. The loader transfers the
contents of the SPE register array into an internal data
structure and initializes the registers before filling the LS
with the binary SPE code via DMA (3). The loader will
stop and signal completion to the PPE via an external
interrupt. This interrupt is forwarded to the device driver
by the PPE (4), which runs a tiny firmware layer as part of
the only software prevailing on the PPE (as introduced in
the previous section).
While the SPE binary code executes, it generates
mapping faults (4) whenever a reference to an unmapped
host-resident address is encountered. This applies, for
instance, to parameter data that may reside in a control
block on the host. The corresponding memory mapping is
established by the device driver via MMIO (5), and the
host-resident data is then transferred via DMA (6). When
the SPE stops execution, the host is signaled once again
via interrupt and initiates a context switch and unload
procedure, as described in Reference [9]. Similar to the
previous steps, this phase involves unloader binary code
that facilitates a swift transfer of the context data to the
host memory via the SPE DMA engines.
Memory addressing model
One of the concepts of the directCell approach is to define
a global address mapping that allows the SPEs to access
host memory and the host to access the MMIO-mapped
registers of the Cell/B.E. processor, LS, and local main
memory. The mechanism that achieves this is based on
using the memory management unit (MMU) of the host,
the MMU and I/O MMU of the Cell/B.E. processor, and
the PCIe implementation features, which we show in
the rest of this section.
The BAR setup section introduces how MMIO
requests from the host pass the PCIe bus and the section
on inbound mappings shows how those are translated to
Cell/B.E internal addresses. These internal addresses are
further translated by the I/O MMU to PowerPC real
addresses (RAs).
The IOMMU defines two translation ranges:
1. Requests directed to the SPE MMIO space are
furnished with a predefined displacement by the
southbridge (I/O chipset). The I/O MMU translates
these displaced addresses to the base address of the
SPE MMIO spaces within the Cell/B.E. processor
addressing domain and preserves offsets to navigate
within this range.
2. Requests directed to Cell/B.E. processor memory are
translated 1:1 by the I/O MMU, also preserving the
offset for internal navigation.
IBM J. RES. & DEV. VOL. 53 NO. 5 PAPER 2 2009
Figure 2 shows a simplified model for MMIO
communication. The MMIO transfer is triggered by
writing to virtual addresses (VAs) on the host that are
translated by the host MMU to RAs that, in turn, map to
the host PCI bus. The MMIO requests are delivered as
PCI memory cycles to the accelerator PCIe core where
they are further redirected to the interface between the
southbridge and Cell/B.E. processor interface, as
described below in the section on inbound mappings. At
that point, the I/O MMU maps the requests to the
corresponding units on the Cell/B.E. processor.
The SPEs transfer data via DMA to or from the
accelerator system upon remote initiation by the host via
MMIO. The target of the data-transfer operations may be
local Cell/B.E. processor addresses (LS or local main
memory) or remote host memory. The host addresses
must be mapped to the I/O range of the Cell/B.E.
processor; hence, accesses to these regions do not satisfy
the coherency requirements that apply for local main
memory. The device driver is responsible for locking the
host memory regions to prevent swapping. For future
enhancements of the directCell model, we encourage the
introduction of a mechanism on the host system to
explicitly request local main memory buffering to
improve performance and coherency considerations.
Further enhancements could provide a software-managed
mechanism to prefetch and cache host-resident
application buffers into local main memory [19].
The memory mappings are established through direct
updates of the SPE TLBs whenever the SPE encounters
an unmapped address. This handling is initiated by a
page fault, as described in the section on interrupt
management. In our current implementation of
directCell, the host device driver performs this
management via MMIO. The current implementation
features a page size of 1 MB; hence, a single SPE can
work on 256 MB of application data buffered in local
memory or host-resident memory at once without the
need of a page table. This area can be increased if larger
common page sizes are available on the host and the
accelerator. To introduce hardware management of the
TLBs to directCell, the firmware layer would need to be
augmented to manage a page table in the local main
memory.
The flow of DMA operations is shown in Figure 2. The
SPE views a DMA operation as a transfer from one VA
to another. When reading data from the host system (blue
arrows), the target address translates to an LS address,
and the source address translates to an RA in the I/O
domain, which points to the outbound memory region of
the accelerator PCIe core. The mapping of PCIe bus
addresses to host RAs is likely to involve another
inbound translation step; however, this is transparent to
H. PENNER ET AL. 2:7
 Host MMIO operations
Host VA Host RA
Accelerator
memory mapping
SPE MMIO mapping
Mappings of individual pages
across address spaces
Cell/B.E. VA Cell/B.E. RA
Cell/B.E.
Accelerator memory MMU
and SPE LS
Host memory
mapping in I/O space
Figure 2
The various levels of addressing translation and indirection between the host and the accelerator in directCell: (a) a control channel
implemented by means of MMIO; (b) a data-transfer channel implemented via DMA. (VA: virtual address; RA: real address.)
the accelerator system and entirely dependent on the host
OS, which directCell does not interfere with.
PCIe
PCIe was developed as the next-generation bus system
that replaces the older PCI and PCI-X** standards. In
contrast to PCI and PCI-X, PCIe uses a serial
communication protocol. The electrical and bit layer is
similar to the InfiniBand protocol, which also uses 8b10b
coding for the bit layer [20]. The entire PCIe protocol
stack is handled in hardware. The programmer does not
need to build any queues or messages to transfer actual
data. Everything is done directly with the processor load
and store instructions, which move data from and to the
bus. It is possible to map a memory region from the
remote side to the local memory map. This mapping is
software compatible with the older PCI or PCI-X
standards. Because of this, a mapping can be created so
that the local processor can access remote memory
directly via load or store operations and effectively it does
not need any software libraries to use this high-speed
communication channel.
2:8 H. PENNER ET AL.
Cell/B.E. RA
Accelerator
PCIe bus memory
SPE MMIO
Cell/B.E.
I/O MMU
(a)
Accelerator DMAs from host Accelerator DMAs to host
Host RA
PCIe bus User-space
buffer
(b)
PCIe is used as a point-to-point interconnection
between devices and a system. This allows centralizing the
traffic routing and resource management and enables
quality of services. Thus, a PCIe switch can prioritize
packets. For a multimedia system, it could result in fewer
dropped frames and lower audio latency. PCIe itself is
based on serial links. A link is a dual-simplex connection
using two pairs of wires. One pair is used for transmitting
data and one is used for receiving data. The two pairs are
known as a PCIe lane. A PCIe link consists of multiple
lanes, for example, x1, x2, x4, x8, x16, or x32.
PCIe is a good communication protocol for high-
bandwidth and low-latency devices. For example, a
PCIe 2.0 lane can transmit 500 MB/s per direction, which
yields to 16 GB/s per direction for an x32 link
configuration. A very compelling possibility of the PCIe
external cabling specification is the option to interconnect
an external accelerator module via a cable to a
corresponding host system.
PCIe endpoint configuration
PCIe defines point-to-point links between an endpoint
and a root-complex device. In a standalone server
IBM J. RES. & DEV. VOL. 53 NO. 5 PAPER 2 2009
 whereas the Cell/B.E. processor-based system
Host southbridge is configured to run in endpoint mode. The
Accelerated application
configuration is done by the service processors of the two
systems prior to booting. From the host system, the
Windows libspe Cell/B.E. processor-based system is seen as a PCIe device,
with significant device identification, which allows the
1) Move accelerator memory window
2) DMA to accelerator memory Cell./B.E. BIOS (basic I/O system) and OS to configure the MMIO
3) Operate SPEs system driver ranges and load the appropriate device drivers. The
scalability of this model of endpoint coupling is limited
Host central
processing unit only by how much I/O memory BIOS can map.
(x86)
PCe inbound mapping
I/O MMU MMU
In order to remotely access the Cell/B.E. processor-based
Host system, the device driver and the firmware layer configure
southbridge the Cell/B.E. processor-based system southbridge to
correctly route inbound PCIe requests to the Cell/B.E.
processor and the local main memory. This inbound
Accelerator mapping involves PCI base address registers (BARs) on
the bus level and PCI inbound mappings (PIMs) on the
PCIe core
southbridge level. Both concepts are introduced in the
Outbound BAR BAR BAR Configuration
following sections.
mapping space
PIM PIM PIM
BAR setup
The PCI Standard [21] defines the BARs for device
128 MB 128 MB configuration by firmware or software as a means to
Southbridge assign PCI bus address ranges to RAs of the host system.
internal bus The BARs of the PCIe core in the southbridge are
Companion chip configured by PCIe config cycles of the host system
(southbridge) Cell/B.E. BIOS during boot. After the configuration, it is possible
system
to access internal resources of the accelerator system
memory
MMU through simple accesses to host RAs. This makes it
I/O MMU
possible to remotely control any Cell/B.E. processor-
related resources such as the MMIO register space, SPU
0x0 0x8000
resources, and all available local main memory. However,
to achieve full control of the accelerator system, an
Registers

Registers
DMA

DMA

MMIO additional step is required, which is introduced in the

region Cell/B.E. processor section on inbound mappings.
LS LS ... The directCell implementation uses three BAR
configurations to create a memory map of the device
resources into the address space of the host system:

Figure 3 1. BARs 0 and 1 forming one 64-bit BAR provide a

Overview of the inbound addressing mechanisms of the Cell/B.E.
southbridge. Inbound PCIe requests are mapped to distinct ranges
within the southbridge internal address space, which in turn are
mapped to specific units of the Cell/B.E. processor. The address
mappings can be changed at runtime.
configuration such as the QS22 BladeServer, the
southbridge of the Cell/B.E. processor-based system is
configured as a root complex to support add-in daughter
cards. For directCell, the host system southbridge
configuration is unchanged and runs as a root complex,
128-MB region of prefetchable memory used to
access all of the accelerator main memory (Figure 3).
2. A 32-bit BAR 2 is mapped to the internal resources
of the end-point configuration itself, making it
possible to change mappings from the host. This
BAR is 32 KB in size.
3. BARs 5 and 6 forming one 64-bit BAR are used to
map the LS, the privileged state, and problem state
registers of all SPEs of the Cell/B.E. processor.
Figure 3 is an overview of this concept. We can see how
the driver can conveniently target different BARs for
directly reaching the corresponding hardware resources.

IBM J. RES. & DEV. VOL. 53 NO. 5 PAPER 2 2009 H. PENNER ET AL. 2:9
Inbound mappings
The Cell/B.E. processor companion chip, southbridge,
introduces PIM registers to provide internal address
translation capabilities. PIMs apply a displacement
within the southbridge internal bus address space, as
shown in Figure 3.
For all BARs, corresponding PIM registers exist to
map addresses on the PCIe bus to the southbridge
internal bus [22] addresses connected to the southbridge-
to-Cell/B.E. processor interface.
The PIM register translation for BAR 2 is set up by the
accelerator firmware to refer to the MMIO-accessible
configuration area of the PCIe core. The PIMs for BARs
0 and 1 and 5 and 6 are adjusted by the host device driver
during runtime. In order to do so, the device driver uses
the inbound mapping for BAR 2. The resources of the
Cell/B.E. processor are made accessible to a PPE space
address by the I/O MMU setup (described previously in
the section on PCIe and depicted in the lower part of
Figure 3).
The PIM register translation for BAR 0 and 1 is
changed during the runtime of host applications
according to the needs of the driver. The driver provides a
movable window across the complete main memory of
the accelerator, though at a given point in time, only
128 MB of contiguous accelerator memory space is
available (also shown in Figure 3).
Interrupt management
The asynchronous execution of SPE programs requires
an interrupt handling infrastructure for directCell. SPE
programs use interrupts to signal such events as mapping
faults, errors, or program completions.
Several layers are involved in handling SPE interrupts.
An external interrupt is routed to the interrupt controller
on the PPE, which is handled by an interrupt handler
in the PPE resident firmware layer. This handler
determines which SPU triggered the interrupt and directly
triggers an interrupt to the PCIe core on the southbridge,
which is in turn handled by the host device driver.
Three types of SPE interrupts are handled by
directCell:
1. SPU external interrupt: page-fault exception
handling, DMA error, program completion.
2. Internal mailbox synchronization interrupt.
3. Southbridge DMA engine interrupt.
The inclusion of the PPE in the interrupt-handling
scheme is necessary because the provided hardware does
not allow the direct routing of the SPE interrupt to the
PCIe bus. We propose to include the ability to freely
reroute interrupts to different targets in future processors
and I/O devices.
2 : 10 H. PENNER ET AL.
Table 1 Throughput from SPE local store to x86-based host
memory.
Configuration Throughput (MB/s)
QS22 at 3.2 GHz, SMP, PCIe x4, reads 719.22
QS22 at 3.2 GHz, SMP, PCIe x4, writes 859.44
PXCAB at 2.8 GHz, PCIe x16, reads 2,333.95
PXCAB at 2.8 GHz, PCIe x16, writes 2,658.60
Performance evaluation
In this section, first results of the directCell
implementation are presented. The results are sectioned
into a discussion on general throughput and latency
measurements for specific directCell hardware
configurations and a discussion of an exemplary
application that we apply to our implementation of
directCell. The latter section elaborates on the effort of
porting the application to directCell as well as application
performance data.
General measurements
These tests aim to determine the throughput and latency
characteristics of the host and accelerator interconnect in
two distinct directCell hardware configurations:
1. A BladeCenter server-based configuration IBM
QS22 blade with an IBM HS21 blade. The
interconnect is PCIe Gen1, x4.
2. A PowerXCell processor-based accelerator board
PCIe add-in card connected to a standard x86-based
advanced technology extended personal computer.
The interconnect is PCIe Gen1, x16.
All measurements are performed by a small firmware
kernel running on the PPE of the Cell/B.E. processor-
based system. The kernel runs on the PowerXCell PPE in
real addressing mode and programs the SPE memory
flow controller DMA facilities via MMIO and transfers
LS contents to the host memory and back. Because the
MMIO operations are significantly slower than
programming the DMA directly from the SPE, the
methodology is not ideal; however, each test determines
the additional latency of all involved MMIO operations
and subtracts it from the overall duration.
Table 1 lists the throughput from the SPE LS to x86-
based host memory. The values shown were measured
with two SPEs submitting requests; however, a single SPE
will saturate the PCIe bus in both system configurations.
With regards to latency, we conducted several
measurements to determine the overhead of the I/O path.
We regard the hop from the PowerXCell processor to the
IBM J. RES. & DEV. VOL. 53 NO. 5 PAPER 2 2009
 6,000
1,800

1,600 Standalone QS22 1 SPE

SPE, southbridge 5,000 Standalone QS22 8 SPEs (avg.)

Simulations per second (millions)

1,400 loop back, local memory
directCell QS22 1 SPE
SPE, southbridge
1,200 directCell QS22 8 SPEs (avg.)
loop back, SPE 4,000
Latency (ns)

directCell PXCAB 1 SPE

1,000 SPE, southbridge
directCell PXCAB 8 SPEs (avg.)
PCIe, host memory
800 3,000

600
2,000
400

200
1,000
0
PXCAB QS22 PXCAB QS22
2.8-GHz 3.2-GHz SMP 2.8-GHz 3.2 GHz-SMP 0
read read write write

Figure 5
Figure 4 Overall application performance.
Latency of 16 KB SPE transfers to various targets (using processor
timebase).

completion, the control block is transferred back to main

southbridge separately from the overall access to the host
memory. This is possible by providing a PowerXCell
processor address that routes the request to the
southbridge in a way that it gets immediately routed back
to the PowerXCell processor by the southbridge. One of
these loop-back tests refers back to LS, a second accesses
the Cell/B.E. processor-resident DDR2 (double-data-rate
2) memory. The results for both tests are shown in
Figure 4, along with results for end-to-end data transfers
from LS to the x86 host memory.
Application porting
As an exemplary use case of directCell, we have ported a
Cell/B.E. processor-based application from the financial
services domain for European option trading. The
underlying software is a straightforward implementation
of a Monte Carlo pricing simulation. A derivative of this
application has been part of an IBM showcase at the
Securities Industry and Financial Markets Association
Technology Management and Conference and Exhibit
[23]. The computing-intensive tasks of the application lie
in the acquisition of random integer numbers and Moro’s
inversion of those [24]. The standalone Linux-based
version of the application transfers and starts one or more
SPE threads that will in turn read in a control block from
main memory before starting the simulation. The control
block contains such parameters as the number of
simulations that should be performed, the simulated span
of time, and the space to return result data. After
memory.
The development of the application port involved
porting the PPE code to the x86 architecture and
Windows as well as providing for endianness conversion
of the application control block in the SPE program. Of
the PPE code, 16% of all lines are reused verbatim. The
remaining code underwent mostly syntax changes to
conform to the data types and library function signatures
of Windows. The control flow of the application was not
changed; all lines of SPE code are reused verbatim.
Some additional 85 lines of code were needed to perform
endianness conversion of the control block data. The
conversion code makes use of the spu_shuffle
instruction with a set of predefined bit-swap patterns.
With adequate programming language support, such a
code block can be easily generated by a compiler, even for
complex data structures. The application port was
performed on a Microsoft Windows Server** 2008-based
implementation of directCell.
Application results
We compare the application performance on a standalone
native Cell/B.E. processor running Linux on a
BladeCenter QS22 server as well as a directCell
configuration of the same system with a BladeCenter
HS21 server and a directCell PXCAB configuration
connected to an AMD64. The most important
application parameters are set to 10 timesteps and
1 million simulations per run per SPE.
Figure 5 shows that the number of simulations per
second for the QS22-based directCell configuration is

IBM J. RES. & DEV. VOL. 53 NO. 5 PAPER 2 2009 H. PENNER ET AL. 2 : 11

800
700
Standalone QS22 1 SPE
600 directCell QS22 1 SPE
directCell PXCAB 1 SPE
500
Duration ( ␮s)
Standalone QS22 8 SPEs (avg.)
directCell QS22 8 SPEs (avg.)
400 directCell PXCAB 8 SPEs (avg.)
300
200
100
0
spe_context_create spe_program_load spe_context_destroy
Figure 6
Duration of host-resident libspe2 functions.
only marginally lower than the number achieved in the
nonhybrid standalone system. The overall performance
for the PXCAB-based directCell configuration is linear to
the lower processor frequency of PXCAB compared with
QS22.
We also evaluated the duration of distinct libspe2
function calls. Under Windows Server 2008, these
measurements were conducted by using the
QueryPerformanceCounter function. Under PowerPC-
Linux, the gettimeofday function was used to acquire
the durations.
The results shown for host-resident function calls in
Figure 6 reveal that the host processors used in the
directCell model perform certain tasks faster than the
PPE of the PowerXCell processor. Also, the frequency of
the function calls has a significant impact on their average
duration.
Finally, we measured the duration of handling SPE
page and segment faults in directCell. For this, we used a
PCIe analyzer that triggers first upon interrupt assertion
on the bus and triggers again upon the bus cycle that
updates the TLB of the SPEs with a new memory
mapping. This method omits the time that the interrupt
and the resulting MMIO cycle need to propagate through
the Cell/B.E. processor-based system, but the orders of
magnitude in terms of time for the handling of the page
fault on the bus and the host system handling allow us to
neglect this, as shown in Table 2.
Conclusion and outlook
We have presented directCell, an approach to attach a
Cell/B.E. processor-based accelerator to a Windows-
2 : 12 H. PENNER ET AL.
Table 2 SPE memory fault measurements.
Interrupt type Duration (ls)
SPE segment fault 15.984
SPE page fault 20.56
based x86 host system using PCIe connectivity. This
configuration taps today’s software environment while
the native Cell/B.E. processor programming model is
maintained, leveraging existing code for the CBEA.
The tight coupling of the accelerator and its low
communication latency allows for a broad range of
applications that go beyond HPC and streaming
workload while offering an integrated and manageable
system setup. The viability of this approach has been
demonstrated in a prototype. The PPE portion of the
Cell/B.E. processor runs a slim firmware layer
implementing initialization, memory management, and
interrupt handling code, granting the x86 host system full
access to the SPEs.
Further research should be undertaken to optimize
memory management and the work split between host
control software and the Cell/B.E. processor PPE
firmware components. Exploitation of the memory local
to the Cell/B.E. processor-based accelerator can improve
application performance transparently. Efforts to
evaluate scalability across several accelerator engines as
well as building an environment for efficient development
can lead to broader use of directCell.
*Trademark, service mark, or registered trademark of
International Business Machines Corporation in the United States,
other countries, or both.
**Trademark, service mark, or registered trademark of NVIDIA
Corporation, Microsoft Corporation, InfiniBand Trade
Association, HyperTransport Technology Consortium, PCI
Special Interest Group, OpenMP Architecture Review Board
Corporation, Apple, Inc., Linus Torvalds, or The Open Group in
the United States, other countries, or both.
***Cell Broadband Engine is a trademark of Sony Computer
Entertainment, Inc., in the United States, other countries, or both
and is used under license therefrom.
References
1. CUDA Zone, NVIDIA Corporation; see http://
www.nvidia.com/object/cuda_home.html#.
2. L. Seiler, D. Carmean, E. Sprangle, T. Forsyth, M. Abrash, P.
Dubey, S. Junkins, et al., ‘‘Larrabee: A Many-Core x86
Architecture for Visual Computing,’’ ACM Trans. Graph. 27,
No. 3, 1–15 (2008).
3. IBM Corporation, IBM Cell Broadband Engine Technology;
see http://www-03.ibm.com/technology/cell/index.html.
4. IBM Corporation (June 9, 2008).Roadrunner Smashes the
Petaflop Barrier. Press release; see http://www-03.ibm.com/
press/us/en/pressrelease/24405.wss.
IBM J. RES. & DEV. VOL. 53 NO. 5 PAPER 2 2009
 5. ‘‘MPI: A Message-Passing Interface Standard Version 2.1,’’
Message Passing Interface Forum, 2008; see http://www.
mpi-forum.org/docs/mpi21-report.pdf.
6. IBM Corporation, Accelerated Library Framework
Programmer’s Guide and API Reference; see http://
publib.boulder.ibm.com/infocenter/systems/scope/syssw/topic/
eiccn/alf/ALF_Prog_Guide_API_v3.1.pdf.
7. IBM Corporation, Data Communication and Synchronization
Library Programmer’s Guide and API Reference; see http://
publib.boulder.ibm.com/infocenter/systems/scope/syssw/topic/
eicck/dacs/DaCS_Prog_Guide_API_v3.1.pdf.
8. IBM Corporation, IBM Dynamic Application Virtualization;
see http://www.alphaworks.ibm.com/tech/dav.
9. OpenMP Architecture Review Board, OpenMP Application
Programming Interface Version 3.0, May 2008; see http://
www.openmp.org/mp-documents/spec30.pdf.
10. N. Trevett, ‘‘OpenCL Heterogeneous Parallel Programming,’’
Khronos Group; see http://www.khronos.org/developers/
library/2008_siggraph_bof_opengl/OpenCL%20and%
20OpenGL%20SIGGRAPH%20BOF%20Aug08.pdf.
11. IBM Corporation, IBM BladeCenter QS22; see ftp://
ftp.software.ibm.com/common/ssi/pm/sp/n/bld03019usen/
BLD03019USEN.PDF/.
12. IBM Corporation, Cell/B.E. Technology-Based Systems; see
http://www-03.ibm.com/technology/cell/systems.html.
13. IBM Corporation, SPE Runtime Management Library
Version 2.2; see http://www-01.ibm.com/chips/techlib/
techlib.nsf/techdocs/1DFEF31B3211112587257242007883F3/
$file/SPE_Runtime_Management_API_v2.2.pdf.
14. K. Koch, ‘‘Roadrunner Platform Overview,’’ Los Alamos
National Laboratory; see http://www.lanl.gov/orgs/hpc/
roadrunner/pdfs/Koch%20-%20Roadrunner%20Overview/
RR%20Seminar%20-%20System%20Overview.pdf.
15. A. Bergmann, ‘‘Spufs: The Cell Synergistic Processing Unit as
a Virtual File System’’; see http://www-128.ibm.com/
developerworks/power/library/pa-cell/.
16. A. Heinig, R. Oertel, J. Strunk, W. Rehm, and H. J. Schick,
‘‘Generalizing the SPUFS Concept–A Case Study Towards
a Common Accelerator Interface’’; see http://private.ecit.
qub.ac.uk/MSRC/Wednesday_Abstracts/Heinig_Chemnitz.pdf.
17. IBM Corporation, Software Development Kit for Multicore
Acceleration Version 3.1, SDK Quick Start Guide; see http://
publib.boulder.ibm.com/infocenter/systems/scope/syssw/topic/
eicce/eiccesdkquickstart.pdf.
18. IBM Corporation, Cell Broadband Engine Programming
Handbook; see http://www-01.ibm.com/chips/techlib/techlib.nsf/
techdocs/9F820A5FFA3ECE8C8725716A0062585F.
19. IBM Corporation, Cell Broadband Engine SDK Example
Libraries: Example Library API Reference, Version 3.1; see
http://publib.boulder.ibm.com/infocenter/systems/scope/syssw/
topic/eicce/SDK_Example_Library_API_v3.1.pdf.
20. PCI-SIG, PCI Express Base 2.0 Specification Revision 1.1,
March 28, 2005; see http://www.pcisig.com/specifications/
pciexpress/base2/.
21. PCI-SIG, PCI Conventional Specification 3.0 & 2.3: An
Evolution of the Conventional PCI Local Bus Specification,
February 4, 2004; see http://www.pcisig.com/specifications/
conventional/.
22. IBM Corporation, CoreConnect Bus Architecture; see http://
www-01.ibm.com/chips/techlib/techlib.nsf/techdocs/
852569B20050FF7785256991004DB5D9/$file/crcon_pb.pdf.
23. J. Easton, I. Meents, O. Stephan, H. Zisgen, and S. Kato,
‘‘Porting Financial Markets Applications to the Cell
Broadband Engine Architecture,’’ IBM Corporation; see
http://www-03.ibm.com/industries/financialservices/doc/content/
bin/fss_applications_cell_broadband.pdf.
24. IBM Corporation, Moro’s Inversion Example; see http://
publib.boulder.ibm.com/infocenter/systems/scope/syssw/topic/
eiccr/mc/examples/moro_inversion.html.
IBM J. RES. & DEV. VOL. 53 NO. 5 PAPER 2 2009
Received September 22, 2008; accepted for publication
January 14, 2009
Hartmut Penner IBM Systems and Technology Group, IBM
Boeblingen Research and Development GmbH, Schoenaicher Strasse
220, 71032 Boeblingen, Germany (hpenner@de.ibm.com).
Mr. Penner is a Senior Technical Staff Member responsible for the
firmware architecture of Cell/B.E. processor-based systems. He
holds an M.S. degree in computer science from the University of
Kaiserslautern, Germany. During his career at IBM, he has been
involved in many different fields, working on Linux, the GNU
compiler collection, and various firmware stacks of IBM servers.
Utz Bacher IBM Systems and Technology Group, IBM
Boeblingen Research and Development GmbH, Schoenaicher Strasse
220, 71032 Boeblingen, Germany (utz.bacher@de.ibm.com).
Mr. Bacher is an architect for hybrid systems. He earned his B.S.
degree in information technology from the University of
Cooperative Education in Stuttgart, Germany. He developed high-
speed networking code for Linux on System z*. From 2004 to
2007, he led the Linux on Cell/B.E. processor kernel development
groups in Germany, Australia, and Brazil. Since 2007, he has been
responsible for designing the system software structure of future
IBM machines.
Jan Kunigk IBM Systems and Technology Group, IBM
Boeblingen Research and Development GmbH, Schoenaicher Strasse
220, 71032 Boeblingen, Germany (jkunigk@de.ibm.com).
Mr. Kunigk joined IBM in 2005 as a software engineer after
receiving his B.S. degree in applied computer science from the
University of Cooperative Education in Mannheim, Germany.
Prior to his involvement in hybrid systems and the directCell
prototype, he worked on firmware and memory initialization of
various Cell/B.E. processor-based systems.
Christian Rund IBM Systems and Technology Group, IBM
Boeblingen Research and Development GmbH, Schoenaicher Strasse
220, 71032 Boeblingen, Germany (christian.rund@de.ibm.com).
Mr. Rund received his M.S. degree in computer science from the
University of Stuttgart, Germany. He joined IBM in 2001 as a
Research and Development Engineer for the IBM zSeries* Fibre
Channel Protocol channel. He has recently been involved in
firmware and software development for the Cell/B.E. processor
and the directCell prototype hybrid system.
Heiko Joerg Schick IBM Systems and Technology Group,
IBM Boeblingen Research and Development GmbH, Schoenaicher
Strasse 220, 71032 Boeblingen, Germany (schickhj@de.ibm.com).
Mr. Schick earned his M.S. degree in communications and
software engineering at the University of Applied Sciences in
Albstadt-Sigmaringen and graduated in 2004. Currently, he is the
firmware lead of the Quantum Chromodynamics Parallel
Computing on the Cell Broadband Engine (QPACE) project,
which is a supercomputer mission of IBM and European
universities and research centers to do quantum chromodynamics
parallel computing on the Cell Broadband Engine Architecture.
H. PENNER ET AL. 2 : 13