This article has been accepted for inclusion in a future issue of this journal.
Content is final as presented, with the exception of pagination.
IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS
Acceleration of EM-Based 3D CT Reconstruction
Using FPGA
Young-kyu Choi, Student Member, IEEE, and Jason Cong, Fellow, IEEE
AbstractReducing radiation doses is one of the key concerns
in computed tomography (CT) based 3D reconstruction. Although
iterative methods such as the expectation maximization (EM) algo-
rithm can be used to address this issue, applying this algorithm to
practice is difcult due to the long execution time. Our goal is to de-
crease this long execution time to an order of a few minutes, so that
low-dose 3D reconstruction can be performed even in time-crit-
ical events. In this paper we introduce a novel parallel scheme that
takes advantage of numerous block RAMs on eld-programmable
gate arrays (FPGAs). Also, an external memory bandwidth reduc-
tion strategy is presented to reuse both the sinogram and the voxel
intensity. Moreover, a customized processing engine based on the
FPGA is presented to increase overall throughput while reducing
the logic consumption. Finally, a hardware and software ow is
proposed to quickly construct a design for various CT machines.
The complete reconstruction system is implemented on an FPGA-
based server-class node. Experiments on actual patient data show
that a 26.9 speedup can be achieved over a 16-thread multicore
CPU implementation.
Index TermsAccelerator architectures, computed tomog-
raphy, eld-programmable gate arrays, parallel architectures, ray
tracing.
I. INTRODUCTION
C OMPUTED tomography (CT) is a commonly used
methodology that produces 3D images of patients. It
allows doctors to non-invasively diagnose various medical
problems such as tumors, internal bleeding, and complex
fractures. CT has an advantage over other imaging techniques
in that it generally requires a shorter acquisition time than
magnetic resonance imaging (e.g., [1]) and generally provides
better image quality than ultrasound (e.g., [2]). However, high
radiation exposure from CT scans raises serious concerns
about safety. This has triggered the development of low-dose
compressive sensing-based CT algorithms. Instead of tradi-
tional algorithms such as the ltered back projection (FBP) [3],
iterative algorithms such as expectation maximization (EM)
[4], [5] are used to obtain a quality image with considerably
less radiation exposure. Moreover, the iterative algorithm has a
Manuscript received May 21, 2015; revised August 04, 2015; accepted Au-
gust 06, 2015. This work was supported in part by the Center for Domain-Spe-
cic Computing under the National Science Foundation (NSF) Grant CCF-
0926127 and in part by NSF Grant CCF-0903541. It was also supported in part
by C-FAR, one of six centers of STARnet, a Semiconductor Research Corpo-
ration program sponsored by MARCO and DARPA. This paper was recom-
mended by Associate Editor E. Lam.
The authors are with the Computer Science Department, University of Cal-
ifornia, Los Angeles, Los Angeles, CA 90095 USA (e-mail: ykchoi@cs.ucla.
edu; cong@cs.ucla.edu).
Color versions of one or more of the gures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identier 10.1109/TBCAS.2015.2471813
1932-4545 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
1
better noise reduction property and offers better spatial resolu-
tion by using a more natural geometric model than FBP [6].
However, these advantages are achieved at the cost of much
extended computation time [7]. This problem occurs because of
the iterative nature of EM. For example, literature reports 14
minutes and 1.52 hours of CPU computation while processing
64 and 500 projections of 128 128 and 736 64 pixels, re-
spectively [8], [9]. This becomes more problematic with actual
patient images, which require larger input data for accurate mea-
surement. Our testset, which has 15,022 projections of 672 16
pixels, requires 3.3 hours of computation with 16 threads on a
CPU Intel Xeon E5-2420. However, our requirement is to do re-
construction in under two minutes, as part of the low-dose adap-
tive reconstruction ow (Section II-B), so that doctors can give
immediate on-site feedback to the patient. The performance gap
between the multicore CPU result and our goal is approximately
100 times.
In order to reduce the execution time of CT reconstruction,
some systems perform computation on the cloud [10], [11].
However, this may raise concerns about private health infor-
mation being transmitted through the web. Also, the system
reliability may be compromised due to sporadic network
problems. Instead, we envision a supercomputer-in-a-box,
where the computing node is located next to a CT machine and
reliably produces a high-resolution image. To meet the high
computational demand, such nodes can be constructed using a
hardware accelerator such as a graphics processing unit (GPU)
or eld-programmable gate array (FPGA).
Most previous work on GPUs and FPGAs take advantage of
abundant parallel-working cores with efcient work distribution
methods [12][14] and thread divergence avoidance strategies
[15], [16]. However, they are mostly based on FBP. When im-
plementing iterative CT with accelerators, several challenges
still exist.
One challenge is the high data transfer requirement. Previous
research has shown that a bottleneck will be reached where
adding more computing elements no longer improves the per-
formance [9]. The reason is that CT reconstruction is a memory-
bounded application where the performance is dominated by
the system memory bandwidth [9], [17]. For example, an im-
plementation based on [9] transfers 974 GB of data per itera-
tion to DRAM for real patient image reconstruction. To solve
this bandwidth problem, previous GPU and FPGA acceleration
papers proposed bit-width optimization [8], [18], locality im-
provement [17], [19], prefetching and masking [9], and data
movement reduction [20]. However, the improvement was lim-
ited to 1.3 3.1 . To achieve our two-minute goal, a more
radical algorithmic and architectural improvement is needed.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
2 IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS
Fig. 1. CT scan trajectory. The parameters are from M.125000 dataset. Modied from [23].
One potential for a large increase in the system bandwidth
can be found in the FPGA architecture. Although the FPGA
platform has a slower clock frequency (100 MHz versus
5001000 MHz) and less DDR bandwidth than the GPU, it can
customize the computation and the data communication. For
example, it contains many block RAM (BRAM) that can be
used to perform a large amount of local data transfer. However,
BRAM usage alone is not very effective unless DRAM can
supply the data to BRAM without stalling.
Another important challenge is to achieve a high degree of
parallelization while efciently resolving access conict to local
memory. Problems such as race condition and bank conict
occur when several rays try to update a voxel at the same time.
Unlike most other FBP-based acceleration work that achieves
high parallelism with point-like voxel, our work incorporates
volumetric voxel for a realistic geometric model. However, this
complicates the design with serialization and non-stencil access
pattern. Chen et al. proposed parallelizing rays that are suf-
ciently apart [22], but the large distance limits the amount of
parallelism. A new approach is needed that can avoid the ac-
cess conict while still achieving large parallelism with min-
imal overhead.
In this paper we present a complete and working CT re-
construction system implemented on a server-class node with
FPGA coprocessors. It incorporates several FPGA-friendly
techniques for acceleration. The contributions of this paper
include:
Ray-driven voxel-tile parallel approach: This approach
exploits the computational simplicity of the ray-driven ap-
proach, while taking advantage of both ray and voxel data
reuse. Both the race condition and the bank conict prob-
lems are completely removed. Moreover, it can easily in-
crease the degree of parallelism with adjustment of tile
shape. A strategy for the tile shape decision is also pre-
sented.
Variable throughput matching optimization: We pro-
vide strategies to increase the performance for designs
that have a variable and disproportionate throughput
rate between processing elements (PE). In particular,
the logic consumption is reduced by exploiting the low
complexity of frequently computed parts of the ray-driven
approach. The logic usage is further reduced by using
small granularity PEs and module reuse. This leads to
overall throughput improvement with module duplication.
Ofine memory analysis for irregular access patterns
in the ray-driven approach: To efciently tile the voxel
for irregular memory access, an ofine memory analysis
technique is proposed. It exploits the input data indepen-
dence property of the CT machine. Also, a compact storage
format is presented.
Customized PE architecture: We present a design that
achieves high throughput and logic reuse for each PE.
Design ow for rapid hardware-software design: A ow
is presented to generate hardware and software for various
CT parameters.
This work is an extension of our previous conference publica-
tion in [23]. The previous work mainly focused on exploiting the
data reuse to reduce off-chip access. In this work we added the
parallelization scheme, ofine memory analysis technique, vari-
able throughput optimization, and the automated design ow.
Due to the new optimization, this work is 5.2 times faster than
[23] on the same dataset.
The remainder of this paper is organized as follows: Section II
will provide the background and analysis of the algorithm and
the platform. It also provides a brief analysis of the problem and
highlights the challenges for constructing an efcient design.
Section III explains the ray-driven, voxel-tile parallel approach
and the ofine memory analysis technique. The design ow, de-
tails for the customized PE design, and throughput optimization
strategies are provided in Section IV. The experimental results
can be found in Section V.
II. BACKGROUND AND PROBLEM ANALYSIS
A. CT Scanner Geometry and Dataset Description
In a CT scan, as shown in Fig. 1, radiation from the source
penetrates the patients body and gets measured by an array of
detectors. The source and the detector array proceed in a helical
fashion, where the pair move in a circular direction in the -
plane and in a lateral direction along the -axis.
In our clinical setting, the CT scan was acquired using a
helical cone beam protocol on a Siemens Somatom Sensa-
tion 16 scanner. The focus-detector distance (FDD) and the
focus-isocenter distance (FID) (both shown in Fig. 1) are
1,040 mm and 570 mm, respectively. The circular rotation rate
( ) of source and detector array is 1,160 samples per rotation.
The lateral movement per sample ( ) of source and detector
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

CHOI AND CONG: ACCELERATION OF EM-BASED 3D CT RECONSTRUCTION USING FPGA 3

is 4.8 . The detector array consists of 672 channels and 16

rows.
For the dataset, we use CT images (labeled M.125000) ac-
quired from an actual patient at our institution using a protocol
approved by the institutions review board (IRB). To simulate a
reduced radiation dose, we used 15,022 samples, which is 25%
of the originally obtained data. The size of the reconstructed
volume is 512 (Nx) by 512 (Ny) by 372 (Nz) voxels. The size
of a voxel is 0.77 mm ( ) in each dimension.

B. Adaptive Flow for Lung Cancer Screening

Our aim is to construct an automatic lung cancer screening
system. First, the patient goes through a low-dose CT scan. Then
the result is converted into 3D images using a reconstruction al-
gorithm. Next, reconstructed images are processed by our com-
puter-aided cancer nodule detection algorithm [24], [25]. If a
suspicious nodule is found, a second full-dose CT scan is se-
lectively performed to obtain a high-quality image around the
target image slices.
Even with 25% low-dose scans, experiments show that the
nodule detection sensitivity is maintained at 100% using the
compressive sensing techniques in [26]. This leads to signicant
dose reduction for patients, with or without cancer. For example,
for a patient that has a suspicious cancer nodule that spans across Fig. 2. EM-based iterative reconstruction process, assuming that the ray tracing
37 slices (2.8 cm), the dose reduction is approximately 65% method has been used for the forward and the backward projection.
[ ].
In this paper we concentrate on acceleration of the 3D recon- TABLE I
struction, which is the most time-consuming part of our cancer PROFILING RESULT USING 16 THREADS ON INTEL XEON E5-2420
screening system.

C. 3D Reconstruction Algorithm and Problem Analysis

1) Expectation Maximization (EM) for 3D Reconstruction:
The objective of 3D CT reconstruction is to recover the object
image intensity from
which is the measurement obtained from the CT scan. The
problem can be described as solving the linear equation
where is the matrix that contains how much each voxel of an
object will be projected onto the detector array.
EM is an iterative method of solving the above problem by
progressively rening the object by nding the most likely in-
tensity, given CT scan observation. The overall algorithm is
shown in Fig. 2. The rst step of EM is the forward projection,
which obtains the projection of the estimated image. Using the
notation in [26], this can be described as: , where
is the image intensity in the th-iteration, and is the projec-
tion result which is also called the sinogram. After the sinogram
is compared with the input observation ( ), it is back-pro-
jected onto the image in the backward projection step:
, where is an element of , and and
are the indices in the sinogram and image domains, respec-
tively. Finally, the previous estimate of the image intensity
is updated to using the equation
, where is the normalization constant. These opera-
tions are repeatedly performed until the estimated object image
, converges.
After these EM steps, total variation (TV) regularization [21]
is performed on the image to remove noise and blurring. The
reader is referred to [26] for details of this process.
The proling result using 16 threads on Intel Xeon E5-2420 is
shown in Table I. It suggests that the forward projection and the
backward projection should be the focus of FPGA acceleration.
We have decided to accelerate only EM and exclude TV, since
the portion of TV is very small.
2) Forward and Backward Projection: The computation pat-
tern for forward and backward projection is described in Fig. 3.
Although the equation looks simple, the challenge lies in ef-
ciently nding the intersection length of a ray onto a voxel ( ).
It is difcult to precompute and store to memory, because it
requires 243 GB even in a sparse matrix form. Such large data
is needed because of the large number of rays (15, 022 672
16) and the image (512 512 372).
Rather than storing a matrix, two types of efcient methods
are commonly used for forward and backward projections: the
ray-driven [27], [28] and voxel-driven [29], [30] approaches.
(1) The ray-driven approach computes the integral
along the line of all rays. is the intersection length, and
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

4 IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS

is the image intensity. The voxel-driven approach takes the

opposite direction, by updating all sinogram that are projected
from a particular voxel.
Since the radiation from the source spreads out to the de-
tectors in a cone-beam shape (Fig. 1), several rays may try to
access the same voxels near the source. This does not become
a problem when using the ray-driven approach in the forward
projection, because voxel intensity is read-only. In the back-
ward projection, however, race conditions may exist because
the voxel intensity will be read and written. Note that the voxel-
driven approach has opposite characteristics, i.e., access conict
may exist in the forward projection because several voxels may
Fig. 3. Memory access and computation pattern of forward and backward
update the same sinogram. But conict does not happen in the projection.
backward projection, because each voxel only reads from sev-
eral sinogram.
In terms of computational complexity, frequently used com-
putation in the ray-driven approach is much simpler. As can
be seen in Fig. 3 and Algorithm 1, the voxel position and the
intersection length can be implemented with a few comparators
and adders. For the voxel-driven approach, the weights ( )
of detectors projected from an image voxel are computed
by nding the ratio between the source-to-voxel and the
source-to-detector distance [29], [30]. This magnication factor
computation requires frequent division and multiplication. This
can be simplied by using a look-up table [30], but it requires
additional memory access. More complex computation leads to
Fig. 4. Computation for ray geometry data, intersection length, and voxel
larger FPGA logic resource consumption. This has a negative coordinate. The pseudocode is given in Algorithm 1 and Algorithm 2. Modied
impact on performance, since module duplication is heavily from [27].
used to increase the throughput in our design (Section IV-D).
Thus, we have decided to use the ray-driven approach.
3) Review of Ray-Driven Approach: The computational traversing through a 2D grid of voxels, as in Fig. 4. of the cur-
kernel of our algorithm is derived from the ray-driven approach rently traversed voxel (shaded grey) and of the next traversed
proposed by Zhao and Reader [27]. Their method, also referred voxel can be obtained using the update equation in Algorithm
to as the ray tracing algorithm, is an enhanced variant of the is the distance of a ray that has been traversed so
widely used Siddons algorithm [28]. far, and and are the distance of a ray that will be
Algorithm 1: Intersection length and coordinate com- traversed if the ray next passes through the x-grid or y-grid, re-
putation for ray tracing. spectively. and are the ray distance between two voxels
in each dimension. This equation can be extended to the 3D case
by adding direction to this equation.
Algorithm 2: Ray geometry data computation. This has
been shortened for brevity. The complete computation and its
meaning can be found in [27].

The main advantage of this algorithm is the on-the-y con-

struction of the intersection length and the position of a ray by
exploiting the repetitive nature of the traversal. Consider a ray
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
CHOI AND CONG: ACCELERATION OF EM-BASED 3D CT RECONSTRUCTION USING FPGA
The ray geometry computation produces a set of parameters
needed for ray tracing: , the initial value in
each dimension , the initial value
, and initial voxel coordinate . Compared
to the ray tracing, this requires more resources because it im-
plements complex functions shown in Algorithm 2. The source
and detector positions and can be obtained by multiplying
their distance from the center (FID, FDD) by the rotation rate
( ), added by the -direction movement ( ). This requires four
sine/cosine units and four multipliers for implementation. Then,
at the image boundary, and , is computed. After
comparison, is obtained. Next, and can be ob-
tained as shown in Algorithm 2. After resource sharing, this re-
quires one oating point division unit and about 15 additional
multipliers. Note that although it is expensive in terms of re-
source consumption, the overall portion becomes small by ex-
ploiting the difference in the module throughput, as will be ex-
plained in Section IV-D.
The disadvantage of this algorithm is that it serializes the ray
tracing computation, which interferes with achieving massive
parallelism for GPU and FPGA implementation. Also, the voxel
coordinate changes irregularly in all , , directions which
causes frequent memory bank conict. Moreover, this algorithm
has non-stencil access patterns which makes it hard for com-
pilers to analyze.
D. Convey HC-1ex Platform
This work has been implemented and veried using the
Convey HC-1ex platform. The platform includes an Intel Xeon
5138 host CPU and four Xilinx Virtex-6 LX760 FPGAs [31].
This platform was selected because of the high coprocessor
memory bandwidth and efcient scatter-gather memory access
capability [32] of the coprocessor memory controller. Also,
cache-coherent shared memory space eases development. The
coprocessor memory has 16 dual in-line modules. With the
memory controller running at 300 MHz, it has a theoretical
coprocessor memory bandwidth of 80 GB/s. Each FPGA pro-
vides 16 FIFO-based DRAM channels, and these channels are
connected to the DRAM in a full crossbar.
III. RAY-DRIVEN VOXEL-TILE PARALLELISM
A. Tile-Based Parallelism for FPGA
Our initial experiment demonstrated that an implementation
similar to [9] transfers 974 GB of data per iteration for our test
dataset. This puts heavy pressure on the DRAM bandwidth. A
solution to this problem can be found in the abundant BRAMs
present in the FPGA. We can take advantage of the large internal
bandwidth provided by independently working 5,760 BRAM
modules in four FPGAs.
To use BRAM efciently, a number of challenges must be
considered. The rst problem is to decide which data is stored
in the BRAM. In our design, voxel intensity is stored in the in-
ternal BRAM. This is because of the inherent data reuse rate
in the ray-driven approach. When each ray passes through mul-
tiple voxels, the intermediate accumulated sinogram (forward
projection) or sinogram for updating (backward projection) can
be stored as a local variable and referenced without accessing
5
Fig. 5. Proposed ray-driven voxel-tile parallel approach. Each ray tracing PE
will have a tile of voxels in its BRAM, and process one ray at a time.
the external storage. Since each ray traverses through an average
of 1,004 voxels in our dataset, the data access ratio between the
voxel intensity and the ray data (ray geometry parameter and
sinogram) is 1,004 to 1. Since this means that voxels require
higher bandwidth, they are stored in the BRAM.
The second aspect that should be considered is the data reuse
rate. If BRAM accesses the DRAM frequently, the DRAM
bandwidth will become the bottleneck. As a result, BRAM will
often stall waiting for data to arrive. This suggests that the data
reuse rate in BRAM should be high. This can be achieved by
grouping a contiguous set of voxels. Such voxel tiling benets
from the ray traversal pattern; since a ray is likely to traverse
through nearby voxels, a tile of nearby voxels can share the
data from a similar set of rays. Thus, the ray data reuse rate
is increased. Our previous work showed that the voxel reuse
rate can be up to 831 [23]. Since the sinogram reuse was
achieved with a ray-driven approach, our algorithm can exploit
both types of reuse. This leads to a large reduction in off-chip
memory transfer.
Last, we should be able to extract enough parallelism to take
advantage of available BRAMs. As mentioned in Section II-C2,
a race condition and bank conict may occur when several rays
try to update the same voxel. This becomes more problematic
with Zhao and Readers algorithm [27] since it introduces se-
rialization into ray tracing. For the race condition problem, an
atomic operation can be used to update the voxels, but it will de-
grade the performance signicantly. Chen et al. proposes paral-
lelizing sufciently spaced rays [22], but it will limit the amount
of parallelism and may cause frequent memory bank conict.
As a solution to the above problems, we propose a new paral-
lelization scheme which can be classied as a ray-driven voxel-
tile parallel approach. Each PE will have access to a BRAM
that stores a tile of voxels. One PE will be able to perform just
one ray tracing at a time, but instead, multiple PEs work on
different tiles of voxels to obtain high parallelism, as shown
in Fig. 5. After processing all rays in a tile, the PE will pro-
ceed to the next voxel tile. This approach can also be viewed
as exploiting the parallelism by dividing a ray with multiple
segments and performing ray-tracing starting in each voxel tile.
Parallelizing voxels while still using a ray-driven approach may
seem counter-intuitive at rst, but this is designed to exploit both
the low computational complexity of a ray tracing method and
also the high BRAM system bandwidth of voxel tiling. Note that
the voxel tiling idea itself was proposed previously with FBP in
GPU [12], [13], but it was not in the context of EM, nor was it
combined with ray tracing as our scheme.
The advantage of the proposed parallel approach is that we
can easily adjust the parallel factor by changing the size of the
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
6 IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS
voxel tile (decision process will be discussed in Sections III-C
and IV-D). Another advantage is that a race condition cannot
occur between rays since each PE computes one ray at a time.
Moreover, bank conict is removed since each PE has dedicated
BRAM access.
The proposed method, however, creates several new prob-
lems. First, the irregular access pattern of ray tracing makes it
difcult to analyze which rays access the voxel tile. Second, ray
data has to be fetched several times. Third, access conicts exist
in the ray sinogram. The solutions to these problems are dis-
cussed in the following subsections.
Note that it is also possible to reuse rays in the voxel-driven
approach, as was presented in [33]. In their design, parallelism
is achieved by using multiple transaxial projection modules.
B. Offline Analysis for Irregular Memory Access
One of the most complex problems of the voxel-tiled-parallel
scheme is to nd the rays that pass through a certain tile. A
non-stencil, irregular access pattern of ray tracing makes it dif-
cult for compilers to analyze the memory access pattern. An
exhaustive search also cannot be used, since very few among
all rays access a tile; for example, only 0.03% of rays traverse
through a tile of size 64 64 1. It is possible to project a voxel
to the sinogram and nd the rays by computation [15], but this
consumes extra logic resources.
As a solution, we propose making an ofine, proling-based
approach. A software-based memory access analyzer will run
ofine and record the list of rays that access certain tiles. This is
possible because of a special property of CT reconstruction. In
a typical CT usage scenario, the CT patient will change, which
will cause the input dataset to vary. The memory access pattern
of ray traversal, on the other hand, will typically not change,
since it is only dependent on the CT machine parameters. In
other words, the memory access pattern of the CT machine is
independent of input data it is the same across multiple pa-
tients. This allows us to perform an ofine analysis and devise
an optimized design.
However, simply storing the list of individual rays per tile is
not very efcient, since it requires 5.6 GB. Instead, we again
take advantage of the CT machine characteristics. Since each
radiation source has rays spreading out in a cone-beam shape
(Fig. 1), it is likely that nearby rays will access the same voxel
tile. This allows the list of rays to be compacted by storing only a
continuous range of rays. Also, since a voxel tile is a rectangular
shape, projecting the voxel tile back to the detector array will
produce a list of detectors that is close to a rectangular shape.
Thus, it becomes possible to approximate the ray list with a rect-
angular bounding box. This is expressed as (highest/lowest de-
tector channel #, highest/lowest row #), as shown in Fig. 6. The
ray bounding box expression is stored per each projection per
each voxel tile. The proposed method of generating the list of
rays requires 58 MB of memory access, which is a large im-
provement over the original list.
Note that a method for representing the ray traversal path in
the voxel domain has been proposed in [33]. A clever run-length
encoding is used to reduce the size of the representation.
Fig. 6. Storage format that approximates the detector channel and row for each
projection.
TABLE II
AVERAGE RAY ACESSES PER VOXEL WITH VARYING GROUP OF RAYS
C. Tile Shape Decision
As long as enough parallelism is guaranteed, the voxel tile
should be made as large as possible. The reason is that the sino-
gram has to be accessed again when it traverses to a different
voxel tile. However, the limitation on the size of the BRAM
(12.7 MB) forces a decision on the shape and size of the data
stored in the internal memory.
The shape of the tile should be decided by the trajectory of
the CT scan. Table II shows the average number of accessed
rays that traverse through a voxel. Among a group of rays in
the 672 channels 16 detector rows (single projection), each
voxel is accessed only 2.3 times on average. The reason for such
a small number of accesses is that radiation emission spreads
out in a cone shape (Fig. 1), and very few rays access the same
voxel when far from the source. On the other hand, there is al-
most a linear increase in the average number of accessed rays
( ) when the ray group is expanded to con-
secutive projections ( ). The reason is that detector
and source movement along the -axis ( ) is slow compared
to the circular rotation rate ( ). Thus, it is likely that rays of
nearby projections intersect almost identical - planes. The
difference between the rays of nearby projections is in the pen-
etration angle, which only changes the order of voxel access
but not the actual list of voxels accessed. This suggests that, for
voxels, it is advantageous to use a at x-y surface tile rather than
a cube tile to maximize the reuse between adjacent rays.
In addition to the available BRAM size, the voxel tile size is
also affected by the available logic resources and the external
memory bandwidth. As will be discussed in Section IV-D, we
can place 256 ray tracing PEs into the platform. Assuming about
4.5 MB (35% of total BRAM resource) has been assigned for
ray tracing, we can use a voxel tile size of 64 64 1, which
is the most at shape conguration.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

CHOI AND CONG: ACCELERATION OF EM-BASED 3D CT RECONSTRUCTION USING FPGA 7

TABLE III TABLE IV

BRAM SIZE AND DRAM TRANSFER WITH VARIOUS TILE SIZES COMPARISON OF DRAM DATA TRANSACTION (PER ITERATION)

The relationship between the required external memory band-

width and the BRAM size is shown in Table III. Since voxels
are accessed once, only
of memory transfer is used for the voxel, and the rest is for the
sinogram. The results in the table conrm that bandwidth re-
quirement is low when a large amount of internal memory is
available. Also, it shows that when the BRAM size is xed, a
at-shape tile (e.g., 128 128 1) has more DRAM transfer
savings than a cube-like tile (e.g., 64 64 4).

D. Synchronization

Although voxel access conict was removed by adopting
voxel tile parallelism, sinogram access conict has been newly
introduced in the forward projection. To solve this problem,
we decided to use the barrier synchronization method. Each
PE will update its sinogram value to a memory location that is
different from other PEs. This does not mean memory location
will be increased by the number of available PEs, because a
ray traverses through a limited number of voxel tiles. After
performing an ofine analysis of nding all voxel tiles that rays
pass into, adjacent voxel tiles are assigned to the same PE to
reduce the number of PEs that might access the same sinogram.
Experiments show that the DRAM memory allocation size for
sinogram increases by eight times for our dataset. This does not
have a severe negative impact on the performance because the
amount of data transfer remains the same for the ray tracing PE.
E. Comparison
The DRAM data transaction comparison between a design
that only reuses ray data (e.g., [9]) and the proposed design
is presented in Table IV. Reusing only the sinogram requires
large off-chip memory bandwidth to transfer voxel intensity. In
the proposed scheme, the voxels in a tile are transferred only
once, because they are reused among several rays. At the same
time, ray data is still being reused, because ray tracing is per-
formed within a tile. Although the sinogram data access has in-
creased 8.6 times for a 64 64 1 tile size, this has little effect
on overall data transfer, since its portion was small compared to
the voxel transfer. Considering the combined value of sinogram
and intensity, there is a 55.3 reduction in external memory ac-
cess.
Table IV also shows that there is little difference for a
combined amount of BRAM and DRAM access. However, the
performance drastically improves, because FPGA has a much
higher BRAM bandwidth than DRAM.
Fig. 7. Software and hardware design ow.
IV. DESIGN FLOW AND ARCHITECTURE
A. Overall Design Flow
Our design is based on an assumption that even though the pa-
tient changes, the CT scan parameter will not vary much. This
allows engineers to optimize the design ofine (Section III-B).
A possible usage scenario is that the CT machine will provide
some limited number of FPGA-acceleration settings that are
most frequently used. Radiologists can choose one of these set-
tings and quickly obtain the reconstructed result.
The overall design ow is shown in Fig. 7. The software ana-
lyzer will record the memory access pattern into a le and pro-
vide guidelines on the hardware architecture. The hardware de-
sign, written in C, is then fed into the Vivado HLS 2013.1 tool to
generate an RTL design. The RTL les are then combined with
the host and the DRAM interface RTL. Next, the bitstream is
created using the Convey development tool chain that is based
on the Xilinx ISE tool. The host software performs necessary
pre- and post-processing such as reading the ray list informa-
tion or reading and writing the image le.
B. Software Analyzer
The software analyzer is responsible for generating two types
of information: the list of rays per voxel tile and the hardware
design parameters. The range of rays is easily obtained by mea-
suring the minimum and the maximum of radiation detectors
per projection for each voxel tile (Section III-B). Hardware de-
sign parameters include the tile size, shape, and the module
duplication information. The reason this has to be extracted is
because the memory access pattern will change as the CT pa-
rameter changes. These parameters are decided based on the
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
8 IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS
Fig. 8. Mapping between the algorithm and the architecture for the proposed system.
measurement of throughput of ray tracing PE, number of rays
passing into a voxel tile, and the number of DRAM accesses
(e.g., Table III), as will be explained in more detail in the next
subsection. Note that the analyzer has been implemented in soft-
ware because it is easier to make modications, and also because
the execution time is not very constrained.
C. Accelerator Architecture
1) Overall Architecture: The overall architecture and the
mapping from the algorithm are shown in Fig. 8. It shows three
types of customized PEs. Ray tracing of forward and backward
projection is mapped to the ray tracing PE. Ray geometry PE
generates the parameters needed for ray tracing. The sinogram
comparison and the voxel update part are mapped to the com-
parison/update PE.
General-purpose modules have been extracted from the cus-
tomized PEs. In Fig. 8, C represents the loop iterator generator
for both tile and the rays. D represents the DRAM access unit.
If the number of PEs do not match, a network logic, denoted
N, is used to either gather or distribute the data. It employs
round-robin scheduling to allocate similar transfer rates to the
receiving PEs.
2) Accelerator Design Principles: We employed several
principles for high-performance accelerator design. The rst
principle is to divide PEs as nely as possible for aggressive
pipelining. For example, even small modules such as the
data distribution logic, loop iterator generator, and DRAM
access logic were detached from ray tracing and ray geom-
etry computation PEs. This allows us take advantage of the
different throughput rate of PEs, because logic area can be
saved by having multiple low-throughput PEs share access to
high-throughput PEs. This leads to throughput optimization
using module duplication, as will be explained in Section IV-D.
The second principle is module reuse. If multiple functions
can be mapped to the same PE, it can save area and enable more
module duplication. For example, ray tracings from both for-
ward and backward projections are mapped to the same PE.
The third principle is to use FIFO logic for inter-module
communication. Since many modules are designed as
high-throughput streaming modules, FIFO simplies the data
transfer process.
Finally, we concentrated on pipelining different tasks. Overall
execution time can be reduced, not only by enumerating one
type of PE, but also by overlapping the computation of different
types of PEs. The advantage is that it allows local communica-
tion between directly connected PEs and removes data storage
time to memory. Note that such task parallelism is hard to be
used efciently with GPU and multicore CPU, and gives an in-
herent advantage to FPGA-based approaches.
3) Ray Tracing PE: Ray tracing PE is responsible for ac-
cumulating (forward projection) or updating (backward projec-
tion) the intensity of voxels, as shown in Fig. 3. Due to the sim-
ilarity in their computation patterns, they can be combined into
one PE, as shown in Fig. 9. An operation control bit is used to
select either the forward or the backward projection mode.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
CHOI AND CONG: ACCELERATION OF EM-BASED 3D CT RECONSTRUCTION USING FPGA
Fig. 9. Ray tracing architecture that supports both forward and backward
projection. The pseudocode for the intersection length and the voxel coordinate
generator can be found in Algorithm 1.
The computation unit is tightly coupled with the internal
BRAM for high bandwidth. To achieve a throughput of one
voxel update per cycle, each updated voxel should be written
to BRAM while the next voxel is being read. This requires a
two-port BRAM. Note that there is no race condition due to
reading and writing to the same address in BRAM, because
a ray is guaranteed to move one coordinate in each cycle.
Also, the external DRAM channel is shared among several ray
tracing PEs, because the PEs BRAM does not need to access
the off-chip memory frequently (Section III-A).
Even though PEs share the same DRAM channel, they were
designed to work independently of other PEs. A different design
choice could be to share the ray parameters between nearby PEs
by interconnecting PEs in a grid structure. The advantage of this
scheme is that the sinogram and the ray parameter can be reused
and passed to the adjacent PEs. However, an implementation of
this design showed that the overall performance was degraded
by about 2 compared to the independent version. The reason
is that although DRAM bandwidth had been reduced, each PE
had to wait for other PEs to pass the rays. This increased the
idling cycles of PEs. Thus, PEs were made to work indepen-
dently to increase the utilization ratio and obtain better overall
performance.
4) Ray Geometry Parameter PE: The ray geometry pa-
rameter PE reads the tile and ray range information from the
DRAM (Section III-B). Next, in parallel with the high-latency
parameter computation in Algorithm 2, the sinogram of the ray
is prefetched from the DRAM. This information is fused with
the ray geometry parameters ( , , , is based on our experience that the Xilinx placement and route
) and passed to the ray tracing PE.
With help from the Xilinx Vivado high-level synthesis (HLS)
tool, this module was redesigned into a pipelined version that
has a throughput of one ray per cycle. Since the tool only sup-
ported an initiation interval (II) of 4 for certain oating-point
operations, we converted the whole design into a xed-point
design. The word length of the variables are shown in Table V.
Note that the xed-point conversion also helps reduce the logic
consumption.
Another challenge was the large logic consumption for the
sine and cosine operations. To solve this problem, we imple-
mented a xed-point table lookup based cosine/sine design.
Using the symmetry of cosine/sine value, angle folding with
was performed to reduce the table size.
9
TABLE V
WORD LENGTH OF VARIABLES ( )
5) Comparison & Update PE: The comparison computation
( ) and the update computation (
(1)) have a similar form, and can be combined into the same
PE to share the expensive division operator. Since both opera-
tors are completely parallel, all data are consecutively read and
written to the DRAM in a streaming fashion. The comparison
computation processes 4.8 GB [ 1 4
B] of sinogram data, and the update computation processes 0.36
GB [ ] of voxel data per iteration.
Since the amount of transferred data is small compared to the
ray tracing and ray geometry computation, we only placed 16
comparison/update PEs.
D. Variable Throughput Optimization With Module
Duplication
Since the ray parameter is reused during the ray traversal,
there is a large throughput difference between the ray tracing
PE and the ray geometry parameter PE. For the ray tracing PE,
the throughput depends on the number of voxels traversed in
a tile: for a 64 64 1 tile, II can be as little as 1 cycle, or as
large as . The ray geometry PE, on the other
hand, can generate one ray data per cycle. Such disproportionate
and variable PE throughput presents a challenge for optimized
design. Note that the presented problem is different from most
previous work on throughput and module duplication (e.g., [34],
[35]), which assume a xed data consumption ratio between
producer and consumer PEs.
To compensate for the difference, ray tracing PEs are fur-
ther duplicated. The duplication factor of the PEs can be deter-
mined by a resource-constrained optimization problem where
the execution time is minimized subject to a lookup table (LUT),
ip-op (FF), BRAM, multiplier (DSP), and BRAM/DRAM
bandwidth.
The resource constraints are shown in Table VI. We set the
available resource as 40% of total LUT and FF, and 70% of
total DSPs and BRAMs of four Virtex-6 LX760 FPGAs. This
tool will have difculty if the resource consumption rate is too
high. Moreover, the Convey DRAM controller occupies logic
resources. The bandwidth of DRAM is set to 40 GB/s, which is
the practical upper limit of DRAM bandwidth obtained in the
test of the Convey memory controller system.
Next, the resource consumption of PEs needs to be measured.
The logic resource consumption is easily obtained from Vivado
HLS, as shown in Table VI. The problem is in the throughput
measurement, which varies for each ray. To solve this problem,
rst the analyzer software collected statistics on the average
throughput rate of a ray tracing PE, which was found to be
58.8 cycles per ray. Next, we attached a large FIFO to each ray
1Memory allocation overhead for synchronization (Section III-D).
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
10
TABLE VI
LOGIC/MEMORY CONSTRAINT & CONSUMPTION PER PE
tracing PE. The effect is that of averaging the diverse throughput
to the expected value.
The average ratio between the ray throughput of ray geometry
PE and that of ray tracing PE is . To
have some safe margins caused by the varying throughput, we
decreased the ratio to 1 : 16. Based on Table VI, it becomes
possible to place 17 ray geometry parameter PEs and 264 ray
tracing PEs. For regularity of the design, we decided to place
16 ray geometry parameter PEs and 256 ray tracing PEs.
As a result, even though a single ray geometry PE consumes
6 times more LUT than a single ray tracing PE, it occupies 37%
less LUT after duplication. This is because the duplication factor
of the ray geometry PE is small. It can be observed that the logic
saved from sharing high-throughput PEs has increased perfor-
mance because it allowed more duplication of low-throughput
PEs.
Our proposed optimization shows how the performance
can be improved if more frequently occurring computation
has lower complexity. The ray tracing PE has frequent and
computation (Algorithm 1), which only requires addition.
Thus, it can be heavily duplicated. The ray geometry parameter
PE has more complex computation and thus consumes more
logic, but less duplication is needed because only one compu-
tation is performed per ray in a voxel tile. Thus, it conrms
that with the proposed optimization, it is possible to improve
the performance of the ray-driven approach by exploiting the
simple computation pattern of the ray traversal.
V. EXPERIMENTAL RESULTS
A. Performance
1) Performance Breakdown: The performance breakdown is
shown in Table VII. Each iteration of reconstruction takes 8.9
seconds; thus it takes 7.4 minutes to complete the reconstruction
with 50 iterations. We also present the rate of voxel updates as
Giga updates per seconds (GUPS) for easier performance com-
parison. The performance was measured on an actual execution
using the Convey HC-1ex platform.
Forward projection takes an amount of time similar to back-
ward projection although it requires about 2 of DRAM band-
width (Table IV). This suggests that the DRAM bandwidth is
no longer the bottleneck.
The reason that the compare and update part has a small
GUPS is because only a quarter of Conveys DRAM channel
was utilized to reduce the number of PEs to 16. It has little neg-
ative impact on the overall performance because the overall por-
tion is still relatively small.
Note that if TV is added to the system, it will take 1.0 seconds
for computation, per iteration, assuming that 16 threads on Xeon
E5-2420 are used. Also, an additional 0.8 seconds per iteration is
IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS
TABLE VII
BREAKDOWN OF EXECUTION TIME
TABLE VIII
EXECUTION TIME COMPARISON WITH VARIOUS ACCELERATOR PLATFORMS
THAT EXPLOITS THE RAY PARALLELISM (UNIT: MINUTES, 50 ITERATIONS)
needed to send 372 MB of image back and forth between FPGA
and CPU through the front-side bus.
2) Effect of the Proposed Parallel Approach: To nd the ef-
fect of the proposed parallel approach, we implemented the re-
construction algorithm using three different accelerators-mul-
ticore CPU, GPU, and FPGA. These versions exploit the ray
parallelism but not the proposed voxel data reuse. All versions
use the same CT parameter and the same input data as the pro-
posed FPGA version. For multicore CPU, we used the Xeon
E5-2420 which is based on a newer process technology than
the FPGA platform. We used 16 threads. For GPU, three Tesla
M2070 GPUs were used to approximately match the number of
Virtex-6 FPGAs used. A total of 9216 threads were used. For
FPGA, we used the same Convey platform but incorporated op-
timization in [9] for comparison. Note that the CPU and FPGA
versions keep enough distance between the threads to resolve
access conict, as suggested by [22], but the GPU version uses
atomic operations to avoid a race condition among thousands of
threads.
Table VIII shows that the proposed algorithm on FPGA has
26.9 , 16.8 , and 4.6 speedup over the ray-parallel imple-
mentation on multicore CPU, FPGA, and GPU, respectively.
The performance gain is considerable. Even though the Convey
platform has smaller DRAM bandwidth and a 28 and 5.8
slower clock frequency than the CPU and GPU platforms, it has
higher performance because of the reduced external memory
transfer with data reuse, task pipelining, and the customized cir-
cuitry.
One possible question could be whether other accelerators
can also benet from the proposed techniques. Somewhat
counter-intuitively, experiments showed that, compared to
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
CHOI AND CONG: ACCELERATION OF EM-BASED 3D CT RECONSTRUCTION USING FPGA
TABLE IX
SCALABILITY TEST AND PROJECTION OF EXECUTION TIME ON CONVEY HC-1EX PLATFORM
TABLE X
TEST RESULT USING VARIOUS CT PARAMETERS
Table VIII, the proposed scheme degrades the performance
in multicore CPU and GPU. The difference can be found in
FPGAs ability to use task parallelism and make local data trans-
fers between PEs. As mentioned in Section II-C3, ray geometry
parameter, which involves complex operations like cosine and
division, has to be recomputed each time a ray traverses through
different voxel tiles. This increases the computational burden
in CPU and GPU. On the other hand, the FPGA can efciently
compute these parameters in parallel with the ray tracing. This
suggests that, combined with an efcient data transfer scheme
and a high degree of task parallelism, FPGA may outperform
the GPU implementation despite having a disadvantage in the
clock frequency and the DRAM bandwidth.
3) Scalability Test: Table IX shows the measured execution
time with 1, 2, and 4 of Virtex-6 LX760 FPGAs. It shows that the
performance almost linearly scales with the resource used. This
is because the throughput almost scales with the number of PEs
used, provided that the system can satisfy the required DRAM
bandwidth. Since Convey HC-1ex has an effective bandwidth of
about 40 GB/s, we estimate that it can support up to about 3,500
ray tracing PEs. The table also contains the estimated result as-
suming the Convey HC-1ex platform has been updated with the
latest Xilinx Virtex-7 2000T or Ultrascale VU13P FPGAs [31],
which contain 2.6 and 3.4 more LUTs than LX760 FPGAs,
respectively. It shows that the two-minute goal can be achieved
if FPGAs are replaced with these latest products.
4) Performance With Various CT Parameters: The design
ow was validated using various CT scan parameters, as shown
in Table X. In addition to the M.125000 image that was mainly
used for experiment, we tested another patient image labeled
0960B with 25% (5,865) projections to 100% (23,464) projec-
tions. Note that the ofine analyzer has to be executed once for
each setting since the memory access pattern has changed. The
GUPS gure shows that the performance is maintained in other
settings as well.
B. Resource and Power Consumption
The resource consumption is shown in Table XI. Placing mul-
tiple copies of PEs led to about a 60% utilization ratio of LUTs.
11
TABLE XI
RESOURCE CONSUMPTION (4 FPGAS)
This complicated placement and routing, and as a result, the
clock frequency had to be reduced to 100 MHz.
Since LUT utilization is very high, it would be difcult to
place more PEs into the design. Thus, it can be inferred that
the next step of achieving further improvement would be to nd
a suitable way of simplifying the design of the ray tracing PE.
Since BRAM utilization is also high, such a modication would
lead to a smaller tile size per PE, but it would allow us to increase
the BRAM bandwidth and use more BRAMs independently.
The power consumption of the Convey system was measured
with a power meter. The system static power consumption had
a somewhat high value of 577 W, since the Convey server is
not very power efcient. In the FPGA execution state, however,
the system consumed just 16 W of extra dynamic power. There
was little power variation in different modes of execution (i.e.,
in forward projection or backward projection).
C. Accuracy
Some slices (512 512 1) of the nal reconstructed image
are shown in Fig. 10.
The original CPU software version has single-precision
oating-point variables, whereas the FPGA version has various
word length of the internal variables. The conversion from
oating-point to xed-point representation results in some
quantization errors. Following a similar approach used in [36]
to obtain quantitative difference, we measured signal-to-noise
ratio (SNR), root-mean-squared error (RMSE), universal
quality index (UQI), and correlation coefcient (CC). Since
the ground truth is not known, the 50th iteration output of
the oating-point software version is assumed to be the
ground truth. After 50 iterations, the FPGA output image for
the M.125000 dataset has , ,
, and , which suggests that the dif-
ference is small enough for most purposes. Fig. 11 shows how
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
12
TABLE XII
COMPARISON WITH OTHER ACCELERATION WORK ON 3D CT RECONSTRUCTION
Fig. 10. Reconstructed lung image from M.125000 testset. (a) Slice 100.
(b) Slice 200. (c) Slice 300; R0960B testset (25% projection). (d) Slice 50.
(e) Slice 100. (f) Slice 150.
Fig. 11. Reconstructed image difference in SNR between oating-point CPU,
GPU and xed-point FPGA, compared to the image output from the 50th
iteration of the oating-point CPU version.
the CPU, GPU, and FPGA versions converge to the assumed
ground truth in SNR after each iteration.
We have also processed the output image by our computer-
aided cancer nodule detection algorithm [24], [25]. Experiment
shows that the same cancer candidate nodule is detected in both
FPGA and CPU output image at slice 138 151. This suggests
that the quantization error has little effect on the accuracy of the
subsequent detection algorithm.
IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS
D. Discussion of Other Related Work and Comparison
Previous work on accelerating CT reconstruction has pro-
posed efcient parallelization techniques. Scherl et al. use a
master-slave approach for a deeper pipeline and present com-
parisons among multi-core CPU, Cell, GPU, and FPGA [12].
Okitsu et al. explain how to efciently divide the workload into
multiple GPUs [13]. Xu and Mueller show how to exploit var-
ious specialized graphic hardware for CT reconstruction [14].
Chou et al. propose a divergence and bank conict-free algo-
rithm for GPU with data sampling and reordering [15]. Xiao
et al. present techniques to avoid thread divergence in a GPU
by converting conditional operations into arithmetic operations
[16].
Also, some work has proposed data transfer reduction tech-
niques. Chidlow and Mller proposed a bit-splitting technique
over different color channels to maintain the quality of the result
with reduced data bits [8]. Baer and Kachelrie discovered that
using a half-precision oating point can reduce the data transfer
on the Xeon Phi architecture [18]. Gac et al. described a loop
reordering method to improve the temporal and spatial access
locality [19]. Zhou et al. report that batching rays similar in di-
rection helps improve the memory efciency [17]. Yan et al.
proposed a cyclic data copy method to reduce the data transfer
from renderer to texture memory [20].
Most of the acceleration work described above is based on
FBP. The acceleration work on iterative algorithms can be found
in [8], [9], [22], [33], [37][39]. The latest work is [9], [37],
which uses the same ray tracing algorithm and the same plat-
form as this paper. In their work, they propose a hybrid solu-
tion of GPU and FPGA for energy efciency. They also present
a data prefetching scheme to decrease the read latency and a
sparse access technique to reduce the data transfer. Another
recent work is [33], which is based on a separable footprint
(SF) algorithm. They present a method that reuses rays in the
voxel-driven approach and a method that reduces the represen-
tation size of the ray traversal path. Also, a high throughput is
achieved using a clever water-lling buffer scheme.
Table XII shows the performance comparison with other
FPGA and GPU implementations. Direct comparison is dif-
cult since other work uses different platforms, images, and
algorithms. Thus, the voxel update rate (GUPS) is used for
approximate comparison.
The table shows that our work has a superior performance
over previous work [22], [23], [33], [37] based on iterative al-
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
CHOI AND CONG: ACCELERATION OF EM-BASED 3D CT RECONSTRUCTION USING FPGA
gorithms. Compared to [37] which uses the same EM algorithm,
our proposed implementation has higher parallelism (256 versus
64), and slower clock frequency (100 MHz versus 150 MHz).
The main difference is the new parallel scheme, data reuse ap-
proach, and the throughput-optimized architecture which ac-
counts for about a 16 difference in performance.
Note that our work even has better GUPS performance than
most previous FBP-based work. FBP uses point voxels, which
allows the weight computation to be simplied with multiplica-
tion of cosine/sine value and interpolation. The ray tracing algo-
rithm used in this paper is more compute-intensive and harder
to parallelize since it serially computes the intersection of rays
onto a volumetric voxel.
VI. CONCLUSIONS
This paper describes the acceleration of the EM algorithm on
the Convey FPGA platform. We have shown that parallelizing
ray-driven voxel tiles can solve the access conict problem and
increase memory bandwidth. Also, reusing sinogram and voxel
can largely reduce external memory access. Optimization based
on ofine analysis can be performed to exploit the fact that
memory access is independent of the input patient data. More-
over, a customized PE architecture and optimization strategies
were presented to increase the system throughput. The experi-
mental results suggest that the combination of an efcient local
data transfer scheme and task pipelining can lead to a large
speedup in FPGA-based design. The proposed system can out-
perform other parallel platforms and achieve a 26.9 speedup
over the 16-thread multicore CPU implementation.
ACKNOWLEDGMENT
The authors would like to thank Dr. W. Hsu (UCLA
Radiology) and Dr. M. Yan (UCLA Mathematics) for the
C code, Dr. Y. Zou (Arista Networks) for the FPGA code
[9], and D. Wu (UCLA Computer Science) for the GPU
code of the CT reconstruction. They would also like to thank
J. Wheeler (UCLA Computer Science) for proofreading this
manuscript. They are grateful to Dr. A. Bui (UCLA Radiology)
and Dr. W. Hsu for sharing the chest CT dataset.
REFERENCES
[1] B. Li, E. Weber, and S. Crozier, A MRI rotary phased array head coil,
IEEE Trans. Biomed. Circuits Syst., vol. 7, no. 4, pp. 548556, 2013.
[2] J. Um et al., A single-chip 32-channel analog beamformer with 4-ns
delay resolution and 768-ns maximum delay range for ultrasound med-
ical imaging with a linear array transducer, IEEE Trans. Biomed. Cir-
cuits Syst., vol. 9, no. 1, pp. 138151, 2015.
[3] L. Feldkamp, L. Davis, and J. Kress, Practical cone beam algorithm,
J. Opt. Soc. Amer. A, vol. 1, no. 6, pp. 612619, 1984.
[4] A. Dempster, N. Laird, and D. Rubin, Maximum likelihood from in-
complete data via the EM algorithm, J. Roy. Stat. Soc., Series B, vol.
39, no. 1, pp. 138, 1977.
[5] L. Shepp and Y. Vardi, Maximum likelihood reconstruction for emis-
sion tomography, IEEE Trans. Med. Imag., vol. 1, no. 2, pp. 113122,
1982.
[6] R. Nelson, S. Feuerlein, and D. Boll, New iterative reconstruction
techniques for cardiovascular computed tomography: How do they
work, and what are the advantages and disadvantages?, J. Cardiovas-
cular Computed Tomogr., vol. 5, no. 5, pp. 286292, 2011.
[7] M. Beister, D. Kolditz, and W. Kalender, Iterative reconstruction
methods in X-ray CT, Phys. Med., vol. 28, no. 2, pp. 94108, 2012.
13
[8] K. Chidlow and T. Mller, Rapid emission tomography reconstruc-
tion, in Proc. Eurographics/IEEE TVCG Workshop Volume Graph.,
2003, pp. 1526.
[9] J. Chen et al., A hybrid architecture for compressive sensing 3-D CT
reconstruction, IEEE J. Emerg. Sel. Topics Circuits Syst., vol. 2, no.
3, pp. 616625, 2012.
[10] B. Meng, G. Pratx, and L. Xing, Ultrafast and scalable cone-beam CT
reconstruction using MapReduce in a cloud computing environment,
Med. Phys., vol. 38, no. 12, pp. 66036609, 2011.
[11] J. Rosen, J. Wu, J. Fessler, and T. Wenisch, Iterative helical CT re-
construction in the cloud for ten dollars in ve minutes, in Proc. Int.
Meeting Fully 3D Image Recon. in Rad. and Nucl. Med., 2013, pp.
241244.
[12] H. Scherl et al., Evaluation of state-of-the-art hardware architectures
for fast cone-beam CT reconstruction, Parallel Comput., vol. 38, pp.
111124, 2011.
[13] Y. Okitsu, F. Ino, and K. Hagihara, High-performance cone beam re-
construction using CUDA compatible GPUs, Parallel Comput., vol.
36, no. 2, pp. 129141, 2010.
[14] F. Xu and K. Mueller, Real-time 3D computed tomographic recon-
struction using commodity graphics hardware, Phys. Med. Biol., vol.
52, pp. 34053417, 2007.
[15] C. Chou, Y. Chuo, Y. Hung, and W. Wang, A fast forward projection
using multithreads for multirays on GPUs in medical image reconstruc-
tion, Med. Phys., vol. 38, no. 7, pp. 40524065, 2011.
[16] K. Xiao, D. Chen, X. Hu, and B. Zhou, Efcient implementation of
the 3D-DDA ray traversal algorithm on GPU and its application in ra-
diation dose, calculation, Med. Phys., vol. 39, no. 12, pp. 76197625,
2012.
[17] B. Zhou, X. Hu, and D. Zhen, Memory-efcient volume ray tracing
on GPU for radiotherapy, in Proc. IEEE Symp. Applicat. Specific Pro-
cessors, 2011, pp. 4651.
[18] M. Baer and M. Kachelrie, High performance parallel beam and per-
spective cone-beam backprojection for CT image reconstruction on
pre-production Intel Xeon Phi, in Proc. Int. Meeting Fully 3D Image
Recon. in Rad. and Nucl. Med., 2013, pp. 233236.
[19] N. Gac, S. Mancini, M. Desvignes, and D. Houze, High speed 3-D
tomography on CPU, GPU, and FPGA, EURASIP J. Embed. Syst.,
vol. 2008, pp. 112, 2008.
[20] G. Yan, J. Tian, S. Zhu, Y. Dai, and C. Qin, Fast cone-beam CT image
reconstruction using GPU hardware, J. X-Ray Sci. Technol., vol. 16,
pp. 225234, 2008.
[21] L. Rudin, S. Osher, and E. Fatemi, Nonlinear total variation based
noise removal algorithms, Physica D, vol. 60, pp. 259268, 1992.
[22] J. Chen et al., EM+TV for reconstruction of cone-beam CT with
curved detectors using GPU, in Proc. Int. Meeting Fully 3D Image
Recon. in Rad. and Nucl. Med., 2011, pp. 363366.
[23] Y. Choi, J. Cong, and D. Wu, FPGA implementation of EM algorithm
for 3-D CT reconstruction, in Proc. 22nd IEEE Int. Symp. Field-Pro-
grammable Custom Comput. Mach., 2014, pp. 157160.
[24] N. Duggan et al., A technique for lung nodule candidate detection
in CT using global minimization methods, Energy Minim. Methods
Comput. Vis. Pattern Recognit., pp. 478491, 2015.
[25] S. Shen, A. Bui, J. Cong, and W. Hsu, An automated lung segmen-
tation approach using bidirectional chain codes to improve nodule de-
tection accuracy, Comput. Biol. Med., vol. 57, pp. 139149, 2015.
[26] M. Yan and L. Vese, Expectation maximization and total variation-
based model for computed tomography reconstruction from undersam-
pled data, Proc. SPIE Med. Imag., pp. 79612X-18, 2011.
[27] H. Zhao and A. J. Reader, Fast ray-tracing technique to calculate line
integral paths in voxel arrays, in Proc. IEEE Nucl. Sci. Symp. Conf.
Rec., 2003, vol. 4, pp. 28082812.
[28] R. Siddon, Fast calculation of the exact radiological path for a three-
dimensional CT array, Med. Phys., vol. 12, pp. 252255, 1985.
[29] B. D. Man and S. Basu, Distance-driven projection and backprojection
in three dimensions, Phys. Med. Biol., vol. 49, no. 11, pp. 24632475,
2004.
[30] A. Ziegler, T. Khler, T. Nielsen, and R. Proksa, Efcient projection
and backprojection scheme for spherically symmetric basis functions in
divergent beam geometry, Med. Phys., vol. 32, no. 12, pp. 46534663,
2006.
[31] All Programmable FPGAs and 3D ICs Xilinx, 2015 [Online]. Avail-
able: http://www.xilinx.com/products/silicon-devices/fpga.html
[32] J. Bakos, High-performance heterogeneous computing with the
Convey HC-1, IEEE Comput. Sci. Eng., vol. 12, no. 6, pp. 8087,
2010.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
14
[33] J. Kim, J. Fessler, and Z. Zhang, Forward-projection architecture for
fast iterative image reconstruction in X-ray CT, IEEE Trans. Signal
Process., vol. 60, no. 10, pp. 55085518, 2012.
[34] J. Cong et al., Combining module selection and replication for
throughput-driven streaming programs, in Proc. Conf. Design,
Autom. Test in Europe, 2012, pp. 10181023.
[35] J. Cong, M. Huang, and P. Zhang, Combining computation and com-
munication optimizations in system synthesis for streaming applica-
tions, in Proc. ACM/SIGDA Int. Symp. Field-Programmable Gate Ar-
rays, 2014, pp. 213222.
[36] X. Li and S. Luo, A compressed sensing-based iterative algorithm
for CT reconstruction and its possible application to phase contrast
imaging, Biomed Eng. Online, vol. 10, 2011.
[37] J. Chen, J. Cong, M. Yan, and Y. Zou, FPGA-accelerated 3D recon-
struction using compressive sensing, in Proc. ACM/SIGDA Int. Symp.
Field-Programmable Gate Arrays, 2012, pp. 163165.
[38] B. Keck et al., GPU-accelerated SART reconstruction using the
CUDA programming environment, Proc. SPIE Med. Imag., pp.
72582B-19, 2009.
[39] F. Xu and K. Mueller, Accelerating popular tomographic reconstruc-
tion algorithms on commodity PC graphics hardware, IEEE Trans.
Nucl. Sci., vol. 52, no. 3, pp. 654663, Jun. 2005.
Young-kyu Choi (S14) received the B.S. and M.S.
degrees in electrical engineering from Seoul National
University, Seoul, Korea.
Currently, he is working toward the Ph.D. de-
gree in computer science under the guidance of
Prof. Jason Cong at the University of California, Los
Angeles, Los Angeles, CA, USA. He has worked
at LG Electronics (20082011) and Inha University,
Incheon, Korea (20112012). His research interests
include characterization and implementation of
FPGA-friendly applications.
IEEE TRANSACTIONS ON BIOMEDICAL CIRCUITS AND SYSTEMS
Jason Cong (F00) received the B.S. degree from
Peking University, Beijing, China, in 1985, and the
M.S. and Ph.D. degrees from the University of Illi-
nois at Urbana-Champaign, Champaign, IL, USA, in
1987 and 1990, respectively, all in computer science.
At the University of California, Los Angeles
(UCLA), Los Angeles, CA, USA, he is a Chan-
cellors Professor in the Computer Science Depart-
ment and holds a joint appointment in the Electrical
Engineering Department. He is the Director of the
Center for Domain-Specic Computing (CDSC),
Codirector of the UCLA/Peking University Joint Research Institute in Science
and Engineering, and the Director of the VLSI Architecture, Synthesis, and
Technology (VAST) Laboratory. Additionally, he served as Chair of the UCLA
Computer Science Department (20052008). His research interests include
energy-efcient computing, customized computing for big-data applications,
synthesis of VLSI circuits and systems, and highly scalable algorithms. He has
authored more than 400 publications in these areas.
Dr. Cong has received 10 best paper awards, two 10-Year Most Inuential
Paper Awards (from ICCAD14 and ASPDAC15), and the 2011 ACM/IEEE
A. Richard Newton Technical Impact Award in Electric Design Automation.
He was elected an ACM Fellow in 2008. He was the recipient of the 2010 IEEE
Circuits and System (CAS) Society Technical Achievement Award For seminal
contributions to electronic design automation, especially in FPGA synthesis,
VLSI interconnect optimization, and physical design automation.