548 IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS, VOL. 54, NO.
1, FEBRUARY 2007
Comparative Study of VLSI Solutions to
Independent Component Analysis
Hongtao Du, Student Member, IEEE, Hairong Qi, Senior Member, IEEE,
and Xiaoling Wang, Student Member, IEEE
Abstract—The advent of independent component analysis (ICA)
has brought a paradigm shift to signal and image processing.
ICA that extracts independent source signals by searching for a
linear or nonlinear transformation and minimizing the statistical
dependence between components has the promise of effective
unsupervised signal separation capability. Due to the computation
complexity of ICA and commonly high-volume data sets used in
signal and image processing, the ICA process, however, is very
time-consuming. Very large scale integration (VLSI) solutions
with optimal parallelism provide potentially faster and even re-
al-time implementations for ICA algorithms. In this paper, the
authors study these solutions and discuss their limits. Critical
challenges are identified, and issues associated with the VLSI
implementation of ICA algorithms are designed. Design recom-
mendations that have potentials in performing complicated ICA
algorithms on large throughput are provided.
Index Terms—Application-specific integrated circuit (ASIC),
CMOS integrated circuits, field-programmable gate array
(FPGA), independent component analysis (ICA), reconfigurable
architectures, very large scale integration (VLSI).
I. I NTRODUCTION
I NDEPENDENT component analysis (ICA) is the most pop-
ular technique developed to solve the blind source sep-
aration (BSS) problem [1]. Under the assumption of signal
independence across sources, the task of BSS and ICA is to
separate and recover independent source signals from linear or
nonlinear mixed sensor observations, in which both sources and
the unmixing matrix are unknown. ICA not only decorrelates
the signals that are of second-order statistics using a minimum
of a priori information but also reduces higher order statistical
dependencies between reconstructed signals. This principle has
been used in other applications, such as recognition [2] and
hyperspectral image analysis [3]. In particular, ICA is very
effective for unsupervised source estimations given only the
observations of mixed signals.
Although powerful, the complicated arithmetics, the iterative
computation with slow convergence rate, and the generally
Manuscript received November 17, 2004; revised April 9, 2005. Abstract
published on the Internet September 15, 2006. This work was supported in part
by the Office of Naval Research under Grant N00014-04-1-0797.
H. Du and H. Qi are with the Department of Electrical and Com-
puter Engineering, University of Tennessee, Knoxville, TN 37996-2100 USA
(e-mail: hdu1@utk.edu; hqi@utk.edu).
X. Wang is with the SONY Advanved Technology Center, San Jose, CA
95134 USA (e-mail: grace.wang@am.sony.com).
Digital Object Identifier 10.1109/TIE.2006.885491
0278-0046/$25.00 © 2007 IEEE
large volumes of raw and processed data have caused ICA
algorithms time-consuming processes for software implemen-
tation. Hardware implementation provides not only an optimal
parallelism but also potentially faster and real-time solutions.
While software implementation is useful for investigating the
capabilities of ICA algorithms and is sufficient for most ap-
plications, hardware implementation is essential to fully ben-
efit from the parallel architecture and to facilitate high-speed
processing. The major difference between hardware and soft-
ware implementations lies in the fact that hardware subroutines
are executed by integrated circuits (ICs) instead of a series
of microinstructions. Hardware implementation also solves the
insufficient memory problem encountered by software for large
data sets and high dimensionality.
During the last decade, advances in very large scale
integrated (VLSI) circuit technologies have allowed de-
signers to implement some ICA algorithms on fully ana-
log CMOS circuits, analog–digital (AD) mixed-signal ICs,
digital application-specific ICs (ASICs), and general field-
programmable gate arrays (FPGAs). Both analog CMOS cir-
cuits and mixed-signal ICs are fully customized by designers
using either analog or AD mixed technologies, where the
silicon is utilized in the most efficient manner but the develop-
ment expense is incredibly high. The digital nonprogrammable
ASICs such as standard-height library and mask gate arrays are
also full-custom VLSI and are used to implement designs at
high circuit density by specifying interconnections during latter
stages of the IC manufacturing process. In addition, the large
amount of available standard libraries of basic logic cells makes
the design expense much cheaper and the design process much
faster. The FPGAs based on the reconfiguration technology are
the most economic and efficient solutions to ICA algorithms
since they allow end users to modify and configure their designs
multiple times. Specifically, recent rapid increase in the density
of FPGAs has made it possible to implement large ICA designs
with a completely hardware-driven approach.
This paper identifies critical challenges and design issues for
hardware implementation of ICA algorithms, studies existing
VLSI solutions, and discusses future directions. Section II
briefly describes ICA principles and different software imple-
mentation algorithms and addresses several VLSI design chal-
lenges. Section III studies existing VLSI solutions to various
ICA algorithms and demonstrates our ASIC and FPGA designs.
Section IV presents some new VLSI technologies and discusses
potential solutions to the design challenges. Finally, Section V
concludes this paper.
DU et al.: COMPARATIVE STUDY OF VLSI SOLUTIONS TO ICA
II. ICA
ICA is a method of finding a linear nonorthogonal coordinate
system in any multivariate data [4]. The directions of the axes of
this coordinate system are determined by both the second-order
and higher order statistics of the original data. The goal is to
perform a linear transform that makes the resulting variables
as statistically independent from each other as possible. Let
s1 , . . . , sm be the m source signals that are statistically inde-
pendent, and no more than one signal is Gaussian distributed;
that is, none of si gives any information on other signals.
The n observed signals x1 , . . . , xn are unmixed by an m × n
unmixing matrix or weight matrix W to generate the source
signals in the following ICA unmixing model:
S = WX (1)
where W = [w1 , . . . , wm ]T and wi = [wi1 , . . . , win ],
i = 1, . . . , m.
A. Principal Approaches
There are two different research communities that have con-
sidered the analysis of independent components [5], including
the mixed sources separation and the unsupervised learning
based on information theory. The study of separating mixed
sources observed in an array of sensors has been a classical
and difficult signal processing problem. Herault and Jutten
were the first working on BSS who, in their seminal work [6],
introduced an adaptive Herault–Jutten (H–J) algorithm in a
simple feedback network topology y = −Wy + x, where x
is the observation, y is the separated signal, and W is
the unmixing matrix with zero diagonal terms [wii = 0, i =
1, . . . , min(m, n)] that is able to separate several unknown in-
dependent sources. Their approach has been further developed
by many other researchers. Common was one of them, who
elaborated the concept of ICA and proposed cost functions
related to the approximate minimization of mutual information
between sensors [7]. In parallel to BSS studies, unsupervised
learning rules based on information theory were proposed
whose goal was to maximize the mutual information between
the inputs and outputs of a neural network [8]. Roth and
Baram [9] and Bell and Sejnowski [10] independently derived
stochastic gradient learning rules for this maximization and
applied them, respectively, to forecasting, time-series analy-
sis, and blind separation of sources. In their work, Bell and
Sejnowski put the BSS problem into an information-theoretic
framework and demonstrated its effectiveness in the separation
and deconvolution of mixed sources [10].
Intuitively speaking, the key to estimating the ICA model is
non-Gaussianity since the source signals are desired to contain
the least Gaussian components. Hence, the definition and esti-
mation of a contrast function that measures the non-Gaussianity
of independent components is necessary for the identifiability
of the model. There are many different representations of the
contrast function, where high-order cumulates, mutual informa-
tion, and negentropy are the most important ones.
The fourth-order cumulate or kurtosis is the classical mea-
sure of non-Gaussianity of signals. It is defined as kurt(u) =
549
E{u4 } − 3(E{u2 })2 for a zero-mean random variable u [1],
[11]. If u is a Gaussian random variable, its kurtosis would
be zero. For those probability densities peaked at zero, their
kurtosis is positive, and for those flatter probability densities,
their kurtosis is negative. Hyvärinen and Oja derived an ob-
jective function in [1] to find W that maximizes the kurtosis
kurt[WT x(t)], where W is the corresponding weight matrix
and x(t) is the observation on time variable.
Mutual information, which is inspired by information theory,
is a natural measure of the dependence between random vari-
ables, making it a good candidate for finding non-Gaussianity
of the signals. The mutual information I between m random
variables

m yi , i = 1, . . . , m, is defined as I(y1 , y2 , . . . , ym ) =
i=1 H(y i ) − H(y), where H(y) is the differential entropy
of the random variable [12]. The minimization of mutual in-
formation corresponds to finding the most independent compo-
nents. The information maximization (InfoMax) principle [10]
is derived from the minimization of mutual information and,
correspondingly, maximization of the output entropy.
Another very important measure of non-Gaussianity is given
by negentropy, which is derived based on the information-
theoretic quantity of differential entropy [11]. The negentropy
of a random vector y is defined as J(y) = H(ygauss ) − H(y),
where H(ygauss ) denotes the differential entropy of a Gaussian
random variable ygauss , which has the same covariance matrix
as y. The advantage of using negentropy as a measure of non-
Gaussianity is that it is well justified by statistical theory [11].
Because it is difficult to calculate negentropy, an approxima-
tion is usually given as J(Y) ≈ {E[G(Y)] − E[G(Ygauss )]}2 ,
where G(Y) is a nonquadratic function [1]. In ICA algorithms,
Y = wT X, where wT is the transpose of the weight vector
w. The FastICA algorithm [1] is developed to find w that
maximizes the above objective function. It uses a fixed-point
iteration scheme to find a direction such that the projection of
weight vectors maximizes an approximation of negentropy.
In addition, the approach of output divergence minimiza-
tion [13] uses Kullback–Leibler (K–L) divergence or relative
entropy of the output signals as the objective function and
minimizes the divergence with gradient descent learning. The
maximum likelihood method [5] assumes that the input mixed
signals are mutually independent and maximizes the likelihood
of the input signals.
B. VLSI Implementation Challenges
As mentioned above, many ICA algorithms are slow
processes in signal/image processing applications due to the
complicated arithmetics as well as the time-consuming iterative
computation. VLSI is an ideal algorithm implementation carrier
and offers many features such as high processing speed, which
is extremely desired in ICA implementations. The complicated
arithmetics of ICA is one of the main barricades in ICA
hardware implementation, especially in synthesis procedure.
Therefore, hierarchy and modularity techniques in VLSI design
are essential for most ICA implementations to overcome the
complexity of ICA algorithms. The hierarchy, or the divide
and conquer technique, involves dividing an ICA process into
subprocessing modules until the complexity of the bottom
550 IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS, VOL. 54, NO. 1, FEBRUARY 2007
Fig. 1. Design flow using RCs.
level submodules becomes manageable. These submodules are
independently developed, then integrated together and put into
a design and development environment for performing tasks
such as synthesis, optimization, placement, and routing. The
use of modularity enables the parallelism of the design process
and facilitates the development of generic modules in various
designs. Recently, the concept of modularity has gained more
attention because of a need to reduce the design cycle and the
development cost. For example, Cho and Lee [14] implemented
the InfoMax ICA on the analog CMOS circuit in 2001, and
Cauwenberghs [15] implemented the same algorithm using the
AD mixed-signal technology in 2003. Both implementations
utilized modular architectures such that the designs could be
easily expanded to larger chips and further integrated into mul-
tichip systems for a large number of input and output channels.
In our previous work [16], some ICA-related reconfigurable
components (RCs) have been developed for purposes of reuse
and retargeting. The design and use of RCs make system
synthesis faster, easier, and more efficient and reduce time-to-
market, which directly leads to higher revenues [17]. Compared
to the effort of one-time use, it takes about 50% more time to
verify code for reuse and reconfiguration, but other designers
would benefit for 70% [17] or more design time reduction
from these RCs. As another advantage, the well-tested RCs
also reduce design risks by avoiding verifications. RCs that
are parameterized using generic maps as to make them highly
flexible for future instances have helped focusing the synthesis
work mainly on configuration and interconnection. During the
implementation process, the necessary RCs are fetched from
the library, as demonstrated in Fig. 1. The generics are con-
figured according to application specifications, whereas input
and output ports are interconnected to form processes or sub-
processes. In addition, these ICA-related RCs can be modified,
improved, and extended to some new RCs as necessary for
specific categories of ICA applications.
Another VLSI design challenge concerning the ICA im-
plementation is the iterative computation that requires large
amount of processing time and RAM to store intermediate
results. The best solution to this problem is to reduce the ICA
algorithm at the early preprocessing stage of the VLSI de-
sign process. Computationally efficient ICA algorithms would
contain less iterations and converge rapidly to the desired
accuracy. Researchers have recently accelerated ICA processes
by including nonlinear learning parameters. For example, the
α-ICA algorithm [18] uses the α-logarithm to minimize the
mutual information among source signals. The fuzzy infer-
ence system [19] uses fuzzy-based adaptive learning rate to
measure the second-order and higher order correlation coeffi-
cients. Additionally, under some assumptions and constraints,
noniterative ICA algorithms, such as the algebra ICA [20] and
Molgedey–Schuster ICA [21], can be developed to approximate
the true solutions.
ICA algorithms are mostly applied in signal and image
processing field, which usually entails large volumes of raw or
preprocessed data that are transferred in and out of the VLSI
designs. A successful ICA hardware implementation therefore
requires careful design of internal RAM and I/O bandwidth.
If transferring data directly to the VLSI chip is not a feasible
option, then data transfer can be accomplished through shared
memory or external software programs depending on specific
applications.
III. C OMPARATIVE S TUDY OF E XISTING S OLUTIONS
TO ICA A LGORITHMS
In the past few years, many advantages found in VLSI
designs have inspired the implementation of complex ICA
algorithms.
A. Analog CMOS Integration
When ICA algorithms are implemented on analog CMOS
circuits, end users synthesize progressive processing at the
transistor level; that is, the low-level element transistors and
necessary interconnections can be customized. The application
specified by analog circuits utilizes the minimum amount of
transistors and the shortest interconnections to achieve high cir-
cuit density and, correspondingly, low circuit delay and power
consumption. Within a limited chip area, designers can specify
tens of millions of transistors to integrate computation for
ICA algorithms. These huge amount of transistors generate the
massively parallel analog CMOS circuits and make high-speed
processing possible. In the implementation process, a complex
ICA algorithm is decomposed into several subprocesses, each
of which is accomplished by a cluster of transistors to per-
form the corresponding function. The popularly used modular
structure can then be extended to any size design easily. The
final circuit is either connected to external peripherals, such as
AD convert/digital–analog (DA) converter, or integrated with
a digital signal processor and an AD/DA converter within
one chip.
The low-power high-density analog CMOS circuit is one of
the earliest solutions to ICA algorithms. In 1995, Cohen and
Andreou [22] designed two analog CMOS chips to implement
the H–J ICA algorithm for blind separation of mixed speech
signals. The chip core size (transistor counts) varied according
to the design scope. For example, the two-input × two-output
chip used 76 transistors, and the five-input × five-output chip
used 715 transistors. The core was integrated with analog I/O
interfaces, weight coefficients, and adaptation blocks on one
chip. The fabrication was targeted to a 2-µm, n-well, two-
poly two-metal CMOS process, in which chips employed three
power lines, namely Vdd (+2.5 V), Vss (−2.5 V), and Gnd. In
the same year, Gharbi and Salam [23] also implemented the
DU et al.: COMPARATIVE STUDY OF VLSI SOLUTIONS TO ICA 551

TABLE I
COMPARISON OF ANALOG, MIXED-SIGNAL, AND ASIC SOLUTIONS

H–J algorithm on a 2.22 × 2.25 mm tiny chip using the 2.0-µm

CMOS technology. More recently, Cho and Lee [14] designed
a fully analog CMOS chip to implement the InfoMax theory-
based ICA algorithm, in which analog multiplier, nonlinear
function, and weight update circuits were used. The fabricated
four-input × four-output ICA chip used a 0.6-µm, p-well, two-
poly three-metal AMS CMOS process and had a 2.8 × 2.8 mm
active area. Both the input and output voltages ranged between
−2.5 and +2.5 V.
Although very efficient in circuit design, the analog CMOS
integration suffers from high expense of the workstation-based
Fig. 2. Pilchard board.
development system ($150 000) and slow turnaround time
(usually eight weeks), which make it subpotential for most ICA
designs and fast implementations [17]. In addition, the fabri- onto a feedforward system. A mixed-signal adaptive parallel
cated analog CMOS circuits do not necessarily exhibit the same VLSI architecture was then used to implement this feedforward
behavior as they do in the ideal simulations. This phenomenon network. The elements in the unmixing matrix were stored
is called the “transistor mismatch,” which is caused by three in 12-bit precision digital cells of the mixed-signal architec-
identifiable factors, namely: 1) edge effects; 2) striation effects; ture, which was implemented using fully differential switched-
and 3) random variations [24]. General solutions include capacitor simpled-data circuits. The chip was fabricated using
increasing transistor size (W and L) to afford more current and a 0.5-µm two-poly three-metal CMOS technology, and the
using cocentric structure to allow matched transistors to share core size was 3 × 3 mm. The three-input × three-output chip
the same surrounding structures. However, it would take more could unmix three independent components. This ICA VLSI
time and cost more to solve these physical problems. processor was used as a front end of the system integration,
in which it separated the mixed analog acoustic inputs and then
fed the digital outputs to Xilinx FPGA for further classification.
B. AD Mixed-Signal Techniques Although the aforementioned research efforts offer possibly
The AD mixed-signal techniques inherit all the advantages better solutions to some ICA applications compared to the
of analog CMOS circuits, but the architecture is digitally re- analog CMOS integration, AD mixed-signal techniques do not
configurable and is capable of implementing a class of similar improve much on the development expense and the prototype
applications. turnaround period.
In the neuromorphic autoadaptive systems project conducted
at The Johns Hopkins University [15], Celik et al. [25] simpli-
C. ASIC Solutions
fied and generalized the H–J, the InfoMax, the natural gradient
[26], and the Cichocki–Unbehauen algorithm [27] into a com- The previously discussed analog CMOS and AD mixed-
mon outer-product rule, which was approximated and mapped signal technologies provide user full-custom solutions to ICA
552 IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS, VOL. 54, NO. 1, FEBRUARY 2007
TABLE II
COMPARISON OF FPGA SOLUTIONS
algorithms. End users are generally required to have sufficient
knowledge and focus more on detailed analog physical prob-
lems and basic component designs. Therefore, the application
domains are comparatively limited, and the development costs
significant time with expenses. Fortunately, the fast-blooming
digital VLSI technologies like ASIC and FPGA allow end users
to concentrate on the algorithm implementation itself because
IC vendors provide enormous standard libraries. ASICs and
FPGAs are therefore called user semicustom solutions.
From the aspect of circuit density and efficiency, the non-
programmable ASICs cover the lower end of analog CMOS
and AD mixed-signal full-custom VLSIs and the higher end
of reprogrammable FPGAs. Compared to reprogrammable
FPGAs, nonprogrammable ASICs retain the benefits of com-
pact circuit design and low power consumption. Although the
nonprogrammable feature increases the design expense and
risk, ASICs that typically contain ten million logic gates or
more are the appropriate solutions to very complex ICA de-
signs. For example, the standard-height library cell is a design
technique for nonprogrammable ASICs, where the vendors
develop standard-height library cells for the implementation
of large amount of functions. When implementing an ICA
algorithm, the end users only need to select necessary cells
that are logic level components with constant height on chip
and then specify interconnections between layers such as poly,
metal1, and metal2 according to their designs.
To pursue potential solutions to the InfoMax-based ICA
algorithms on higher density digital ICs, the Computa-
tional NeuroSystems Laboratory, Korea Advanced Institute of
Science and Technology [28], is designing an ASIC chip for
ICA and will use it as a front end to control noise in speech
recognition.
In our previous work [29], we synthesized the FastICA-
based parallel ICA (pICA) algorithm on ASIC using standard-
height library cells. In our design, the necessary logic cells were
selected or developed in very compact format. All the cells were
filled horizontally in multiple standard cell rows with standard
height of 66/76 λ. These rows overlapped with each other
after inversion or flip-flop, so that they shared Vdd and Gnd
to save space and reduce circuit delay. In addition, the space
between rows for wiring could be justified as needed. Such
standard-height architecture brought higher performance and
more compact core area for hardware implementation, therefore
providing better solution to the ICA algorithms. Our standard-
height cell-based ASIC synthesis aimed at Taiwan Semicon-
ductor Manufacturing Company 0.18-µm process, where λ
was equal to 0.1 µm, and the CMOS fabrication process had
six metal layers and one polylayer. The voltage of the target
chip was 1.8 V for applications, and a thick oxide layer was
used for 3.3-V transistors [30]. For a pICA process containing
estimations of four weight vectors, the chip size was 1186.34 ×
1184.49 µm. The aspect ratio of the core was set to 1.001
for the convenience of placement and routing, which optimally
mapped the structural interconnection of standard-height cells
expressed in a schematic view onto the physical architecture.
All the analog, mixed-signal, and ASIC solutions depend
on compact designs from end users at the beginning stage
and fabrication of application-specific chips from hardware
companies at the final stage. Table I compares most up-to-
date analog CMOS, AD mixed-signal, and ASIC solutions to
ICA algorithms. The chip size and I/O reflect the compactness
of individual ICA designs, whereas the fabrication parameters
and voltage reflect the trend of the VLSI technology develop-
ment. Obviously, “low-voltage circuit,” which directly results
DU et al.: COMPARATIVE STUDY OF VLSI SOLUTIONS TO ICA 553

Fig. 3. Capacity utilization of Xilinx VIRTEX V1000EHQ240-6 for different numbers of weight vectors in pICA. The dotted lines denote the maximum capacity
of Xilinx VIRTEX V1000EHQ240-6. (a) Delay. (b) Slice. (c) Transistor. (d) Equivalent gate.

TABLE III
PERFORMANCE COMPARISON BETWEEN RECONFIGURABLE FPGA SYSTEM AND SINGLE FPGA IMPLEMENTATION

in squared power conservation, and “small chip size,” which
requires compact circuit design, are the current trend.
D. FPGA Solutions
Among all the VLSI technologies, FPGAs provide the most
economic and efficient solutions to comparatively simple ICA
algorithms and could provide lower cost substitute of nonpro-
grammable ASICs. Unlike nonprogrammable VLSI devices,
FPGAs are standard and general-purpose products fabricated
by hardware companies before end users implement specific
ICA designs on them. FPGAs are developed based on re-
configurable technologies, in which end users are allowed
to modify their designs for multiple times and program the
interconnections in a few hours instead of waiting several weeks
for the final fabrication and metalization. These savings in the
development expense and the turnaround time of prototyping
directly lead to time-to-market reduction and profit increase.
Most FPGAs contain 2000 to 2 000 000 logic gates [31] and
use architectures that support a balance between logic resources
and routing resources [32]. Typical FPGAs are composed of
a two-dimensional array of input/output blocks, interconnects,
and configurable logic blocks (CLBs) that can be customized
to implement logic functions. The programmable interconnects
554 IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS, VOL. 54, NO. 1, FEBRUARY 2007
Fig. 4. Global run-time reconfiguration flow.
between these CLBs allow end users to implement the mul-
tilevel logic functions [33]. FPGA vendors prefabricate rows
of gates and programmable connections, whereas end users
specify and interconnect the programmable CLBs to perform
the desired ICA algorithms.
In recent years, FPGAs have become the most popu-
larly used devices for various VLSI implementations of ICA
algorithms. In 2001, Lim et al. [34] implemented two small-
size independent component neural network (ICNN) prototypes
that were based on mutual information (between input and
output) maximization and output divergence minimization. The
implementation was on Xilinx Virtex XCV 812E, which con-
tains 0.25 million logic gates [35]. All the variables in the
network were represented as fixed-point numbers. The input
signals were stored in 1-Mb RAMs and iteratively read by
the ICNNs until the weight-updating process converged. Two
7-neuron ICNN prototypes were implemented, and comparison
was conducted in terms of cost and performance to evaluate
which one was more suitable for hardware implementation.
In [36], Nordin et al. proposed a pipelined ICA architecture
for potential FPGA implementation. The InfoMax algorithm
programmed in MATLAB from Tony Bell [10] was first broken
down into four modules, each of which was translated into C,
and then into HDL to implement on the four-stage pipelined
FPGA array. Since each FPGA in this pipeline does not have
data dependence with others, all blocks could be further imple-
mented and executed in parallel.
Fig. 5. FPGA developments versus Moore’s law.
In 2002, Satter and Charayaphan [37] implemented an im-
proved InfoMax BSS algorithm proposed by Torkkola [38] on
a Xilinx Virtex E FPGA, which contains 0.6 million logic gates.
They used the Xilinx CAD tools of System Generator V2.1 and
ISE 4.1 for code generating, synthesis, and routing. However,
this method is not optimal from the aspects of both synthesis
architecture and circuit compactness. As a result, the capacity
limit led to the maximum iteration number being prelimited
to 50 and the buffer size to 2500 samples. Since the required
memory size exceeded the capacity of the FPGA, a MATLAB
program was used instead, which was executed on the external
memory. Wei and Charoensak [39] used the same method in
[37] to implement a noniterative algebra ICA algorithm devel-
oped by Yamaguchi and Itoh [20] for the purpose of speeding
up motion detection in image sequences. The target FPGA
was Xilinx Virtex E. Although the design consumed 90 200
of the 600 000 logic gates, the application only supported the
unmixing of two independent components.
In 2003, Kim et al. [40] combined the InfoMax ICA-based
BSS algorithm and the ICA adaptive noise canceling algo-
rithm, which minimized entropy between signals and noise.
An expandable modularized processor design for this ICA-
based speech recognition system was elaborated and imple-
mented on Altera EP20K600EBC652-1 FPGA, which contains
0.6 million logic gates and supports embedded memory. The
observed acoustic signals collected from analog microphones
were represented as 12-bit integers, whereas the elements in
the weight matrix W were represented as 30-bit floating-point
data. In our previous work [16], the FastICA-based pICA algo-
rithm was implemented on Xilinx VIRTEX V1000EHQ240-6
FPGA with a capacity of one million logic gates. The target
FPGA was embedded on the pilchard reconfigurable computing
platform, which was plugged in a Sun workstation as shown in
Fig. 2. The pilchard reconfigurable computing platform uses the
168-pin dual inline memory module RAM slot as an interface.
It is compatible with the PC133 standard [41], thereby achiev-
ing very high data transfer rate. In the synthesis process, the
pICA algorithm was first simulated by ModelSim from Mentor
Graphics, then synthesized by Synopsys FPGA Compiler2, and
DU et al.: COMPARATIVE STUDY OF VLSI SOLUTIONS TO ICA
Fig. 6. Xilinx Virtex II-based platform with logic gates of eight million.
finally placed and routed by Xilinx XVmake. After implement-
ing the pICA on the Xilinx V1000E embedded on the pilchard
board, we achieved the maximum frequency of 20.161 MHz
(minimum period of 49.600 ns) and the maximum net delay of
13.119 ns. The pICA used 92% slices of the V1000E.
Since FPGAs are standard VLSI products, devices in the
same series have similar internal structures. Chip utilization
percentage and operation frequency are then very important
parameters for evaluating the compactness and efficiency of
individual designs. Table II compares recent FPGA solutions
to various ICA algorithms.
Although the characteristics of reconfigurability and reusable
life cycle bring FPGA more advantages than other VLSIs, they
inversely result in lower circuit density and higher circuit delay,
thereby bringing capacity limitation. During the placement
and routing processes of the pICA synthesis, we observed
that several capacity constraints barricaded single FPGA from
implementing complex algorithms like pICA. Fig. 3 shows the
relationship between the number of weight vectors need to
be estimated in pICA process and the capacity utilization of
the FPGA Xilinx VIRTEX V1000E, which is evaluated using
metrics including delay, slice, transistor, and equivalent gate.
We find that the circuit delay increases significantly after the
number of weight vectors exceeds five, whereas the number
of slices, transistors, and equivalent gates increases linearly.
Overall, a single Xilinx VIRTEX V1000E can accommodate
a pICA process with at most four weight vector estimations
that takes 92% of the maximum capacity. Even one weight
vector estimation occupies 29% of the capacity of this FPGA.
This capacity constraint of single FPGA is not only for ICA
algorithms but also a challenge for all complicated applications
on large data sets.
E. Reconfigurable FPGA System
Although FPGAs are economic solutions to complex ICA
algorithms, its capacity constraint remains a problem. On the
other hand, ASICs can provide sufficient capacity, but its de-
velopment process is too expensive, and the turnaround time is
too long. Hence, we took the advantage of the reconfigurability
of FPGAs and constructed a dynamically reconfigurable FPGA
system, in which the IC capacity limit was overcome by sacri-
ficing the overall processing time [42].
In a general FPGA platform, all processes of an ICA al-
gorithm are integrated in one design and synthesized on one
FPGA, which can be executed for multiple times. In the
555
Fig. 7. Architecture of HPRC.
dynamically reconfigurable FPGA system that we presented
in [42], we divided all processes of pICA into three groups:
1) submatrix; 2) external decorrelation; and 3) comparison.
Each group was synthesized using Synopsys FPGA Compiler2
and then placed and routed by Xilinx XVmake. Individual
group modules were downloaded on the FPGA in a serial mode.
The submatrix module was first downloaded to configure the
pilchard FPGA platform. After the submatrix module was ex-
ecuted and the task finished, the external decorrelation module
was then downloaded to reconfigure the same FPGA. Since the
immediate outputs from the preceding submatrix module were
commonly used as inputs of the subsequent configuration of the
external decorrelation module, the external memory was used to
store these intermediate signals that were originally the internal
variables in single FPGA implementation.
To compare the performance of the reconfigurable FPGA
system with that of the single FPGA implementation, we list the
design and device utilization ratios for each group in Table III.
We take the estimation of a weight matrix of 20 weight
vectors as an example. The reconfiguration process of the
reconfigurable FPGA system is illustrated in Fig. 4, in which
both the number of executions and the order of executions are
predefined. In this example, the reconfigurable FPGA system
estimates 20 weight vectors given the observation signals. Since
the submatrix module can estimate and decorrelate four weight
vectors each time, we need to execute the submatrix module
for five times after configuring the FPGA. To decorrelate these
five submatrices, the external decorrelation module needs to be
hierarchically executed for four times. The comparison module
executes one time. A shell script file is written to control the
whole reconfiguration flow.
IV. R ECENT D EVELOPMENTS AND P OTENTIAL S OLUTIONS
We have presented several effective solutions to ICA algo-
rithms in the previous section. However, we also realize the
potential problems associated with these solutions. The devel-
opment cost of full-custom design is prohibitively high and
considered a subpotential solution for most ICA algorithms.
The capacity constraint problem of FPGAs impedes some
complicated ICA algorithms from hardware implementation.
The reconfigurable FPGA system requires extra processing
time for reconfigurations and frequent data input/output, which
significantly slow down the whole process and decline the
benefit of hardware implementation. In this section, we discuss
556 IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS, VOL. 54, NO. 1, FEBRUARY 2007
Fig. 8. HPRC implementation.
two possible directions that could provide potentially better
VLSI solutions to ICA algorithms.
A. FPGA/ASIC Technology Development
The required computational power of complicated
algorithms or applications like ICA is the driving force
for the fast development of FPGA/ASIC technologies. Current
phenomenal growth of FPGA/ASIC technologies has been far
beyond the Moore’s law prediction of doubling in the number
of transistors per IC every 18 months [43]. A comparison
between the FPGA capacity tendency and Moore’s law is
demonstrated in Fig. 5. Current FPGA has provided end
users powerful computing and processing capabilities. Taking
the new Xilinx Virtex II Pro as an example, this FPGA is
manufactured based on the technology of 0.15-µm eight-layer
metal process with 0.12-µm high-speed transistors and allow
end users to implement designs on 8 000 000 logic gates with
420-MHz internal clock speed and 840-Mb/s I/O [44]. Fig. 6
shows a Xilinx Virtex II Pro-based FPGA platform, which is
in compliance with the Peripheral Component Interconnect
(PCI)-X 133-MHz, PCI 66-MHz, and PCI 33-MHz standards.
B. High-Performance Reconfigurable
Computing (HPRC) Platform
HPRC provides a potential solution for high processing
speed as well as the capacity limit by connecting the recon-
figurable FPGA systems or reconfigurable computing nodes in
a network. HPRC exploits the benefits of parallel processing
from high-performance computing (HPC) in conjunction with
adaptive hardware acceleration associated with reconfigurable
FPGA systems. HPC seeks extreme computing power associ-
ated with supercomputers and parallel computing [45], whereas
the reconfigurable FPGA system utilizes the reconfigurability
quality of FPGAs.
As the architecture shown in Fig. 7, an HPRC platform
consists of several computing nodes (CPU) and reconfigurable
FPGA platforms. For the purposes of data exchange and syn-
chronization, all computing nodes are interconnected by the
interconnection network (ICN), whereas RC boards are inter-
connected via reconfigurable ICN. The individual computing
nodes and RC boards are connected with each other through
high-speed channels like memory bus or PCI bus.
HPRC can be implemented using multiple pilchard reconfig-
urable computing platforms, which are plugged in the memory
bus of the Sun workstations. As shown in Fig. 8 [46], the Sun
workstations (computing nodes) are interconnected through the
Ethernet. The pilchard platforms (RC boards) communicate
with each other through shared files. During the implementation
of the pICA algorithm, the modules described in Section III-E
are distributed on individual computing node. All modules
are then configured and executed in the parallel mode and
collaboratively perform the whole process.
With the fast development of current VLSI technologies,
we can expect large-capacity low-power consumption FPGAs
coming very soon. Networking these powerful FPGAs us-
ing the HPRC technology, we anticipate significant improve-
ments in hardware solutions to complex ICA algorithms in the
near future.
V. C ONCLUSION
VLSI implementations of ICA algorithms require extremely
efficient hardware designs and sufficient IC resources. In this
paper, we summarized some design guidelines for general ICA
implementations. We also studied several existing solutions in
the categories of analog CMOS, AD mixed signal, ASICs,
and FPGAs. Although each technology has its own charac-
teristics, none of them can balance between a high-density
low-cost design and a shorter turnaround development period.
New development in FPGAs, combined with the HPRC tech-
nology, shows great promise in a comprehensive hardware
solution to complex ICA algorithms. We hope the discussions
here could shed lights on hardware solutions to other complex
algorithms.
DU et al.: COMPARATIVE STUDY OF VLSI SOLUTIONS TO ICA
R EFERENCES
[1] A. Hyvärinen and E. Oja, “A fast fixed-point algorithm for independent
component analysis,” Neural Comput., vol. 9, no. 7, pp. 1483–1492,
Oct. 1997.
[2] M. Bartlett and T. Sejnowski, “Neural information processing systems-
natural and synthetic,” in Viewpoint Invariant Face Recognition Using
Independent Component Analysis and Attractor Networks. Cambridge,
MA: MIT Press, 1997, ch. 9, pp. 817–823
[3] M. Lennon, G. Mercier, M. C. Mouchot, and L. Hubert-Moy, “Indepen-
dent component analysis as a tool for the dimensionality reduction and
the representation of hyperspectral images,” in Proc. SPIE Remote Sens.,
Toulouse, France, Sep. 2001, vol. 4541, pp. 2893–2895.
[4] T. W. Lee, M. S. Lewicki, and T. J. Sejnowski, “ICA mixture models for
unsupervised classification of non-gaussian classes and automatic context
switching in blind signal separation,” IEEE Trans. Pattern Anal. Mach.
Intell., vol. 22, no. 10, pp. 1078–1089, Oct. 2000.
[5] T. W. Lee, M. Girolami, A. J. Bell, and T. J. Sejnowski, “A unifying
information-theoretic framework for independent component analysis,”
Int. J. Math. Comput. Model., vol. 39, no. 11, pp. 1–21, 1998.
[6] J. Herault and J. Jutten, “Space or time adaptive signal processing by
neural network models,” in Proc. AIP Conf. 151 Neural Netw. Comput.,
1986, pp. 206–211.
[7] P. Common, “Independent component analysis, a new concept,” Signal
Proces. (Special Issue on High-Order Statistics), vol. 36, no. 3, pp. 287–
314, Apr. 1994.
[8] R. Linsker, “Local synaptic learning rules suffice to maximize mutual
information in a linear network,” Neural Comput., vol. 4, no. 5, pp. 691–
702, 1992.
[9] Z. Roth and Y. Baram, “Multidimensional density shaping by sigmoids,”
IEEE Trans. Neural Netw., vol. 7, no. 5, pp. 1291–1298, Sep. 1996.
[10] A. J. Bell and T. J. Sejnowski, “An information maximization approach to
blind separation and blind deconvolution,” Neural Comput., vol. 7, no. 6,
pp. 1129–1159, Nov. 1995.
[11] A. Hyvarinen, E. Oja. (1999, Apr.). Independent Component Analy-
sis: A Tutorial. [Online]. Available: http://www.cis.hut.fi/aapo/papers/
IJCNN99\_tutorialweb/
[12] T. M. Cover and J. A. Thomas, Element of Information Theory. Hobo-
ken, NJ: Wiley, 1991.
[13] S. Amari, A. Cichochi, and H. Yang, “A new learning algorithm for
blind signal separation,” in Advances in Neural Information Processing
Systems, vol. 8. Cambridge, MA: MIT Press, 1996.
[14] K. S. Cho and S. Y. Lee, “Implementation of InfoMax ICA algorithm with
analog CMOS circuits,” in Proc. Int. Workshop Independent Compon.
Anal. and Blind Signal Separation, Vancouver, BC, Canada, Dec. 2001,
pp. 70–73.
[15] G. Cauwenberghs. (2003). “Neuromorphic autoadaptive systems and in-
dependent component analysis,” Johns Hopkins Univ., Baltimore, MD.
Tech. Rep. [Online]. Available: http://bach.ece.jhu.edu/gert/yip/
[16] H. Du and H. Qi, “An FPGA implementation of parallel ICA for dimen-
sionality reduction in hyperspectral images,” in Proc. IEEE Int. Geosci.
and Remote Sens. Symp., Sep. 2004, pp. 3257–3260.
[17] D. Bouldin, Developments in Design Reuse. Knoxville, TN: Univ.
Tennessee, 2001.
[18] Y. Mtsuyama, T. Nimoto, N. Katsumata, Y. Suzuki, and S. Furukawa,
“α-EM algorithm and α-ICA learning based upon extended logarithmic
information measures,” in Proc. IEEE-INNS-ENNS Int. Joint Conf. Neural
Netw., Jul. 24–27, 2000, vol. 3, pp. 351–356.
[19] S.-T. Lou and X.-D. Zhang, “Fuzzy-based learning rate determination
for blind source separation,” IEEE Trans. Fuzzy Syst., vol. 11, no. 3,
pp. 375–383, Jun. 2003.
[20] T. Yamaguchi and K. Itoh, “An algebra solution to independent
component analysis,” Opt. Commun., vol. 178, no. 1–3, pp. 59–64,
May 2000.
[21] L. Molgedey and G. Schuster, “Separation of a mixture of independent
signals using time delayed correlations,” Phys. Rev. Lett., vol. 72, no. 23,
pp. 3634–3637, Jun. 1994.
[22] M. H. Cohen and A. G. Andreou, “Analog CMOS integration and experi-
mentation with an autoadaptive independent component analyzer,” IEEE
Trans. Circuits Syst. II, Analog and Digit. Signal Process., vol. 42, no. 2,
pp. 65–77, Feb. 1995.
[23] A. B. A. Gharbi and F. M. A. Salam, “Implementation and test results of
a chip for the separation of mixed signals,” in Proc. IEEE ISCAS, May
1995, pp. 271–274.
[24] T. Serrano-Gotarredona and B. Linares-Barranco, “CMOS transistor mis-
match model valid from weak to strong inversion,” in Proc. Conf. Eur.
Solid-State Circuits, 2003, pp. 627–630.
557
[25] A. Celik, M. Stanacevic, and G. Cauwenberghs, “Mixed-signal real-time
adaptive blind source separation,” in Proc. IEEE ISCAS, Vancouver, BC,
Canada, May 2004, pp. V-760–V-763.
[26] A. Cichocki, R. Unbehauen, L. Moszcnski, and E. Rummert, “A new on-
line adaptive learning algorithm for blind separation of sources,” in Proc.
ISANN, Dec. 1994, pp. 406–411.
[27] A. Cichocki, R. Unbehauen, and E. Rummert, “Robust learning algorithm
for blind separation of signals,” Electron. Lett., vol. 30, no. 17, pp. 1386–
1387, Aug. 1994.
[28] Computational NeuroSystems Laboratory, Digital Implementation of
Independent Component Analysis Algorithm, 2003, Daejeon, South
Korea: Dept. Biosystems, Korea Advanced Inst. Sci. Technol. Tech. Rep.
[Online]. Available: http://cnsl.kaist.ac.kr/Research/kscho/icachip.htm
[29] H. Du, H. Qi, and G. Peterson, “Parallel ICA and its hardware imple-
mentation in hyperspectral image analysis,” in Proc. SPIE Defense and
Security Symp., Apr. 2004, pp. 74–83.
[30] MOSIS. (2003). TSMC 0.18 Micrometer Process, Marina del Rey,
CA: MOSIS Company Tech. Rep. [Online]. Available: http://www.
mosis.org/products/fab/vendors/tsmc/tsmc018/
[31] D. Bouldin, ECE 551: Designing Application-Specific Integrated Circuits,
2001.
[32] ——, “Synthesis of FPGAs and testable ASICs,” in Design of Systems on
a Chip. Norwell, MA: Kluwer, 2003.
[33] H. Youssef and S. M. Sait, VLSI Physical Design Automation, Theory and
Practice. Singapore: World Scientific, Jun. 1999.
[34] A. B. Lim, J. C. Rajapakse, and A .R. Omondi, “Comparative study of
implementing ICNNs on FPGAs,” in Proc. Int. Joint Conf. Neural Netw.,
Jul. 2001, vol. 1, pp. 177–182.
[35] Xilinx, Virtex-E 1.8 V Extended Memory Field Programmable Gate
Arrays, 2002. [Online]. Available: http://direct.xilinx.com/bvdocs/
publications/ds025-1.pdf
[36] A. Nordin, C. Hsu, and H. Szu, “Design of FPGA ICA for hyperspectral
imaging processing,” in Proc. SPIE, Wavelet Appl. VIII, 2001, vol. 4391,
pp. 444–454.
[37] F. Sattar and C. Charayaphan, “Low-cost design and implementation of an
ICA-based blind source separation algorithm,” in Proc. 15th Annu. IEEE
Int. ASIC/SOC Conf., 2002, pp. 15–19.
[38] K. Torkkola, “Blind separation of convolved sources based on information
maximization,” in Proc. IEEE Workshop Neural Netw. Signal Process.,
Kyoto, Japan, Sep. 1996, pp. 423–432.
[39] Y. Wei and C. Charoensak, “FPGA implementation of non-iterative ICA
for detecting motion in image sequences,” in Proc. 7th ICARCV, Dec.
2002, vol. 3, pp. 1332–1336.
[40] C. M. Kim et al., “FPGA implementation of ICA algorithm for blind sig-
nal separation and adaptive noise canceling,” IEEE Trans. Neural Netw.,
vol. 14, no. 5, pp. 1038–1046, Sep. 2003.
[41] P. H. W. Leong et al., Pilchard—A Reconfigurable Computing Platform
With Memory Slot Interface. Hong Kong: Chinese Univ. Hong Kong.
[42] H. Du and H. Qi, “A reconfigurable FPGA system for parallel in-
dependent component analysis,” EURASIP J. Embedded Syst. sub-
mitted for publication. [Online]. Available: http://www.hindawi.com/
RecentlyAcceptedArticlePDF.aspx?journal=ES&number=23025
[43] Silicon Moore’s Law. Intel Corporation, Santa Clara, CA, 2004.
Tech. Rep. [Online]. Available. http://www.intel.com/research/silicon/
mooreslaw.htm
[44] Xilinx, Virtex-II Platform FPGAs: Complete Data Sheet, Mar.
2004. [Online]. Available: http://direct.xilinx.com/bvdocs/publications/
ds031.pdf
[45] M. Alexander, Reconfigurable Computing, Pullman, WA: Washington
State Univ. [Online]. Available. http://www.eecs.wsu.edu/~reconfig/
[46] G. D. Peterson and M. C. Smith, “Programming high performance re-
configurable computers,” in Proc. Int. Conf. Adv. Infrastructure Electroni.
Business, Sci., and Education on the Internet, SSGRR, L’Aquila, Italy,
Aug. 2001, pp. 60–68.
Hongtao Du (S’03) received the B.S. and M.S. degrees in electrical engineering
from Northeastern University, Shenyang, China, in 1997 and 2000, respectively,
and the M.S. degree in computer engineering from the University of Tennessee,
Knoxville, in 2003. He is currently working toward the Ph.D. degree in
electrical and computer engineering at the University of Tennessee.
His current research interests include architectures and design methods for
low-power VLSIs, reconfigurable and virtual platforms, parallel/distributed
image and signal processing, and high-performance computing.
558 IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS, VOL. 54, NO. 1, FEBRUARY 2007

Hairong Qi (S’97–M’99–SM’05) received the B.S. and M.S. degrees in Xiaoling Wang (S’02) received the B.S. and M.S. degrees in electrical en-
computer science from Northern JiaoTong University, Beijing, China, in 1992 gineering from Northeastern University, Shenyang, China, in 1997 and 2000,
and 1995, respectively, and the Ph.D. degree in computer engineering from respectively, and the Ph.D. degree in electrical engineering from the University
North Carolina State University, Raleigh, in 1999. of Tennessee, Knoxville, in 2004.
She is currently an Associate Professor with the Department of Electrical and She is a Senior Engineer with the SONY Advanced Technology Center, San
Computer Engineering, University of Tennessee, Knoxville. She has published Jose, CA. Her current research interests are electronic imaging, information
more than 80 technical papers in archival journals and refereed conference processing in distributed sensor networks, distributed data fusion for target
proceedings. She also coauthored a book on machine vision. Her current tracking and classification, and pattern recognition.
research interests include advanced imaging and collaborative processing in
sensor networks, hyperspectral image analysis, and bioinformatics.
Dr. Qi serves on the Editorial Board of the Sensor Letters and is an Associate
Editor for Computers in Biology and Medicine. She is a recipient of the
National Science Foundation CAREER Award and the Chancellor’s Award for
Professional Promise in Research and Creative Achievement.