42nd South Eastern Symposium on System Theory
4
University of Texas at Tyler
Tyler, TX, USA, March 7-9, 2010
Fault Tolerant Block Based Neural Networks
Sai sri Krishna Haridass and David H. K. Hoe
Electrical Engineering Department
University of Texas at Tyler
Tyler, TX 75799 USA
Abstract: Block Based Neural Networks
(BBNNs) have shown to be a practical means for
implementing evolvable hardware on
reconfigurable fabrics for solving a variety of
problems that take advantage of the massive
parallelism offered by a neural network approach.
This paper proposes a method for obtaining a fault
tolerant implementation of BBNNs by using a
biologically inspired layered design. At the lowest
level, each block has its own online detection and
correcting logic combined with sufficient spare
components to ensure recovery from permanent
and transient errors. Another layer of hierarchy
combines the blocks into clusters, where a
redundant column of blocks can be used to replace
blocks that cannot be repaired at the lowest level.
The hierarchical approach is well-suited to a
divide-and-conquer approach to genetic
programming whereby complex problems are
subdivided into smaller parts. The overall
approach can be implemented on a reconfigurable
fabric.
Index terms - Reconfigurable logic, Block
based network, Fault detection and correction
I. Introduction
Artificial Neural Networks (ANNs), modelled
after the biological neural networks of the brain,
have shown the ability to solve a variety of
complex problems due to their unique features
such as massive parallelism with inherent
concurrency. This makes them well suited to
efficiently solve problems such as image and
pattern recognition and data prediction. Software
implementations running on sequential machines
fail to take advantage of the inherent parallelism of
ANNs. Electronic implementations of ANNs face
several challenges. First, the trial and error based
training algorithms for obtaining an optimal
network topology requires frequent changes to the
network structure and parameters such as synaptic
978-1-4244-5692-5/10/$26.00 © IEEE 2010
T2B.
weights and biases. Due to this issue, training
algorithms are usually run first in software prior to
the hardware implementation. A reconfigurable
neural network allows topological changes to be
programmed into the circuit allowing an optimal
topology to be found completely at the hardware
level [1]. Second, implementing ANNs of any
practical size requires a large number of synaptic
interconnections. Block based neural networks
(BBNNs) have been proposed as a means for
reducing this interconnect complexity. Together
with their regular structure, BBNNs are suitable
for implementation on reconfigurable fabrics, such
as FPGAs [5].
ANNs implemented with advanced processing
technology and operating in harsh environments
must be fault tolerant. The continued scaling of
device sizes increases the defect density and
makes the scaled devices more vulnerable to
transient faults due to external influences such as
electromagnetic interference. Thus a totally self-
checking system is required where faults are
automatically detected and corrected. This paper
proposes a architecture for implementing a block
based reconfigurable fault tolerant neural network.
A system based on the proposed architecture is
composed of a number of functional cells that are
interconnected to perform a desired function. The
error detection in the BBNNs is performed in
every block. Once a faulty cell is identified, a
spare block replaces it. A hierarchical approach
arranges the blocks into clusters, with a sufficient
number of spare blocks located in each cluster to
ensure the required level of fault tolerance. This
self-healing mechanism is inspired by complex
biological systems which often are able to remain
operational while sustaining certain levels of
damage [2, 3].
This paper, is organized as follows. First, the
basic concepts of the block based neural network
and its relationship to a genetic system are
357
reviewed. Then the fault tolerant design of the
BBNN system is described.
II. Block-Based Neural Network
A Block-Based Neural Network (BBNN) is a
two-dimensional array of basic neural network
blocks interconnected in nearest neighbor fashion.
Figure 1 depicts the structure of the basic neural
network block. Every block in the network has
four nodes, with each of these nodes acting as
either an input or an output of the block. This
block forms the essential processing element of
the BBNN [4].
Fig. 1. Basic Unit of BBNN
The structure of the BBNN defines the signal
flow between blocks, which in turn automatically
determines the internal configuration or input-
output connections of the basic blocks. Depending
on the flow of data between blocks, four types of
internal configurations are needed, as shown in
Figure 2. Each individual neuron block computes
outputs that are functions of the summation of
weighted inputs and a bias, essentially forming a
simple feedforward neural network [5].
Fig. 2. Four different internal configurations of a
BBNN block (a) 1/3,(b) 3/1,(c) 2/2, (d) 2/2
The output vj of each block is computed as:
358
vj h
= (∑ i∈I )
wij ui + b j , j ∈ J
wij = synaptic weight between ui - vj
ui = input signal of a block
bj =bias of the jth node
The indices i and j denote the input and output
nodes, respectively. The activation function h(•)
can be a linear or a nonlinear function. Figure 3
depicts a cluster of neurons. The column to the
right is a column of spare neurons. The overall
structure of the BBNN defines the signal flow
represented by the arrows between the blocks [5].
Fig. 3. A cluster of blocks with spare cells
III Analogy between BBNN and Biological
Systems
In biological systems, the gene is the basic
unit of heredity in a living organism [7].
Composed of sequences of DNA, genes contain
the information that encodes different biological
traits. The totality of an organism’s hereditary
information is found in its genome, which consists
of genes and the non-coding sequences of DNA. A
computational metaphor is appropriate since the
genome can be viewed as a computer’s operating
system, while the genes are the “individual
subroutines in the overall system that are
repetitively called in the process of transcription”
[6]. At an intermediate level is the chromosome,
which consists of many genes and other regulatory
and nucleotide sequences.
The BBNN referred to in this paper is quite
similar to a biological system in its structure and
functioning. The configuration bits used to specify
the synaptic weights, internal structure, and
intercellular connections can be considered to
encode the basic genetic information of each block
of the BBNN. Each such block can be considered block. The main parts of this cellular block are the
to form a cell whose set of genetic information is weight table, central logic unit, self test logic unit,
contained within a chromosome. At the top level finite state machine control logic unit, input and
of the hierarchy, the complete set of weights and output nodes and their interconnections.
structural information corresponds to the genome. The input nodes are connected to the central
Each cell of the BBNN forms an autonomous logic unit via a multiplier whose other input comes
system capable of self-healing and adaptive from the weight table, which contains the synaptic
change. weight information previously obtained from
The biological model of the BBNN has two running a GA. There are three multipliers per
important implications for its operation. First, by block, two main multipliers (M1 and M2) and a
encoding all the genetic information for each spare unit (Mspare). A block must perform at
block within a chromosome, evolutionary most four multiplications at any given processing
algorithms (EA) can be used to simultaneously step (configuration 2/2), so two clock cycles are
find an optimal solution for the internal structure required to process the data for each block. The
and weight of the BBNN. The use of EAs for use of the spare multiplier plus a time redundant
solving some basic problems with BBNNs have approach forms the basis of our fault tolerant
demonstrated the feasibility of this approach [4, approach.
5]. Second, the biologically-inspired model leads
to a viable layered-based approach to fault tolerant
design [2]. At the lowest level of the cell, self-
correcting logic is used to detect and correct
errors. The next layer consists of a clusters of
cells, which can invoke spare cells in the event the
self-healing ability at the cellular level fails.
Another layer can similarly be formed with self-
healing capability, analogous to cells forming
tissues and organs in biological systems. This
hierarchical arrangement is well-suited to a divide-
and-conquer approach to solving complex
problems where multiple genetic algorithms (GAs)
can be run on one system [9].
IV. Fault Tolerant Mechanism
A fault tolerant system has the ability to detect
and correct errors. On-line fault tolerance is of
prime importance for mission critical systems such
as those operating in aircraft or those operating in
harsh environments (e.g., satellites).
Faults can be classified into two types:
processing faults and run-time faults. Processing
faults occur during production of the system. For
example, the defect density of a silicon chip is due
to the processing faults, which directly impacts the Fig. 4. Functional representation of a Basic Unit of the
yield of the chip. Run-time errors are those errors BBNN
which take place because of some internal or A. Error Checking Algorithm
external perturbations during dynamic operation of
the system. Depending on the region and type of Before the whole system starts processing any
impact the faults can either be permanent or data, a power-on self-test takes place which is
transient. monitored by the self test logic. This test checks
In this paper we propose a structure shown in all three multipliers in every block in the network.
Figure 4 which can tolerate up to one error per Hardware (several multipliers working in parallel)

359
and temporal (repeated attempts with varied previous step (i.e., give M1 and M2 the same input
inputs) redundancy are used to ensure that all the and weight and carry on with their output).
components of the cellular block are functional.
The structure of the BBNN allows for efficient
implementation of these standard approaches for
achieving system fault tolerance. Error checking
codes (such as Hamming codes) will be used to
ensure the integrity of the configuration bits. If the
block is initially found to be faulty, it can
immediately be disabled, and replaced with a spare
block as described below.
Once the self-test is complete, we can have
confidence that every block in the BBNN is
initially error free. The algorithm shown in Figure
7 is then followed for online fault detection during Fig. 5. Evaluation when none of the outputs of the
operation of the BBNN. Following this algorithm multiplier are same.
both inputs arriving at the central control logic are If a match fails from the neighboring neurons,
evaluated using a comparator. For simplicity M1 is then the fault is a permanent one and needs to be
considered the main multiplier and Mspare is the corrected by replacing the faulty block with a
spare multiplier. (M1 and M2 are checked by spare block. This is achieved by bypassing the
Mspare on alternating pairs of clock cycles.) column with the faulty block and invoking the
If both multiplier outputs are equal then it is spare column in the cluster as shown in Figure 6.
assumed that there is no error, the product is
passed to the output, and the Error_count value is
decreased by 1 (if the Error_count value is greater
than zero). The Error_count provides some degree
of history to allow for recovery from transient
errors and to ensure that a block is not prematurely
removed from the system.
If the two multipliers differ in value, then
Error_count is incremented by two, the system is
stalled with the existing values, and the product is
recalculated using the other multiplier in the block Fig. 6(a) Block b2 is a faulty block in the cluster.
(in this case, the one labeled M2). The outputs of
the first two multipliers (M1 and Mspare) are
compared with the output of the new multiplier
M2. If either of the initial two multiplier values
agrees with multiplier M2, this value will be
assigned to the output node.
In the case that M2 does not agree with either
of M1 or Mspare, the multipliers of the
neighboring blocks are invoked. This part of the
logic is shown in Fig. 5. The outputs of multipliers
M2 and M1 of the neighboring blocks are labeled
Nb1_m2_out and Nb2_m1_out, respectively. Fig. 6(b) Blocks b2 and c2 are shifted right with
These outputs enter the block evaluator along with their respective columns to correct the fault.
M1_out and M2_out of the current block for #B_enable is the block enable signal
further comparison. If M1_out of the current block #M1_out is the output of multiplier M1
is equal to Nb1_m2_out and M2_out of current #M2_out is the output of multiplier M2
#Mspare_out is the output of multiplier Mspare
block is equal to Nb2_m1_out then the fault is a
#N_input is the input of the output node
transient one so we neglect the fault and repeat the

360
 #Error_count is the value of the error counter V. Summary and Current Work
#Temp_input is the temporary signal which holds
the output of the neuron temporarily A fault tolerant BBNN using a biologically
inspired hierarchical approach has been
described. Efficient online detection and
Threshold= 15 correction of errors at the cellular and cluster
B_enable=1 level is achievable, resulting in several levels of
redundancy. This should yield robust BBNN
designs suitable for implementation on
if (M1_out = = Mspare_out) reconfigurable nanoscale fabrics. Currently we
Temp_input = M1_out;
are implementing the proposed design on a
if (Error_count > 0)
Error_count = Error_count – 1; Xilinx Virtex 5 FPGA as a proof of concept and
end if will present the results at the conference.
else REFERENCES
Error_count = Error_count + 2;
[1] S. Satyanarayana and Y. P. Tsividis, “A
Stall the system. Reconfigurable VLSI Neural Network,” IEEE J.
Recalculate the product using the multiplier M2. of Solid-State Circuits, vol. 27, no. 1, pp. 67-81,
do Jan. 1992.
{ [2] W. Baker, D. M. Halliday, Y. Thoma, E
check_parameter = 1 Sanchez, G. Tempesti, A. M. Tyrrell, “Fault
if (Mspare_out == M2_out) Tolerance Using Dynamic Reconfiguration on
Mask M1_out the POEtic Tissue,” IEEE Trans. on
Temp_input = Mspare_out; Evolutionary Computation, vol. 11, no. 5, pp.
elseif(M1_out = = M2_out) 666-684, Oct. 2007.
Mask Mspare_out [3] P. K. Lala and B. K. Kumar, “Human immune
Temp_input =M1_out; system inspired architecture for self-healing
else digital systems,” Proceedings, International
Error_count=Error_count +5; Symposium on Quality Electronic Design, pp.
292-297, 2002.
Invoke the neighboring neurons and use [4] S. Merchant, G. D. Peterson, S. Ki Park, S.G.
their multipliers to evaluate if M1 or M2 Kong, “FPGA Implementation of Evolvable
or both multipliers of the current block are Block-based Neural Networks,” IEEE Congress
correct. on Evolutionary Computation, pp. 3129-3136,
if (Nb1_m2_out == M1_out && 2006.
Nb2_m1_out==M2_out) [5] Sang-Woo Moon, Seong-Gon Kong, “Block-
then {fault is a transient fault, so neglect it based neural networks,” IEEE Trans. on Neural
and repeat the previous step of evaluation} Networks, vol. 12, no. 2, pp. 307-317, Mar 2001.
else [6] M. B. Gerstein, C. Bruce, J. S. Rozowsky, D.
check_parameter=0; Zheng, J. Du, et al., “What is a gene, post-
ENCODE? History and updated definition,”
At this point it implies that there is more
Genome Research, vol. 17 (6) 669–681, 2007
than a single error in the neuron block;
(citation on p. 671).
thus we need to disable the block and
replace it with a spare block by shifting [7] D. Noble, “Genes and causation,” Philosophical
the whole column one unit right. Trans. of Royal Society: Series A, vol. 366, pp.
3001–3015, 2008.
end if [8] Y. Jewajinda and P. Chongstitvatana, “FPGA
end if implementation of a cellular univariate
} while (check_parameter==0) estimation of distribution algorithm and block-
based neural network as an evolvable hardware,”
end if IEEE Congress on Evolutionary Computation,
pp. 3366-3373, June 2008.
N_input = Temp_input; [9] E. Cantu-Paz, Efficient and accurate parallel
genetic algorithms, Springer, 2000.
Fig. 7. Error Detection and Correction Algorithm

361