MULTIDIMENSIONAL FUNCTION APPROXIMATION USING NEURAL

NETWORKS

Enăchescu Călin
“Petru Maior” University of Targu Mures, ROMANIA
ecalin@upm.ro

Abstract: Solving a problem with a neural network a primordial task is establishing the
network topology. Generally neural network topology determination is a complex problem
and cannot be easily solved. When the number of trainable layers and processor units is too
low, the network is not able to learn the proposed problem. When the number of layers and
neurons is too high, the learning process becomes too slow. Learning from examples means
being able to infer the functional dependence between input and output spaces X and Z, given
the knowledge of the set of examples T. It means that, after we have “learned” N examples,
when a new input variable x comes in, we need to be able to estimate, according to some
criterion that we will specify, a corresponding value of z. From this point of view learning is
equivalent to a function approximation.

Key words: Neural networks, approximation, learning

I. INTRODUCTION

From 1986 the most popular neural network model is the Multi Layer Perceptron
(MLP) and the most popular learning algorithm is the Back-propagation (BP) method [7]. In
spite of the fact that the classical MLP networks have many advantageous properties, they
have some disadvantages, too. The most important disadvantage is the slowness of the
training procedure, caused by the high number of the trainable layers and the necessity of
error back-propagation. The training process could be faster if the number of trainable layers
can be diminished. That was the motivation for developing neural networks with a single
trainable layer.
Radial basis functions were first introduced in the solution of the real multivariate
interpolation problem [10]. Broomhead and Lowe (1988) [1] were the first to exploit the use
of radial basis functions in designing neural networks. Other major contributions to the
theory, design and application of RBF networks include papers by Moody and Darken (1989)
[8] and Poggio and Girosi (1990) [9].

II. RBF NEURAL NETWORKS TOPOLOGY.

RBF is a feed-forward neural network with an input layer (made up of source nodes:
sensory units), a single hidden layer and an output layer [2]. The network is designed to

VI-17-1
perform a nonlinear mapping from the input space to the hidden space, followed by a linear
mapping from the hidden space to the output space.

The processor units of the hidden layer are different from the processor units of the
MLP networks. The activation functions are radial basis functions (for example Gaussian
functions). These functions generally have two parameters: the center and the width [3].

The output layer is composed by processor units, which are creating normal, simple
linear-weighted value, every unit producing an output. The network has a typical property: the
value of weights between the input and the hidden layer is 1.

The architecture of the RBF neural network is presented in Figure.1 [4].

g1
x1 w1

xi gi wi

xn wK

gK

Figure 1: RBF neural network topology.

If x = (x1 ,..., xn ) is the input vector, g () is the Radial Basis Function and c i is the center
parameter for the function corresponding to neuron i, then the output created by the network
will be:
K K
y = ∑ wi g i (x) = ∑ wi g ( x − c i ) (1)
i =1 i =1

Generally Gaussian function is used:

2
x −c i
−
2σ i2
g i ( x) = e (2)

where σ i is the scale parameter for function corresponding to neuron i.

There are some methods to select the parameters ( c i , σ i ) of the activation function. If
few training points are present, then all of them could be used as center parameter. In this case
the number of the processor units in the hidden layer is equal with the number of training
points. If the number of training points is high, then not all of them might be used. In this

VI-17-2
situation a single neuron for a group of similar training points can be considered. These
groups of similar training points can be identified using clustering methods [5].

III. LEANING STRATEGIES FOR RBF NEURAL NETWORKS

The hidden layer of the RBF Neural Networks may be trained with a supervised learning
algorithm. A descendent gradient-based algorithm can be considered. The aim is to establish
the synaptic weights wi, i = 1,2,…,K of the network.
Let

{
T = (x i , z i ) x i ∈ R n , z i ∈ R, i = 1,2,K, N } (3)

be the set of the training samples.

A clustering algorithm is used on the points of the set T. The cluster centers ci,
i = 1,..., K are considered (in this way the number of the neurons in the hidden layer is K).

Parameters σ i ∈ R , i = 1,..., K can be determined corresponding to the diameter of

clusters. This step is not executed when K is equal with N (K=N), because in this case ci = xi,
i = 1,..., N (every training point is a cluster center too and the value of the width parameters is
σi = 1/N).
If Gaussian function is used as activation function, then at the lth step the global
learning error is
1 N
El = ⋅ ∑ ( z i − yi ) 2 (4)
N i =1
where
( xi −c j ) 2
K −
yi = ∑ w j ⋅ e
2σ 2j
, i = 1,..., N (5)
j =1

Let us to note:
∂E
∆wi = −η ⋅ , i = 1,..., K (6)
∂wi
where η is the learning rate and E is the global learning error.

Weights updating is based on the following correction rule:

wi = wi + ∆wi , i = 1,..., K (7)

When the learning process is finished, M points, which are not from the training set T,
are randomly generated. The corresponding generalization error is defined by the expression
[5]:

VI-17-3
 1 M
Eg = ⋅ ∑ ( zi − yi ) 2 (8)
M i =1

IV. APPROXIMATION AND INTERPOLATION WITH RBF NEURAL NETWORKS

The interpolation problem, in its strict sense, may be stated as follows:

{ }
Given a set of N different points xi ∈ R p | i = 1,..., N and a corresponding set of N real
numbers {d i ∈ R | i = 1,..., N }, find a function F : R → R that satisfies the interpolation
p

condition [2], [6]:

F ( xi ) = d i , i = 1,..., N (9)

The RBF technique consists of choosing a function F that has the following form [10]:
N
F ( x ) = ∑ wi g ( x − xi ) (10)
i =1

where

{g ( x − x ) | i = 1,..., N }
i (11)

is a set of N arbitrary radial basis functions. The known data points xi ∈ R p , i = 1,..., N are
taken to be the centers of the radial basis functions.
A RBF network is considered, with a single processor unit in the output layer, and N
processor units in the hidden layer, where {g ( x − xi ) | i = 1,..., N } is the set of the activation
functions for the hidden processor units. The interpolation problem is reduced to the
determination of weights (learning process) [3].
In an overall fashion, the network represents a map from the p-dimensional input
space to the single-dimensional output space, written as:

s:Rp → R (12)

The map s could be considered as a hypersurface Γ ⊂ R p +1 . The surface Γ is a

multidimensional plot of the output as a function of the input.
In a practical situation, the surface Γ is unknown and the training data are usually
affected by noise. Accordingly, the training phase and generalization phase of the learning
process may be respectively viewed as follows [1], [4]:
- The training phase constitutes the optimization of a fitting procedure for the surface
Γ , based on known data points presented to the network in the form of input-output
examples.

- The generalization phase is a synonymous with interpolation between the data points,
with the interpolation being performed along the constrained surface generated by the fitting
procedure as the optimal approximation to the true surface Γ .

VI-17-4
 V. NUMERICAL EXPERIMENTS.

In this section some experiments and the obtained results are presented. Standard
interpolation problems are considered. RBF neural networks are used for approximating
functions.
The generalized k-means clustering algorithm is used for data clustering and some
comparisons are presented [3].
Experiment: In order to study the properties of the RBF networks obtained as a theoretical
result, we have implemented this type of neural network and studying the learning capabilities
and the generalization capabilities. We have taken in consideration as target function, to be
approximated, the following function:

f : R 2 → R, f ( x, y ) = 10 ⋅ cos( x ) ⋅ sin ( y ) (13)

Fig. 2.: 400 training data, 10 learning epochs. Fig. 3.: 400 training data, 50 learning epochs.

Fig. 4.: 400 training data, 100 learning epochs. Fig. 4.: 400 training data, 500 learning epochs.

VI-17-5
Number 400 training data

of
Learning Error Generalization Error
epochs

10 0.1039422 0.1048457

50 0.0223206 0.0214215

100 0.0127233 0.0127233

500 0.0022711 0.0020923

1000 0.0016508 0.0016481

Table1: Results of the simulations, describing number of epochs, learning error and
generalization error.

VI. CONCLUSIONS

Experiments described in this chapter demonstrate that RBF neural networks can be
successfully used for multidimensional function approximation.

REFERENCES
[1] Broomhead D.S., Lowe D., (1988), Multivariable functional interpolation and adaptive
networks, Complex Systems 2, 321-355.
[2] Enăchescu, C. (1995), Properties of Neural Networks Learning, 5th International
Symposium on Automatic Control and Computer Science, SACCS'95, Vol. 2, 273-278,
Technical University "Gh. Asachi" of Iasi, Romania.
[3] Enăchescu, C. (1996), Neural Networks as approximation methods. International
Conference on Approximation and Optimisation Methods, ICAOR'96, " Babes-Bolyai
University ", Vol. 2., 83-92, Cluj-Napoca.
[4] Enăchescu, C., (1995), Learning the Neural Networks from the Approximation Theory
Perspective. Intelligent Computer Communication ICC'95 Proceedings, 184-187, Technical
University of Cluj-Napoca, Romania.
[5] Enăchescu, C. (1998), The Theoretical fundamentals of Neural Computing, Casa Cărţii de
Ştiinţă, Cluj-Napoca. (in Romanian).
[6] Girosi, F., T. Pogio (1990), Networks and the Best Approximation Property. Biological
Cybernetics, 63, 169-176.
[7] Haykin, S., (1994), Neural Networks: A Comprehensive Foundation, Macmillan College
Publishing Company, New York, NY.
[8] Moody J., Darken C., (1989), Fast learning in networks of locally tuned processing units,
Neural Computation, 1, 281-294.
[9] Poggio T., Girosi F., (1990), Networks for approximation and learning, Proceedings of the
IEEE 78, 1481-1497.
[10] Powell M.J.D., (1988), Radial basis function approximations to polynomials, Numerical
Analysis 1987 Proceedings, 233-241.

VI-17-6