1

TIME SERIES FORECASTING

BY USING

WAVELET KERNEL SUPPORT VECTOR MACHINES

LSE Time Series Reading Group

By Ali Habibnia

(a.habibnia@lse.ac.uk)

31 Oct 2012

Outline
2

Introduction to Statistical Learning and SVM

SVM & SVR Formula
Wavelet as a Kernel Function
Study 1:
Forecasting volatility based on wavelet support vector machine,
Written by Ling-Bing Tang, Ling-Xiao Tang, Huan-Ye Sheng

Study 2:

Suggestion for further research + Q&A

Forecasting Volatility in Financial Markets By Introducing a GAAssisted SVR-Garch Model,

Written by Ali Habibnia

SVMs History
3

The Study on Statistical Learning Theory was

started in the 1960s by Vladimir Vapnik. He is
well-known as a founder (together with Professor
Alexey Chervonenkis) of this theory.
He has also developed the theory of the support
vector machines (for linear and nonlinear input
output knowledge discovery) in the framework of
statistical learning theory in 1992.
Prof. Vapnik has been awarded the 2012
Benjamin Franklin medal in Computer and
Cognitive Science from the Franklin Institute.

History and motivation

4

SVMs (a novel ANN) is a supervised learning algorithm for

Pattern Recognition

Regression Estimation Non Parametric (Applications for function estimation started ~ 1995 called
Support Vector Regression)

Remarkable characteristics of SVMs

Good generalization performance: SVMs implement the Structural Risk Minimization Principle
which seeks to minimize the upper bound of the generalization error rather than only minimize
the training error.

Absence of local minima: Training SMV is equivalent to solving a linearly constrained

quadratic programming problem. Hence the solution of SVMs is unique and globally optimal.

It has a simple geometrical interpretation in a high-dimensional feature space

that is nonlinearly related to input space

The Advantages of SVM(R)

5

Based on a strong and nice Theory:

In contrast to previous black box learning approaches, SVMs allow for some intuition
and human understanding.

Training is relatively easy:

No local optimal, unlike in neural network
Training time does not depend on dimensionality of feature space, only on fixed input
space thanks to the kernel trick.

Generally avoids over-fitting:

Trade-off between complexity and error can be controlled explicitly.

Generalize well even in high dimensional spaces under small training set
conditions. Also it is robust to noise.

Linear Classifiers

g(x) is a linear function:

x2

wT x + b > 0

g (x) = wT x + b
A hyper-plane in the feature
space

(Unit-length) normal vector of the

hyper-plane:
n=

w
w

wT x + b < 0

x1

Linear Classifiers
denotes +1

How would you classify these

points using a linear discriminant
function in order to minimize the
error rate?
Infinite number of answers!

x2

denotes -1

x1

Linear Classifiers
denotes +1

How would you classify these

points using a linear discriminant
function in order to minimize the
error rate?
Infinite number of answers!

x2

denotes -1

x1

Linear Classifiers
denotes +1

How would you classify these

points using a linear discriminant
function in order to minimize the
error rate?
Infinite number of answers!

x2

denotes -1

x1

Linear Classifiers
denotes +1

x2

denotes -1

Which one is the best?

x1

Large Margin Linear Classifier

The linear discriminant function

(classifier) with the maximum
margin is the best
Margin is defined as the width
that the boundary could be
increased by before hitting a
data point
Why it is the best?
q

Robust to outliners and thus

strong generalization ability

x2
safe zone

Margin

x1

Large Margin Linear Classifier

Given a set of data points:

{(xi , yi )}, i = 1, 2, L , n
T

For yi = +1, w xi + b > 0

x2
safe zone

Margin
n

For yi = 1, wT xi + b < 0
n

With a scale transformation on

both w and b, the above is
equivalent to
T

For yi = +1, w xi + b 1

For yi = 1, wT xi + b 1

Support Vectors

x1

Large Margin Linear Classifier

We know that
wT x + + b = 1

x2
safe zone

Margin
n

wT x + b = 1
n

The margin width is:

M = (x + x ) n
w
2
= (x + x )
=
w
w

Support Vectors

x1

Large Margin Linear Classifier

x2

Formulation:
maximize

2
w

safe zone

Margin
n

such that
For yi = +1, wT xi + b 1
For yi = 1, wT xi + b 1
Support Vectors

x1

This is the simplest kind of SVM (Called

an LSVM)
x2

Formulation:
1
minimize
w
2

safe zone
2

Margin
n

such that

For yi = +1, wT xi + b 1
For yi = 1, wT xi + b 1

such that
T

yi ( w xi + b) 1

Support Vectors

x1

The Optimization Problem Solution

16

Quadratic
programming
with linear
constraints

minimize

s.t.

1
w
2

yi ( w T xi + b) 1

Lagrangian
Function
1
minimize L p ( w, b, i ) =
w
2

s.t.

i ( yi ( w T xi + b) 1)

i 0

i =1

The Optimization Problem Solution

17

n
1
2
minimize Lp (w, b, i ) = w i ( yi ( wT xi + b) 1)
2
i =1

s.t.

Lp
w
Lp
b

i 0
n

=0

w = i yi xi
i =1

=0

y
i

i =1

=0

The Optimization Problem Solution

18

The Optimization Problem Solution

19

From KKT condition, we know:

x2

i ( yi (wT xi + b) 1) = 0
n
n

Thus, only support vectors have

x+

i 0

The solution has the form:

n

w = i yi xi =
i =1

x+

x-

Support Vectors

T
get
b
from
y
(
w
xi + b) 1 = 0,

y
x

i i i
i
iSV
where xi is support vector

x1

Soft Margin Classification

20

Slack variables i can be added to allow misclassification of

difficult or noisy examples.

11
2

What should our quadratic

optimization criterion be?
Minimize
R
1
w.w + C k
2
k =1

Hard Margin v.s. Soft Margin

21

The old formulation:

Find w and b such that
(w) = wTw is minimized and for all {(xi ,yi)}
yi (wTxi + b) 1

The new formulation incorporating slack variables:

Find w and b such that
(w) = wTw + Ci is minimized and for all {(xi ,yi)}
yi (wTxi + b) 1- i and i 0 for all i

Parameter C can be viewed as a way to control overfitting.

Non-linear SVMs: Feature spaces

22

General idea: the original input space can always be mapped to some
higher-dimensional feature space where the training set is separable:

: x (x)

The Kernel Trick

23

The linear classifier relies on dot product between vectors K(xi,xj)=xiTxj

If every data point is mapped into high-dimensional space via some transformation
: x (x), the dot product becomes:
K(xi,xj)= (xi) T(xj)
n

A kernel function is some function that corresponds to an inner product in some

expanded feature space.

Kernel methods map the data into higher dimensional spaces in the hope that in this
higher-dimensional space the data could become more easily separated or better
structured.

The Kernel Trick

24

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.

This mapping function, however, hardly needs to be computed because of a tool

called the kernel trick.
The kernel trick is a mathematical tool which can be applied to any algorithm which
solely depends on the dot product between two vectors. Wherever a dot product is
used, it is replaced by a kernel function.
Linear Kernel
Polynomial Kernel
Gaussian Kernel
Exponential Kernel
Laplacian Kernel
ANOVA Kernel
Hyperbolic Tangent (Sigmoid) Kernel
Rational Quadratic Kernel
Multiquadric Kernel
Inverse Multiquadric Kernel
Circular Kernel
Spherical Kernel

13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.

Wave Kernel
Power Kernel
Log Kernel
Spline Kernel
B-Spline Kernel
Bessel Kernel
Cauchy Kernel
Chi-Square Kernel
Histogram Intersection Kernel
Generalized Histogram Intersection Kernel
Generalized T-Student Kernel
Bayesian Kernel
Wavelet Kernel

What Functions are Kernels?

25

For some functions K(xi,xj) checking that

K(xi,xj)= (xi) T(xj) can be cumbersome.
n Mercers theorem:
Every semi-positive definite symmetric function is a kernel
n Semi-positive definite symmetric functions correspond to a semi-positive
definite symmetric Gram matrix:
n

K(x1,x1)

K(x1,x2)

K(x1,x3)

K(x2,x1)

K(x2,x2)

K(x2,x3)

K(xN,x1)

K(xN,x2)

K(xN,x3)

K(x1,xN)
K(x2,xN)

K(xN,xN)

Examples of Kernel Functions

26

K (xi , x j ) = xTi x j

Linear kernel:

Polynomial kernel:

Gaussian (Radial-Basis Function (RBF) ) kernel:

K (xi , x j ) = (1 + xTi x j ) p

K (xi , x j ) = exp(
q

Sigmoid:

xi x j
2
T
0 i

K (xi , x j ) = tanh( x x j + 1 )

Nonlinear SVM: Optimization

27

Some Issues
28

Choice of kernel
- Gaussian or polynomial kernel is default
- if ineffective, more elaborate kernels are needed
Choice of kernel parameters
- e.g. in Gaussian kernel
- is the distance between closest points with different classifications
- In the absence of reliable criteria, applications rely on the use of a validation set or
cross-validation to set such parameters.
Optimization criterion Hard margin v.s. Soft margin
- a lengthy series of experiments in which various parameters are tested

Support vector regression

29

Maximum margin hyperplane only applies to classification

However, idea of support vectors and kernel functions can be used for
regression
Basic method is the same as in linear regression: minimize error
Difference A: ignore errors smaller than e and use absolute error instead
of squared error
Difference B: simultaneously aim to maximize flatness of function
User-specified parameter e defines tube

Ordinary Least Squares (OLS)

Solution:

f(x)

Loss = (wX + b ) Y

f (x) = wx + b

dLoss
=0
dw

(X
x

X w = X TY

Support Vector Regression (SVR)

Solution:

f(x)

f (x) = wx + b

+
0
-

Min

1 T
w w
2

Constraints:

yi wT xi b

wT xi + b yi

Support Vector Regression (SVR)

Minimise:N

f(x)

f (x) = wx + b

+
0
-

1 T
w w + C i + i*
2
i =1

Constraints:
T

yi w xi b + i

wT xi + b yi + i*

i , i* 0

*
x

Lagrange Optimisation
N
1 T
L = w w + C i + i*
2
i =1

Target

i + i yi + wT xi + b
i =1
N

i* + i* yi + wT xi + b
i =1
N

i i + *i i*
i =1

Regression:

)
)

y (x ) = i i* xi , x + b
i =1

Constraints

Nonlinear Regression
f(x)

f(x)

+
0
-

+
0
-

(x)

Regression Formulas
N

Linear:

i =1

Nonlinear:
General:

y (x ) = i i* (xi ), (x ) + b
i =1

y (x ) = i i* xi , x + b

y (x ) = i i* K (xi , x ) + b
i =1

Kernel Types
l
l

Linear:

K ( x, xi ) = x, xi

Polynomial:

K ( x, xi ) = x, xi d

Radial basis function:

x
i

K ( x, xi ) = exp
2

Exponential RBF:

x xi

K ( x, xi ) = exp

Wavelet Kernel
37

The Wavelet kernel (Zhang et al, 2004) comes from Wavelet theory and is given as:

Where a and c are the wavelet dilation and translation coefficients, respectively (the form
presented above is a simplification). A translation-invariant version of this kernel can be
given as:

Where in both h(x) denotes a mother wavelet function. i.e:

A simple View of Wavelet Theory

38

Fourier Analysis
n

Breaks down a signal into constituent sinusoids of different

frequencies

In other words: Transform the view of the signal from timebase to frequency-base.

A simple View of Wavelet Theory

39

Whats wrong with Fourier?

n
n
n

By using Fourier Transform , we loose the time information :

WHEN did a particular event take place ?
FT can not locate drift, trends, abrupt changes, beginning
and ends of events, etc.
Calculating use complex numbers.

Wavelets vs. Fourier Transform

In Fourier transform (FT) we represent a signal in
terms of sinusoids
FT provides a signal which is localized only in the
frequency domain
It does not give any information of the signal in the
time domain

40

Wavelets vs. Fourier Transform

Basis functions of the wavelet transform (WT) are
small waves located in different times
They are obtained using scaling and translation of a
scaling function and wavelet function
Therefore, the WT is localized in both time and
frequency

41

Wavelets vs. Fourier Transform

If a signal has a discontinuity, FT produces many
coefficients with large magnitude (significant
coefficients)
But WT generates a few significant coefficients
around the discontinuity
Nonlinear approximation is a method to benchmark
the approximation power of a transform

42

Wavelets vs. Fourier Transform

In nonlinear approximation we keep only a few significant

coefficients of a signal and set the rest to zero
Then we reconstruct the signal using the significant
coefficients
WT produces a few significant coefficients for the signals
with discontinuities
Thus, we obtain better results for WT nonlinear
approximation when compared with the FT
43

Wavelets vs. Fourier Transform

Most natural signals are smooth with a few discontinuities

(are piece-wise smooth)
Speech and natural images are such signals
Hence, WT has better capability for representing these
signal when compared with the FT
Good nonlinear approximation results in efficiency in
several applications such as compression and denoising
44

Study 1: Forecasting volatility based on wavelet support vector

machine, Ling-Bing Tang, Ling-Xiao Tang, Huan-Ye Sheng
45

combine SVM with wavelet theory to construct a

multidimensional wavelet kernel function to predict the
conditional volatility of stock market returns based on GARCH
model.
General Kernel function in SVM cannot capture the cluster
feature of volatility accurately.
wavelet function yields features that describe of the volatility
time series both at various locations and at varying time
granularities.
The prediction performance of SVM is greatly dependent upon
the selection of kernel functions.
j and k denote the dilation and
translation

Study 1: Forecasting volatility based on wavelet support vector

machine, Ling-Bing Tang, Ling-Xiao Tang, Huan-Ye Sheng
46

Study 1: Forecasting volatility based on wavelet support vector

machine, Ling-Bing Tang, Ling-Xiao Tang, Huan-Ye Sheng
47

Study 1: Forecasting volatility based on wavelet support vector

machine, Ling-Bing Tang, Ling-Xiao Tang, Huan-Ye Sheng
48

Study 2: Forecasting Volatility in Financial Markets By Introducing

a GA-Assisted SVR-Garch Model, Ali Habibnia
49

No structured way being available to choose the free

parameters of SVR and kernel function, these parameters are
usually set by researcher in trial and error (Grid Search and
Cross Validation) procedure which is not optimal.
In this study a novel method, named as GA assisted SVR has
been introduced, which a genetic algorithm simultaneously
searches for SVRs optimal parameters and kernel parameter
(in this study: a radial basis function (RBF)).
The SVM(R) tries to get the best fit with the data, not relying on
any prior knowledge, and it only concentrates on minimizing the
prediction error with a given machine complexity.

FTSE 100 index from 04

Jan 2005 to 29 Jun 2012.

Study 2: Forecasting Volatility in Financial Markets By Introducing

a GA-Assisted SVR-Garch Model, Ali Habibnia
50

51

Suggestion for further research + Q&A

52

Thanks for your patience ;)

LSE Time Series Reading Group

By Ali Habibnia
a.habibnia@lse.ac.uk
31 Oct 2012