Print de Waal Andre

Generalized
Additive Neural
Network Modeling
André de Waal
Centre for Business Mathematics
and Informatics
North-West University
South Africa
Copyright © 2007, SAS Institute Inc. All rights reserved.
Agenda
Motivation
Models
Timeline
Interactive construction algorithm
Automated construction algorithm
Genetic algorithm
Support vector machines
Success story
Copyright © 2007, SAS Institute Inc. All rights reserved. 1

Motivation
Approximately 85% of all companies use

logistic regression as the main modeling
tool in the development of their
scorecards
• LC Thomas
• Credit and Behavioral Scoring
Workshop
• Credit Scoring and Credit Control IX
• Edinburgh, Scotland, 2005
Models
Linear models
E ( yi ) = β 0 + β1 x1i + β 2 x2i + L + β k xki
Generalized linear models
g 0−1 ( E ( yi )) = β 0 + β1 x1i + β 2 x2i + L + β k xki
• The link function g0-1 constrains the
range of the predicted values
• Logit link: [0, 1] (used for probabilities)
• Tanh link: [-1, 1]

Models
Generalized additive models

g 0−1 ( E ( yi )) = β 0 + f1 ( x1i ) + f 2 ( x2i ) + L + f k ( xki )
Multilayer perceptrons
⎛ k ⎞
g 0 ( E ( yi )) = w0 + w1 tanh⎜ w01 + ∑ w j1 x ji ⎟⎟ + L
−1
⎜
⎝ j =1 ⎠
⎛ k ⎞
+ wh tanh⎜⎜ w0 h + ∑ w jh x ji ⎟⎟
⎝ j =1 ⎠
Timeline
NN course
W Potts
2004 2005 2006 2007
W Sarle

1. Construct a GANN with 1 neuron and a

skip layer for each input.
2. Fit a generalized linear model.
3. Initialize the remaining parameters as
random values from a normal
distribution.
4. Fit the full GANN model.
5. Examine each partial residual plot.
6. Prune and add neurons. Repeat.


non-trivial process
intelligent behavior
Knowledge discovery in databases (KDD)

is the non-trivial process of identifying
valid, novel, potentially useful, and
ultimately understandable patterns in data
(Fayyad, Piatetsky-Shapiro & Smyth,
1996).
Artificial intelligence (AI) may be defined
as the branch of computer science that is
concerned with the automation of
intelligent behavior (Luger, 2002).

Timeline
KDDCup 2004
NN course Alan Russell

W Potts Cary
2004 2005 2006 2007
W Sarle Version 1.0

SAS Award
SASA 2003
Best-first search

Architecture notation
0 1 2 3 4
5 6 7 8 9
Architecture transformation
0 1 2 3 4
5

AutoGANN modeling node SAS®

Enterprise Miner™
AutoGANN user interface

Timeline
KDDCup 2004 SAS Forum 2006
CSCCIX 2005
NN course Alan Russell Version 5.2
W Potts Cary
2004 2005 2006 2007
W Sarle Version 1.0

SAS Award Windows NT
SASA 2003 Ph.D.
Improved best-first search (multiple

changes)

Intelligent start (bootstrap)
0 1 3
? ? ?
Ozone example

Inputs
Solution Path

Model Space Statistics (10 077 696)

Solution Path (KDD-Cup 2004)

Model Space Statistics (6.4 * 1056)

AutoGANN node functionality:

• Generalized additive models
−Estimate a specific model
−Search for best model
−Generate partial residual plots
−Perform model averaging
• Linear and logistic regression models
• Variable selection
Transform Variables + Regression ≡

GAM?
≡ ?

GANN scorecard
• Replace with
• Consider univariate functions as

transformations
−Keep logistic regression
Benefits of presented approach

• Not a black box as general artificial
neural networks
• Gives insight into relationships between
input variables and target
• Potential to reduce scorecard building
time
• Can be used to evaluate current
models’ performance

Genetic algorithm
1. Weak encoding scheme

• 102003313
2. Mutation
• 111111111 → 111111511
3. Unit-crossover
• 101111111,111111511 → 101111511
4. Fitness function
• SBC
Support vector machines
SVM GANN
Non-linear
Yes Yes
classifier
Maximal
Yes No
separation
Indifferent to
No Yes
kernel choice
Interpretable
No Yes
results

Success story
South African bank

• Windows NT
• Ran AutoGANN for 1 hour
• Improved on a current model
• Request to purchase AutoGANN within
24 hours
Timeline
KDD-Cup 2004 SAS Forum 2006
CSCCIX 2005
NN course Alan Russell Version 5.2
W Potts Cary M2007
2004 2005 2006 2007
W Sarle Version 1.0 ICDM’07

SAS Award Windows NT
SASA 2003 Ph.D.

Questions
Can an AI search engine and optimization

algorithm compete in outright speed with
classical approaches? Probably not!
algorithm compete in accuracy with
classical approaches? Definitely!
algorithm compete in total modeling
time with classical approaches?
Definitely!

Print de Waal Andre

Transféré par

Informations du document

Description originale:

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Print de Waal Andre

Transféré par

Droits d'auteur :

Formats disponibles

Generalized

Copyright © 2007, SAS Institute Inc. All rights reserved. 1

 Approximately 85% of all companies use

Copyright © 2007, SAS Institute Inc. All rights reserved. 2

 Generalized additive models

Copyright © 2007, SAS Institute Inc. All rights reserved.

2004 2005 2006 2007

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved. 3

1. Construct a GANN with 1 neuron and a

Interactive construction algorithm

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved. 4

Copyright © 2007, SAS Institute Inc. All rights reserved.

Interactive construction algorithm

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved. 5

Copyright © 2007, SAS Institute Inc. All rights reserved.

Interactive construction algorithm

 Knowledge discovery in databases (KDD)

Copyright © 2007, SAS Institute Inc. All rights reserved. 6

NN course Alan Russell

2004 2005 2006 2007

W Sarle Version 1.0

Copyright © 2007, SAS Institute Inc. All rights reserved.

Automated construction algorithm

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved. 7

Automated construction algorithm

Copyright © 2007, SAS Institute Inc. All rights reserved. 8

 AutoGANN modeling node SAS®

Copyright © 2007, SAS Institute Inc. All rights reserved.

Automated construction algorithm

 AutoGANN user interface

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved. 9

2004 2005 2006 2007

W Sarle Version 1.0

Copyright © 2007, SAS Institute Inc. All rights reserved.

Automated construction algorithm

 Improved best-first search (multiple

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved. 10

 Intelligent start (bootstrap)

Copyright © 2007, SAS Institute Inc. All rights reserved.

Automated construction algorithm

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved. 11

Copyright © 2007, SAS Institute Inc. All rights reserved.

Automated construction algorithm

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved. 12

 Model Space Statistics (10 077 696)

Copyright © 2007, SAS Institute Inc. All rights reserved.

Automated construction algorithm

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved. 13

Copyright © 2007, SAS Institute Inc. All rights reserved.

Automated construction algorithm

 Solution Path (KDD-Cup 2004)

Copyright © 2007, SAS Institute Inc. All rights reserved.

Copyright © 2007, SAS Institute Inc. All rights reserved. 14

 Model Space Statistics (6.4 * 1056)

Approximately 85% of all companies use

Generalized additive models

Knowledge discovery in databases (KDD)

AutoGANN modeling node SAS®

AutoGANN user interface

Improved best-first search (multiple

Intelligent start (bootstrap)

Model Space Statistics (10 077 696)

Solution Path (KDD-Cup 2004)

Model Space Statistics (6.4 * 1056)

AutoGANN node functionality:

Transform Variables + Regression ≡

Benefits of presented approach

South African bank

Can an AI search engine and optimization