CogSci 131

Language as a formal system

Tom Griffiths

Admin
Problem Set 0 is due tomorrow at 5pm
Problem Set 1 will be out tomorrow a
little harder than Problem Set 0 and more
representative of what to expect
the Turing machine problem is tricky!

Token manipulation systems

System is defined fully by
a set of tokens
starting positions for those tokens
formal rules stating how token positions can
be changed into other token positions

Rules depend only on current positions,

and define only the next positions

Language as a formal system

Noam Chomsky

Studying the mind in 1950

Behaviorism
explaining complex behaviors through simple
associative learning mechanisms
constructing theories of behavior without
internal mental states or representations

e.g. language (= verbal behavior)

speech acts are a response to environmental
stimuli, with learned sequential structure

The cognitive revolution

Chomsky provided evidence for the idea that
we can model the mind as a formal system
rigorous treatment of mental representations
using human data to evaluate formal proposals

This was part of a more general revolution in

the way we approach behavior
making the study of cognition respectable

Symposium on Information Theory

Often considered the birth of cognitive science
(on 9/11/56, at MIT)
Three famous papers presented:
Allen Newell & Herbert Simon, The Logic Theory
Machine: A complex information processing system
Noam Chomsky, Three models of language
George Miller, The magical number seven

Behaviorist view of language

People form associations between
words and things (semantics)
sheep

People form associations between

words and other words (syntax)
the followed by word makes it more
likely that the will be followed by word

What was Chomsky attacking?

Simplistic behaviorist notions of syntax
Models of language as sequential
e.g., n-th order Markov chains:

P(w i+n | w i ,...,w i+n1 )

Markov chains
w

Transition matrix
P(wi+1|wi)

wi+1 is independent of its history given wi

Markov chains
Chomsky's work in linguistics imply
concomitant understandings of aspects of
mental processing and human nature. His
theory of a universal grammar was seen by
many as a direct challenge to the
established behaviorist theories of the
external environment. The link between
human innate aptitude to language and mind
are innate. The acquisition and
development of innate propensities
triggered by the experiential input of the
time and in later discussions, we are
still far from understanding the genetic
setup of humans and aptitude to language
have been suggested at that time and had
major consequences for understanding how
language is learned by children.

A. A. Markov

What was Chomsky attacking?

Simplistic behaviorist notions of syntax
Models of language as sequential
e.g., n-th order Markov chains:

P(w i+n | w i ,...,w i+n1 )

or, n-grams:

P(w i ,...,w i+n )

P(model, of, language)

P(model, of, quickly)

Language
a set (finite or infinite) of sentences, each finite in
length and constructed out of a finite set of elements

This is a good sentence

Sentence bad this is

1
0

all sequences

linguistic analysis aims to separate the grammatical

sequences which are sentences of L from the
ungrammatical sequences which are not

Grammatical
meaningful
(1) Colorless green ideas sleep furiously.
(2) Furiously sleep ideas green colorless.

probable*

It is fair to assume that neither sentence (1) nor (2)

(nor indeed any part of these sentences) has ever
occurred in an English discourse. Hence, in any
statistical model for grammaticalness, these sentences
will be ruled out on identical grounds as equally
remote from English.

Grammar
a device that generates all of the grammatical
sequences of L and none of the ungrammatical ones

A formal system
tokens
initial positions
rules for moving between positions

such that final positions are sentences

Syntax
Atomic formulas: proposition symbols
(e.g. P, Q), True and False
Complex formulas built out of simple
formulas via rules
if and are okay, () is okay
if and are okay, () is okay
if and are okay, () is okay
if and are okay, () is okay
if is okay, is okay

Finite state grammar

DOG
START

THE

RUNS
FINISH

DOGS

RUN

Finite state grammar

DOG
START

THE

FINISH

DOGS

THE

RUNS

RUN

Finite state grammar

DOG
START

THE

RUNS
FINISH

DOGS

THE DOG

RUN

Finite state grammar

HAIRY
DOG
START

THE

RUNS
FINISH

DOGS

RUN

THE DOG RUNS

THE DOGS RUN
THE HAIRY HAIRY HAIRY HAIRY HAIRY HAIRY

Finite state grammar

HAIRY
DOG
START

THE

RUNS
FINISH

DOGS

RUN

The set of languages generated by finite state

grammars are called regular languages

English is not a regular language

Many simple languages are not regular
e.g.anbn = { ab, aabb, aaabbb, aaaabbbb, }

English exhibits similar dependencies

e.g. the dog the cat chased runs

This center embedding indicates that

English is not a regular language
(provided we include infinitely long sentences)

Phrase structure grammar

Tokens

S, NP, VP, T, N, V
the, man, ball, hit, took

Starting positions

Formal rules

S
S
NP
VP
T
N
V

NP VP
TN
V NP
the
man, ball,
hit, took,

Phrase structure grammar

S
NP
VP
T
N
V

NP VP
TN
V NP
the
man, ball,
hit, took,

S
NP
T
the

VP
N

man hit

NP
T

the

ball

Phrase structure grammar

Context-free languages
rules of the form X Y
e.g. anbn
S aSb, S ab

Context-sensitive languages
rules of the form Z X W Z Y W
e.g. anbncn

Transformational grammar
Phrase structure grammars miss some
structural connections between sentences
e.g. active and passive forms

Hence transformations
e.g. if active form is grammatical, so is passive

Transformational grammar is complicated

complexity of identifying grammatical sentences:
Regular
Context-free

O(n) Context-sensitive
O(n3) Transformational

worse
undecidable

Chomskys project
Identifying a formal system that captures the
structure of human language (and thought)
Ignoring the limitations imposed by finite
human memory resources
This project was distinctive in
postulating rich structures involved in cognition
using human data (linguistic intuitions) to rule out
certain kinds of formal systems
also having implications for computer science

Marrs three levels

constrains

Computation
What is the goal of the computation, why is it
appropriate, and what is the logic of the strategy
by which it can be carried out?

constrains

Representation and algorithm

What is the representation for the input and
output, and the algorithm for the transformation?

Implementation
How can the representation and algorithm be
realized physically?

Break

Up next:
The Chomsky hierarchy

If language were finite

(Sung to the tune of If I were a rich
man, from Fiddler on the Roof. With
apologies to Noam Chomsky.)
If language were finite,
One could memorize
All sentences as if they were just lists.
But its not. True novelty exists.
Language is no finite sys-tem.
A finite state grammar
Is a tempting second thought
But clearly isnt what weve got
The first words in almost any phrase
Can constrain the end in many ways
So how about a push-down automaton
Might that be the very thing?
Just like a finite-state with a proper stack.
There would be just one symbol popped
from the top
Plus one as input from the string
And others there just waiting to pop back.

But hu-man language cannot be context-free

Swiss-German shows why this is true
Thanks to the cross seri-al de-pen-den-cy
So much for the very thought that language
could be finite
Ive shown why the notion just wont do
But now onto what language has to be
Oy!
Language isnt finite
Nor is it finite state
Or even possibly push-down
Nor just strings composed of verb and noun
Language is a complex sys-tem.
There are transformations
Over parse-trees that you
simply cannot code as linear strings
Every sentence is hierarchical
With deep structure that can be revealed
I see language as governed by universal grammar
With a hefty set of rules
And acquisition guided by a device
I see grammar as context sensitive and complex
Not like the grammar taught in schools
Wouldnt such a system be quite nice?

The Chomsky hierarchy

Languages

Machines

Computable
Context sensitive
Context free
Regular

Turing machine
Bounded TM
Push-down automaton
Finite state automaton

Turing machine
( state,read,move,state,write)

rules

state

read/write head
tape

Finite state automaton

( state,read,move,state,write)

rules

state

read/write head
tape

Readable Stack/ Bounded TM

anbm
anbn
anbncn

b
b
Finite state
anbm
s

Pushdown

anbm
anbn

Non-context-free constructions
Cross-serial dependencies
occur in Swiss German and Dutch
in English: respectively
Bob, Jim, and Ted earned $3, $4, and $5 respectively

Cannot be produced by context-free grammar

Humans

The Chomsky hierarchy

Languages

Machines

Computable
Context sensitive
Context free
Regular

Turing machine
Bounded TM
Push-down automaton
Finite state automaton

The power of rules and symbols

Generativity
infinite use of finite means
from tokens, initial positions, and rules, infinitely
many outcomes result
captures (constrained) novelty of language

Structured representations
e.g. hierarchical representations, expressing
relationships at multiple levels of abstraction

Structured representations
Driving

Start car

Step on gas

Move leg

Push gas

Turn on ignition

Take out key

Insert key
in ignition

Turn key

Structured representations
Driving
Cooking

(Humphreys & Forde, 1999)

(Cooper & Shallice, 2000)

Structured representations
Driving
Cooking
Music and dance
Is any behavior not hierarchically organized?

The power of rules and symbols

Generativity
infinite use of finite means
from tokens, initial positions, and rules, infinitely
many outcomes result
captures (constrained) novelty of language

Structured representations
e.g. hierarchical representations, expressing
relationships at multiple levels of abstraction

Next week
Take a look at Problem Set 1!
its harder, may take longer, plan accordingly

Tuesday: Learning structured representations

(or: Not learning structured representations)
The Poverty of the Stimulus argument