Académique Documents
Professionnel Documents
Culture Documents
OFAREAL-TIME
VISION
SYSTEM
FORAN
AUTONOMOUSMODEL
AIRPLANE
Danko
Antolovic
Submitted
tothe
facultyothe
f UniversityGraduateSchool
inpartialfulfillmentof
the
requirements
for
thedegree
Master of
Science
inthe
Departmentof Computer Science
IndianaUniversity
October,
2001
Acceptedbtyhe
Graduate
Faculty,Indiana
University,
in
partial fulfillmentof
the
requirementsfor
the
degree
oMaster
f of
Science.
________________________________________
Prof.
Steven
DJohnson,
. Ph.D.
________________________________________
Thesiscommittee Prof.
Florin
Cutzu,
Ph.D.
________________________________________
Prof.
MichaelE.
Gasser,Ph.D.
September
21,2001.
ii
Copyright©
2001
Danko
Antolovic
ALL
RIGHTSRESERVED
iii
ACKNOWLEDGMENTS
Thisthesisis
daescriptionohafardware/softwaresystemc onstructed
iseries
an of Y790
independentstudy
coursesinthe
Departmentof
Computer
ScienceaIndi
t ana
University.
The
workwasdone
under
the
supervision
oProfessor
f Steven
DJohnson,
. to
whomI am
gratefulfor
hissupport,hisinterestinthe
progressothe
f project and
, for
hisinsightful
andcriticalcomments.Professor
Johnson
constructed
the
cameragimba currently
l iunse.
am
I happy
tohave
hadthe
help
Mr.
ofBryce
Himebaugh,
engineer and
pilot
extraordinaire.
Beside
constructing
the
A/Dconverterand
the
ser vocircuit,
Bryce
has
sharedhisknowledge
andskillthroughmany
helpfuland
enjoyable
discuss ions.
also
I wishtothankProfessor
RobertDeVoe
the
ofIU
Schoolof
Optom etry.
His
expertise
onanimalvisionhashelpedmestablish
broader
a context for
some
othe
f
problemsencountered
inrobotic
perception.
Finally,my
thanksgtooLaurie,my
spouse,for
her
patience
during this,
thelatestof
my
academic
stints.
iv
ABSTRACT
Danko
Antolovic
DEVELOPMENT
OFAREAL-TIME
VISION
SYSTEM
FORAN
AUTONOMOUSMODEL
AIRPLANE
Thisthesisdescribes
raeal-time
embedded
visionsystemcapable of
tracking
two-
dimensionalobjectsin
relatively
a simple
(uncluttered)
scene,
in live
video.
Thisvision
systemisintended
acomponent
as of
raobotic
flightsystem,
used
t keep
o model
a
airplane
iholding
na patternaboveaonbjecton
the
ground.
The
systemuse taswo-
prongedapproachtobjecttracking,taking
into
accountthe
motion
othe
fs ceneand
the
graphic
“signature”
othe
f object.
The
vision
systemconsistsof these
main
components:
m
a otion-detection
andfiltering
ASIC,implemented
onFPGAs,
sace ne-analysisprogram
running
oMotorola
na ColdFire
processor,
daual-portRAMholding
the
ima ge
data,and
daigitalcamera
omotorized
na gimbal.
________________________________________
________________________________________
________________________________________
v
CONTENTS
Page
Acknowledgments iv
Abstract v
1. Introduction
tothe
SkeyeballVision
Project 1
1.1 History
othe
f visionsystem 1
1.2 Structure
othis
f document 4
2. FunctionalOverviewof
the
Vision
System 8
2.1 Visionmethodology 8
2.2 Biologicalparallels 11
3. ProjectStatus 13
3.1 Capabilitiesandlimitations 13
3.2 Measurementsof
the
tracking
speed 13
3.3 Summary
remarksontheperception
problem 16
4. Hardware
Architecture 19
4.1 Architecturalcomponents 19
4.2 Biomorphic
approach
toarchitecture 20
5. DesignSummary 24
5.1 XC4010
digitaldesign 24
5.1.1 Front-endFPGA 24
5.1.2 Back-endFPGA 25
5.2 MCF5307(ColdFire)code 26
vi
6. NTSCVideoSignal 28
6.1 Evenfield 28
6.2 Oddfield 31
7. Formatting
theImage
Scan 33
7.1 Verticalformatting 34
7.2 Horizontalformatting 37
7.3 Auxiliary
components 39
7.4 Signals 40
8. Digitizing
andThresholding 42
8.1 Black-and-white
inversion 42
9. Setting
the
ThresholdAutomatically 43
9.1 Heuristic
procedure 43
9.2 Thresholdcalculation
onthe
back-end
FPGA 45
9.2.1 Data
path 45
9.2.2 Control 47
9.2.3 Signals 47
10. DigitalZoom 51
10.1 Zoomimplementation
onthe
FPGA 52
11. RoundRobin
Procedurefor
Data
Sharing 55
11.1 Statusbyte 57
11.2 Roundrobin
onthe
front-end
FPGA 59
11.3 Roundrobin
onthe
MCF5307
processor 63
12. Pixelread/write
cycle 64
vii
13. Frame
Comparisonand
the
Motion
Vector 68
13.1 Methodology 68
13.2 Computation 69
13.3 Designcomponents 69
13.4 Signals 70
14. Writing
the
MotionVector
toDPRAM 72
15. Parametersof
the
Front-End
FPGA 75
16. IRQ5/ParallelPort
Complex 80
16.1 IRQ5
handler 80
16.2 Duty-cyclegenerator 81
16.3 Servomotionfeedback 81
16.4 Displacementvector 81
16.5 Saccadic
blanking 82
16.6 IRQ/PPcircuitonthe
back-end FPGA 83
17. Auxiliary
Features 88
17.1 Serialcommunication
with
the
MCF5307 88
17.2 Diagnostic
data
logging 88
17.3 Softrestartof
the
visionprogram 88
17.4 Radiocontrols 89
17.4.1Radiodecoder’ssignals 90
18. Feature
Recognitiononthe
MCF5307
Processor 91
18.1 Maindata
structuresin
the
MCF5307
code 93
19. Initializationothe
f SBC5307Board 95
viii
20. Characteristicsof
the
Camera/Servo
System 96
21. Supplementary
Diagrams 98
References 101
ix
1. INTRODUCTION
TOTHESKEYEBALLVISION
PROJECT
Skeyeballisanongoing
projectinthe
Departmentof
Computer
Sc ience
Indiana
at
University.
Itiscenteredaround
radio-controlled
a modelairplane, which
ibeing
s
convertedinto
semi-autonomous
a vehicle.
Itsprimary
perception
ias computer
vision
system,andiwill
t alsobequipped
withattitudesensors,
digital videoandtelemetry
downlink,anddigitalcommand
uplink.
The
objective
ito
sgive
the
airplane
the
autonomy
to
fly
beyond
the
l ine
osight,
f
navigate,andfindobjectsof
interestby
their
visualappearance
ra ther
thanblyocation.
The
objective
othe
f work
describedherewasto
build
vision
a syst emthatfollowsan
objectin
relatively
a simple
(uncluttered)
scene,
in
live
video. This vision
systemwillbe
integratedinto
larger
a robotic
navigation
systemused
to
steer
t he
airplaneinto
holding
patternabove
selected
a feature
ontheground.
1.1 Historyothe
f visionsystem
The
Skeyeballvisionwasfirstenvisioned
asasubsystemimpleme nted
ona
microcontroller
chip.
Soonibecame
t obviousthat
faast(and
nottoo
cos tly)
implementationothe
f early
processing
stageswasneeded:
vis ion
becameaA
n SIC-cum-
microprocessor
system,and
iisstill
t such system
a today.
1
Picture
Aerial
1: view
of
taargetoverflight
The
developmenthasgone
through
two
distinctphases.
The firstphasey ielded
strictly
a
laboratory
prototype:
thehardware
wasbuiltfromproto
boards,
andthe
p rocessorswere
X
a ilinxXC4010FPGA
and
Motorola
a MC68332. Data
were
shared
through
aS
n RAM
onthe
commonbus.
Thisarchitecture
required
considerable
data
copyin g,and
the
25
MHzMC68332processor
wasrather
too
slow
forthe
task.
Nevertheless the
, systemwas
capable
o(slow)
f objecttracking,moving
camera
a osimple
an g imbal.
Picturesa2nd
3
show
the
gimbalandthecircuitry
othe
f firstphase.
We
thenobtainedsome
realistic
footage
bfylying
the
airplane with
the
immobile
camera.
The
laboratory
prototypewascapable
odetecting
f targetfeat uresin
overflightsequences,
buttrackinganobjectreliably
aflight
t speedswasvery
problema tic.
Picture
shows
1 a
2
typicalaerialview:
the
plane
castsitsshadow
nextto
the brightsquare
target(a
brightly
coloredblanketon
thegrass).
Thisfirstphasegave
ufairly
as goodinsightintothe
minimalr equirementsof
such
a
system.The
second
(current)
phase
idescribed
s inthe
restof
the thesis.
Two major
architecturalimprovementsare
faster
a microprocessor
(90
MHz MotorolaColdFire)
and
daual-portRAMfor shareddata.
Elimination
oone
f very
cumbersome
proto
board
has
alsomade
the
ASIC(application-specific
integrated
circuit) implementation
much
easier.
The
fundamentalvisionalgorithmhasnotseen
much
change
over
time,
ex ceptfor
the
additionothe
f thresholdcalculationinthe
secondphase the
- improvementhasbeen
the
increasedspeed.
Muchgreater
modificationshadto
bme ade
ttohe
dat falow
procedures,
totake
advantage
othe
f dual-portmemory
and
better
busarchitecture.
Finally,inthe
second phase,the
systemwasgiven
the
proper
startup
procedure
andradio
controls,andthe
entirecircuitry
wasbuiltso
ato
sbseuitable
f or
mounting
inside
the
airplane.
Pictures4-7
show
the
equipmentbuiltin
the
second
phase:thr eecircuitboards,
the
new
cameragimbalandthe
radio-controlled
power
switch.Pict ure
7also
showsthe
radioandTV
linksconnectedtothe
visionsystem.
3
1.2 Structure
othis
f document
Thisdocumentserves
daualpurpose:itdescribesthe
constructed
sys temas
saolution
to
anengineering/computationalproblemin
broad
terms;italso
describe itat
sthe levelof
detailneededfor
modificationand
further
development.
The
systemdividesitselfnaturally
intoseveralsubsystems. Thefirstfive
sectionsof
this
documentprovide
anoverview,andtheremaining
sectionsdescribe
the
subs ystems
separately,withthe
levelof
detailincreasingwithin
each
subsy stemdescription.
We have
triedtomake
iclear
t where
the
broad
description
endsand
the
deta iled
onebegins:
typically,detaileddescriptionsaregrouped
into
specialized
subsec tions.
The
ultimate
levelof
detail schematics,
- pinoutlistsand
the
sourcecode has
- been
relegatedtoelectronicform.
Thisdocumentcontainssummary
descr iptionsof
those
materials,aswellaspassagesreferring
directly
tosourc deetails.Interestedreader
should
become
familiar
withthe
circuitschematicsand
the
sourcecode.
4
5
Picture
Gimbal,
2: phase1 5 Picture
Circuit
3: boards,
phase
1
Pictures4,5:
Circuitboardsandgimbal,
phase
2
6
Pictures4,5:
Circuitboardsandgimbal,
phase
2
Pictures6,7:
Visionsystem,
radio
and
TVlinks,power
switch
7
2. FUNCTIONALOVERVIEWOF
THEVISION
SYSTEM
2.1 Visionmethodology
Aswe
statedinthe
introduction, the
objective
othis
f work
wasto
build
vision
a system
thatwillfollow
anobjectin
relatively
a simple
scene,
in
live video.
We have
used
two-
a
prongedapproachtobjecttracking,taking
into
accountthe
motion
othe
fs ceneand
the
graphic
“signature”
othe
f object.
Thisapproachwasmotivatedbtyhe
factthatobjectrecognition
is computationally
intensive,andimpossible
toaccomplish
oframe-by-frame
a
n basis with
the
available
hardware.
Nevertheless,the
visionsystemmustoperatefastenoug nhottoallow
the
objecttodriftoutof
the
fieldovision.
f Picture
9illustratest hispoint.
The
objective
ito
srecognize
the
smalldark
objectasthe
targe t,
obtain
itsoffsetfromthe
center
of
the
visionfield,andmove
the
camera
tobring
the
object to
the
center.
Obviously,the
visionsystemmusttake
still
a frame
and
base
its calculationoIn
intt.he
meantime,the
objectwillchange
location,
perhapseven
driftoutof
t hefield.
By the
time
itiscalculated,the
displacementvector
may
wellbe
irrelev ant.
Toavoidthis,the
motion
othe
f scene
itracked
s frame
bfyram e,
andthecamera
moves
tocompensate
for
The
it. center
of
the
fieldmovesalong
with
the target,and
the
displacementvector,whenavailable,willstillbe
meaningful.
It should
bseaid,
however,
thatthe
importance
othe
f driftcompensation
decreased
aw
s uesed m
a uchfaster
processor
inthe
secondphase
othe
f project(see
Section
1.1).
8
Trackingthe
selected
object
Tracking the
overall Recognizing the
tracked
motionothe
f scene. objectwithin
thescene.
Shallow,pixel-specific Deep,
object-specific
computation. computation.
Frame-by-frame
motion Objectlocation
vector,
vector asavailable
Objectlocation
overrides
the
motionvector
Camera movesto
compensate for
scene
motion, or
to
bringthe
objectin
the
centerof
vision
field.
Picture
8Method
: overview
9
Objectrecognitionrequiresseveralstagesby
which
redundantand
non -essentialvisual
informationiparsed
s out,
untilwe
are
leftwith
selection
a owf ell-definedobjects,
also
referredtoathe
s featuresothe
f scene.
Inourcase,
thesta gesare:thresholding,
edge
detection,segmentationintoconnectedcomponents,
clean-up
and
signature/ location
calculation.
End
calc.
Startcalc.
Picture
9Drift
: compensation
Picture
10gives
functional
a overview
of
the
vision
system,
where the
progressively
thinner
arrowssignify
the
reductioninbulk
othe
f visualinformation. Thisisthe typical
“funnel”
othe
f visionproblem,
leading
fromsimple
computationson
large volume
of
*
data
tocomplexoneson
small
a volume,yielding
number
a or
twoats he
result.
*
At30frames
persecond,483lines
perframe,and 644byte-sizedpixels perline,raw
cameraoutput
amounts to9.33Mbytes/sec.
The644samples perl ineocontinuous
f signalconform tothe4:3aspect ratio
prescribedbN y TSC,andonebyteperpixelis
raea listicchoiceocolor
f depth.
10
Toidentify
target
a objectwithinthe
scene,we
used
the
second
mom entsaboutthe
object’sprincipalaxesofinertia
athe
s “signature.”
Second momentsare
invariantunder
rotationsandtranslations,fairly
easy
tocalculate,and
work
well in
simple
scenes.
2.2 Biologicalparallels
Itisnotentirely
surprising
thatcertainfunctionalanalogies should
developbetween
robotic
perception,suchathis
s real-time
vision
system,
and
percepti on
iannimals.
While
suchanalogiesmustnotbe
takentooliterally,
they
provide
glimps
a aeuseful
t
generalizationstobm
e ade
aboutperception
problems,
andww
e illske tchthemoutas
appropriate.
For
example,camera
motionbasedonfeature
recognition
bears
sai milarity
ttohe
(involuntary)saccadic
motion
othe
f eyes,
theone-shotmovementthatbri ngs
daetailof
1
interestintothe
center
ofthe
visionfield. Eyes’
saccadesare
fast,
have
largeamplitude
(aslargeathe
s displacementof
the
objectof
interest),
andthe ayre
ballistic
movements,
i.e.they
are
notcorrectedalong
theway.Such
motion
a icompatibl
s w
e ith
need
a for
speedover
precision:if
the objectidentification
icomputationally
s intensive,
usethe
resulttofullextent,andafast
s aspossible.
Attheendothe
f process,thevisionproduces
fae w
dozen
bytes
every
coupleoframes
f –actualrate
depends onthecontentsof
theimage.
11
Digitalcameraon
2a-servogimbal
Servoduty
cycles
Compositevideo
Syncseparator A/Dconverter
andsampler
8-bitvideo
Sync.signals
Thresholding Current
thresholdcalc.
B/Wimage
Motion
detector
Datasharing
(roundrobin) FPGA’s
Servo
Blanking
motion
MCF5307
Edgedetector
B/Wedgetrace
Segmentation,
removalofsmallfeatures;
signaturecalculation
Featurelist Target
selection
Motion
vector Target
recognition
Displacementvector
Vectorselection
andservodriver
Servoduty
cycles
Picture
10:Functionaloverview
12
3. PROJECTSTATUS
3.1 Capabilitiesandlimitations
We
have
testedthe
lab
visionsystemby
manually
moving
irregular shapesin
vision
a
fieldfairly
free
orandom
f clutter,andalsobm
y eansof
ra otating
table(Section
3.2
below).
The
recognitionsystemtracksitstargetreliably
under
tra nslation
androtation,
anditnhe
presence
several
of shapesintroducedainterference.
s Excess ive
skew
breaksoff
the
tracking,since
the
visionalgorithmmakesno
provision
for
it,
buta onblique
view
of
ca.
20degreesisstillacceptable
(Section3.2).
Likewise,
occlusion
is interpreted
acashange
of
shape,andthe
targetislostwhenpartially
occluded
(e.g.
b dyrifting
beyond
the
edge
of
the
visionfield).
These
limitationsare
obviousconsequencesof
the vision
methodology
describedinSection2.1
and
Picture10.
The
trackingandmotiondetection
workequally wellwith
the
zoomengaged.
As
expected,zoommakesthe
systemmore
reliable
itnracking
smal targets,
l atthe
expense
of
limiting
the
fieldovision
f ttohecentralone-ninth.
3.2 Measurementsofthe
tracking
speed
The
Skeyeballairplane
flieswithincertain
rangesospeed
f anda ltitude;
our
fixed-
camera
flightshave
rangedfrom18
to60mph,with
typical
a speed
of ca.
35 mph.
Likewise,target-overflightaltitudeshave
beenfrom60
t3o20
Cons
ft. equently,
theline
of
sighttothe
targetfeature
changesdirection
relative
ttohe body
othe
f airplane,
with
13
certainangular
velocity,and
the
visionsystemmustbe
able
tkoe ep
uw
p ith
In
itt.he
test
flights,the
targetpassedthroughthe
visionfield
othe
f fixed
cam era
itnime
intervals
rangingfrom0.8to4.2seconds,depending
on
thealtitude
and
velocity
otf he
airplane.
Inorder
toobtainsome
quantitative
measure
othe
f vision’stracki ng
abilities,
we have
constructed
test
a rig ra-otating
table
with
featuresto
track.
The vision
systeml ocks
successfully
ontothe
(largest)
presented
featureand
the
camer taurnsfollowing
the
rotationothe
f table.
Therotationspeedigradually
s increased,
unti the
l tracking
breaks
off
or
targetacquisitionbecomesimpossible.
Picture
11
showsthe
ex perimentalsetup:
the
cameragimbal,the
rotating
table
with
two
features,
andthe TV
screen
showing
the
camera’sview.
Picture
12showsthegeometry
othe
f setup.
Thespeed
othe
f table’ motor
s wasregulated
by
applying
variable
voltage,andtheangular
speed
othe
f table
wa measured
s athe
s time
neededfor
tenturns.A
simple
formula
relatesthetable’sangul ar
speed ω
to
the
camera’s
sweeping
speed, θ:
ω
θ=
d
1+
2
r
14
Picture
1Laboratory
1: set-up
for
the
tracking-spe ed
measurements
ω
r
Picture
1Table
2: speedvs.
thecamera’ssweep
15
isthe
delevation
othe
f camera
above
the
table,
a nd is
rthe
distance
othe
f targetfeature
fromthe
center
the
oftable.
At =
d26.5
cm,
and =
1r c0m,
wefound
thatthe
tracking
wasstillreli ablewith
the
camera sweeping
aanrcathe
t maximumspeed
of:
θ(max)
45
=
degrees/second.
The
camera’sfield
ovision
f ica.s 47
degreeshigh and
ca.
60
degreeswide(see
Section
20),
which
putsthisvision
systemwithin
the
range of
speedsrequired
tkoeep
uw
p ith
the
overflightspeedsthatwere
quoted
above.
Of
course,
tracking
speeddependson
the
complexity of
the
scene.
These
measurements
were
performed
with
twoothree
r shapes,
plusan
in termittently
visible
edgeothe
f table.
The
scene
observed
ireal
an flightis richer
in
features,
butatleastforgrassland
and
trees,
featurestend
thoave
low
contrastand
disap pear
below
the
threshold,which
itnurn
issetto
single
outhigh-contrasttargets.
3.3 Summaryremarksonthe
perceptionproblem
Real-time
perception
canbenvisioned
afasunnel in
which
the
data
volume
ireduced,
s
butthe
algorithmic
complexity
increases.
Typically there
, willbe
severalstageswith
fairly
differentbreadth/depth
ratio.
Thisisintrinsically
not
paroblemamenable
tsoin gle-architecture
processing.
Ofcourse,
sapeed
tradeoff
isin
principle
alwayspossible,
b utengineering
considerationssuch
as
16
power
consumption
andheatdissipation
place
very
a reallimiton
thatapproach.
We
believe
thatitisbetter
to
use
severalprocessor architectures,
each
suitable
for
daifferent
stage
othe
f perception
process.
Appearance
otnhe scene
oconfigurable
f microchips
makesthisgoalboth
realistic
and
appealing.
Robotic
perception
ialso
s problem
a in
embeddedco mputing.
Requirementsimposed
by
the
smallmodelairplaneare
bit
a onthe
stringent side,
andone
can
envision
much
a
more
relaxed
design
foran
assembly
line
osecurit
r syystem.
However,
theneed
for
perception
inaturally
s the
greatestin
mobile
robo ts.In
such
applicationsthe
vision
systemwillalwayshavetboceompactandautonomou s,
because
ibestows
t autonomy
on
m
a obile
device
whose
primary
function
isomething
s other
thancarrying
vision
a system
around.
Architecture
should
follow
function,
starting
aat fairly
low
level.
Forexample,
data
collection
itnhissystemisdone
with
digital
a ca mera
which
serializesthe
(initially)
parallelimage
input.
Thischoice
wasdictated
bgy ood
practicalreasons,
butthe
system
lost
gareatdealof
processing
power
because
oth
f atserialization.Image
inputshould
have
been
done
ipnarallel,
which
itnurn
would
hav reequired
specialized
a device
anda
much
broader
data
path
itnhe
initialprocessing
st age.
The
segmentation
stage
ibetter
s suited
for
impleme ntation
ogneneral-purpose
processors
because
othe
f smaller
data
volume
and
more"sequen tial"
algorithms.
An architectural
alternative
may
bpeossible
here:segmentation
coul bdaettempted
ohighly
an connected
17
neuralnetcircuit,
trading
off
an
exactalgorithm for
an
approximate,
butparallelizable,
search
procedure.
Neuralnetsearches,
onthe
othe hand,
r are usually
slowto
converge
and
may
notimprove
theoverallspeed.
Animalvision
cannotbe
separated
fromcognitivefu nctionsand
motor
coordination,
and
thismustbe
true
for
robotic
vision
awell.
s How m uch
“intelligence”
ibuilt
s intohigh-
levelprocessing
ovisual
f information
dependsont he
ultimate
objectivesofSkeyeball:
for
example,
searching
for
3aD
shape
icomplex
an panorama
ipasroblemdifferentfrom
thatof
hovering
above
prominent
a feature
otnheg round.
In
termsof
steering
andmotor
coordination,
biolog icalparallelsare
relevant.Itisknown
thatinertialmotion
sensorsplay
large
a roleitn hegazecontrolof
mobile
animals[1].
Since
the
inputfromthemotion
sensorsissimpler, and
the
processing
presumably
faster,
thissensory
pathway
providesthe
supporting
motion information
much
faster
than
can
be
obtained
bvyisualprocessing.
Theairplane
may
ve ry
wellbenefitfromaneventual
integration
oits
fvision
and
attitude/motion
senso rs.
Visualperception
ian
sill-posed
problem,
and
exam plesof
functioning
compromisesmay
be
more
valuable
than
exactresults.
Throughoutthi document,
s we pointoutsimilarities
with
biologicalsystemswhich
strike
uas
sinterest ing,
although
wdenootpursue
themin
depth
for
lack
oexpertise
f otnhe
subject.
A
few pitfallsnotwithstanding,
we believe
that
saynthesisofcomputational,
physiologicalandeng ineering
knowledgewillbe
necessary
for
the
eventualdevelopmentof
reliable
and
versat ile
perception
systems.
18
4. HARDWAREARCHITECTURE
4.1 Architecturalcomponents
Picture
is
1an
3overview
of
the
architecture
otf he
vision
system.
Pictureshows
14 all
the
signalspertaining
ttohe
flow
of
data
fromthe camera,
through
the
processorsand
back
tsoervo
motors,
butitomitssome
peripheral details.
Camera The
- “eye”
othe
f systemis
samalldigitalcamera producing
, grayscale
(non-
color)
NTSCvideo
signal;itsother
characteristics are
largely
unknown.
Thecamera
is
mounted on
gimbal
a driven
btywo
servo
motors,with
50-d
a egree
range
omotion
f in
each
direction,
andipermanently
s focused
oinnfin ity.
Sync
separator t–he
NTSCgrayscale
video
signalcontainsthree
sy nchronization
signals.
These
are
extracted
bmy eansof
video
sync
separato LM1881
r bNy ational
Semiconductor,
mounted
oprototyping
an board
along with
supporting
circuitry.
A/D
converter w
– ueseAnalog
Devices’
AD876,which
ipasipeline 1d0-bitconverter.It
ismounted
otnhe
same
proto
board,
with
supporting circuitry
for
itsreference
voltages.
Sampling
control,
thresholding
and
thresholdcalcul ation,
motion
detection
and
zoomare
implemented
adigital
s designson
synchronous
a pai of
rXilinxXC4010
FPGA's,
running
at33.3
MHz.
Start-up
configuration
idone
s with
tw A
o tmel’sAT17LV
config
ROMs.
19
The
entire
objectrecognition
iimplemented
s acod
s e,
running
o90
anMHzMotorola
MCF5307
ColdFire
integrated
microprocessor.
We use caommercialevaluation
board,
SBC5307,
with
megabytes
8 ofDRAM,
start-up
flash
R OM,
expansion
busand
communication
ports.
The
two
processorsshareimage
data
through
32K
a d ual-portSRAM,
CY7C007AV
by
CypressSemiconductor.Data
accessisimplemented
a rasound-robin
procedure,
with
the
objective
ospeeding
f uphigh-volume
data
transfer in
the
early
stagesof
the
vision
process.
The
driverfor
the
servomotorsthatmove
the
camer iaimplemented
s onone
othe
f two
FPGA’s.
Motion
feedback
fromthe
servosisgenerat ed
bPIC16F877
ay microprocessor,
on
the
basisof
servos’analog
position
signals.
4.2 Biomorphic
approachtoarchitecture
Nervoussystemsof
animalsutilize
specialized
hard warealmostby
definition.
There
is
much
evidence
thatbiologicalarchitecturefollows function:
for
example,
theretina,
with
itslayersof
many specialized
typesof
cells,
isapparently
structu
a re
whichhasevolved
to
dealwith
the
initialstagesof
the
vision
funne l,
fromthe
cellular
levelup.
Architecture
othis
f robotic
systemfollowsthe
sam “ebiomorphic”
principle
amuch
s as
possible.In
order
to
increase
overallspeedand
th roughput,
we have
optedfor
ASICsand
dedicated
data
paths,
even
athe
t costof
under-uti lizing
some
components.
Multitasking
20
and
time-multiplexing
are
systematically
avoided.B iologicalsystemsfollow
this
principle
because
oevolutionary
f constraints,
but they
solvethe
real-time
perception
problemwell,
andthe
trade-offsthey
make
appeart botehe
rightones.
21
XC4010
FPGA
back end servo data
16 diagnostics
duty cycles 2 Parallel Serial
Servo duty port port
cycles
servo motion
IRQ5 MCF5307
B/W threshold
Unified cache
(DRAM only)
PIC16F877
DRAM(8M)
controls 4 8 video &
4 Servo position 4
threshold
(analog) enable
15 8
sigs.
Servos addr data
sync signals
enable
2
LM1881 sigs. 7
XC4010 addr 15
video CY7C007AV
digitized video FPGA DPRAM(32K)
front end
Camera data 8
AD876 8
sampling clk.
Picture 13:Architecturaloverview
22
FPGA board SBC5307 evaluation board
PIC16F877 XC4010 FPGA
back end
Servo motion servo highs
signal 14 PP 0-13 Serial
port
RDY PP 14 Parallel
Servo duty port
2 2 cycles ACK PP 15
SWM MCF5307
J8
PWM_1
J1 J4 IRQ5 IRQ5
PWM_2 Feature tracking
Gimbal IRQTRG
J4 J5 board OE 15 8
B/W threshold BWE0
CS4 A D
to/from servo motors
CS5
J8 J9
Picture
14:Architectural
overview d–etailed EVEN/ODD CE_L
SEM_L
p.
23 LATE CLK 8 video &
threshold
15 8
R/W_L
THR RDY OE_L addr data
CE_R
Left
SEM_R port
V. SYNC.
LM1881 R/W_R
Sync.sep. H. SYNC.
OE_R
BUSY_R
from RCA video
camera jack XC4010 FPGA BUSY_L CY7C007AV
dig. video front end Right DPRAM
M/S port
AD876 Shared image
8
converter addr 15 data
sampling
clk.
data 8
23
5. DESIGN
SUMMARY
Thissection
provides
taop-levelsummary
othe
f sc hematicsandcode
moduleswhich
comprise
the
functionalconfiguration
othe
f hardwa re
described
iS
n ection
4.
5.1 XC4010
digitaldesign
Configuration
othe
f twoFPGA
processorsisimpleme nted
with
the
Xilinx
Foundation
2
developmenttool,
eitherasschematicsorasAbelH DL
code. This listgivesan
overview
of
the
functionalitiescontained
itnhe
hi ghest-levelmodules.
Design
componentsand
signalsare
described
throughoutthe textand
itnhe
schematics
themselves.
5.1.1 Front-endFPGA
Thisdesign
consistsof
seven
top-levelschematics and
number
a of
macros.Itutilizes
about
60%of
the
logicalblocks(CLBs)
of
the
XC40 10
FPGA.
VISION_IN –containsthe
entry
pointfor
the
video
sync
signa ls,
verticaland
horizontal
imageframing
and
sampling,
andthe
zoom.
ANALOG_IN i–nputfromthe
A/D
converter,
normalization
tbol ack
reference
level,
black-and-white
thresholding.
FIELD_END p–laceholder
schematic,
invoking
macrosfor
vector output,run-time
parametersand
theroundrobin.
24
VISION_OUT i–nput/outputto
the
DPRAM.
RR_ADDRESS a–ddresscountersfor
image
buffers,
round-robin
a ddressmultiplexer.
MOTION_VECT i–nvokesthe
pixelread/write
cycle,
calculation
o the
f motion
vector.
CLOCKS e–ntry
pointfor
the
externalclock
andresetsign al,generation
ointernal
f
clocks.
5.1.2 Back-endFPGA
The
design
consistsof
five
top-levelschematicsan number
ad of
macros.Itutilizes96%
of
the
CLBsof
theXC4010
FPGA.
DUTY_CYC_PP –IRQ5/parallelportcommunication,
duty
cyclegene rator,
servo-
motion
signal.
HIST_DP d–ata
path
forthe
threshold
search
itnhe
histogr am.
HIST_IO_RAM h–istogramstorageand
management.
HIST_MINMAX c–omparison
logicfor
the
threshold
search,contro ASM.
l
HIST_INTEGRAL –calculation
othe
f histogramand
featuresarea.
25
5.2 MCF5307(ColdFire)
code
Programsrunning
otnheMCF5307
ColdFire
processor are
written
iC
nand
ColdFire
3
assembler,and
compiled/assembled
with
GNU
“gcc”and
“as.”
T hislistgroupsthe
code
modulesby
systemfunction;further
description
is provided
itnhe
textand
itnhe
source
comments.
main.c initialization
- and
main
loop
for
featu re
recognition
Configuration
and
startup:
cache.s cache
- initialization
ConfigRegs.s MCF5307
- configuration,
running
fr omflash
ConfigRegs2.s MCF5307
- configuration,
running
f romDRAM
crt0.s setup
- for the
Clanguage
globals.c init.
- of
globalvariablesfor
functi onalcode
glue.c heap
- setupand
other
book-keeping
start.s processor
- startup
sequence
vector.s vector
- table
Feature
recognition:
ConnectedComponents.c segmentation
- algorithm
Diagnostics.c vision
- system’serror
reporting
FeatureDetector.c feature
- “signatures”
FeaturePoints.c maintenance
- oheap
f data
struct ures
Features.c driver
- modulesfor
acquisition
and tracking
26
GraphDFS.c depth-first
- graph
traversal
SimpleEdge.c edge
- detector
Inter-processcommunication:
CyclesPP.s communication
- throughthe
parallel port
IRQHandler.s handler
- for
theInterrupt(5Proc ess1)
roundRobin.s round-robin
- DPRAMaccess
Servo
motion:
servo.s translating
- displacementsto
servo
du ty
cycles
Auxiliary:
Datalog.c interface
- library
for
thediagnostic data
log
DataOutput.c diagnostic
- data
output
serial.c communication
- library
for
the
serial port(s)
SerialHandler.s UART
- interrupthandler
IRQ7Handler.s handler
- for
theInterrupt(7soft restart)
TermInput.c stub
- forthe
terminalcommand
inpu t
27
6. NTSC
AND
THEEVENTSSYNCHRONOUS
WITHTHEVIDEOSIGN AL
4
Frames
the
ofNTSCtelevision
signal consist
of
two
interleavedfields,
marked
bayn
even/odd
synchronization
signal.
Atthe
beginning
o each
f field
there
ipaseriod
otime
f
when
the
beamretracesback
ttohe
top
othe
f image (atlow
intensity),
paeriod
marked
by
the
verticalsynchronization
signal.
Also,
horiz ontalretracing
between
video
linesis
5
marked
btyhe
horizontalsync. Pictures
15
t1os8how
some
detailsof
theoperat ions
†
synchronouswith
the
video
signal.
We
sample
the
contentofthe
video
signalduring
th even
field
oeach
f frame,
andat
variable
resolution e–very
third
line
itnhe
absence
ozooming,
f every linewhen
the
zoomisengaged.
Motion
detection,
which
compares adjacentvideoframes,
isperformed
simultaneously
with
the
sampling,
between
the
pixel s,
asitwere.An
early
fraction
othe
f
odd
field
iused
s forcommunication
between
process esand
for
the
threshold
calculation.
6.1 Evenfield
Picture
1s5howsthe
beginning
othe
f even
field,
t he
synchronization
signals,
theA/D
sampling
clock,and
the
composite
analog
video
sign Notice
al. thelong
verticalblanking
period
before
the
beginning
othe
f actualimage
tra nsmission.
†
Pictures
15-18,
27,
28,
32,
35,
38
and
4a2re
scre en
shots from
H
a ewlett-Packard
16500B logic
analyzer.
28
Blanking
period Image
transmission
Picture
15:The
evenfield
The
sampling
clock
operatesin
bursts,
during
thes ampled
portion
othe
f even
field
(see
Pictures15,
16,19
and
26).
Firstsample
oeach
f l ine
itaken
s during
the
so-called
back
porch
othe
f horizontalsync,atthe
blackreferenc ientensity.
We use
the
firstsample
as
the
zero
reference
for
theremaining
grayscale
valu esin
thatline:there
iquite
s bit
a of
intensity
wobble
itnhe
camera signal,and
thisreferencing
makesthe
image
steadi er.
Picture
1s6howsthe
video
contentof
one
line,
the sampling
clock
and
thethresholded
digitaldata
obtainedfromthe
video
signal.
Theva luesof
one
(black
pixels)
in
the
middle
of
the
line
correspond
tothe
dip
itnhe
video
sign al,
which
wasin
turn
caused
bdark
ay
objectatthe
top
ocamera’s
f vision
field.
29
Back
porch
Picture
16:Video
signalcarrying
one
line
Picture
1at1.7
7,MHzsampling
rate,
showsthede lay
between
the
sampling
clock
and
6
the
return
odigital
f data.
TheA/D
converter
isp ipelined,introducing
delay
a ofour
f
clock
periods,
whichwaeccountfor
by
delaying
the beginning
opixel
f read/write
cycles.
TheI/O
attheend
othe
f line
extendspastthe
sam pling
clock
btyhe
same
amount.
The
field
ARisthe
pixel’saddressin
the
DPRAM(r ightport),
VIDistheconverter’s
output.
Firsttick
othe
f AD
clock
occursatthee nd
oback
f porch,
andtheresulting
black
reference
value
ilatched
s four
tickslater,as0x3 i1tnhe
VID
signal.
HD isthe
normalized
grayscale
value:notice
thatVID H
– D=
0x31
pastthe
black
reference.
30
DRisthe
thresholded
signal,
showing
itnhiscase balack
objectin
the
firstline.
OE_R,
WE_Rand
CE_Rare
the
memory
controlsignalsof
the DPRAMsrightport.
Picture
1A/D
7: pipeline
delay
Motion
vector
calculationsand
storage
othe
f digit alimageare
described
iS
n ection
12,
dealing
with
tpixel
he read/write
cycle
(p.
64).
6.2 Odd
field
Picture
1s8howsthe
beginning
othe
f odd
field
and the
DPRAM
I/Oassociated
with
it.
Atthistime,
parametersare
updated,
themotion
ve ctor
hasbeen
calculatedand
iwritten
s
31
out(notice
the
twelve
dipsin
the
WEsignal),
and Processu1pdatesthe
statusbyte
(notice
thatthe
semaphore
operation
SEM_Rbrackets the
statusbytereadand
write).
IRQ5
iasserted,
s triggering
the
handler
on
MCF5307 and
starting
the
parallel-port
communication
(see
Section
16,
onIRQ/PP).
Param Motion
vector Statusbyte
Picture
18:The
odd
field
32
7. FORMATTING
THEIMAGESCAN
The
design
receivesthe
sync
signalsalready
separa ted
fromthe
totalvideosignal.Ituses
the
syncstocontrolthe
digitization
and
framing, to
assign
coordinatesto
pixels,
andto
counttotalnumbersof
pixelsand
black
pixelsin
t he
image.
Forconvenience
in
analyzing
the
VCRvideosignal,
which
doesnothave the
even/odd
sync,
thissync
signal
isbeing
generated
internally.
Deciding
awhich
t pointsto
sample
the
video
signal i.e.
, generating
sampling
a clock
for
the
A/D
converter,
isthemain
functionality
derive fdromthe
synchronization
signals.
Verticalformatting
meansthe
selection
ovideo
f li nes,
while
the
horizontalformatting
means
saelection
odiscrete
f sampling
pointson
th ceontinuousvideo
signal.
The two
formatsdiffer
in
details,
andare
made
somewhatmo re
complexbtyhe
presence
othe
f
zoom(see
Section
10,
on
the
digitalzoom).
Also, in
thissystem,
sampling
ilimited
s to
the
even
field
othe
f frame.
Picture
1s9howsthe
formatting
geometry
and
the
si gnalsinvolved,
in
theabsence
of
zooming.
Zoomgeometry
ishown
s iP
n icture
26.
33
Verticalblanking VERT_SYNC
Scanned
line
density;
SCAN_LINE
Scanned lines
sync
per
field
Line
Area
othe
f videosignal; (fixed)
even
field=
digitized
area
Sampling rate
1.83
MHz;
signal1_7_MHZ
DLY_END
=0 Samplesper
line
(fixed)
Blackreference
sample; BP_END
Picture
1Formatting
9: and
sampling
7.1 Verticalformatting
Briefly,
theverticalformatting
circuit(in
the
sc hematic
VISION_IN)
determinesthree
things:
- atwhich
line
itnhe
evenfield
tsotartsampling
- atwhatline
density
tsocan
(how
many
linesto
ski bpetween
scans,
if
any)
- how
many
linesto
scan
(i.e.
when
tsotop)
34
7.1.1 Startingline
Obviously,
sampling
mustbe
suppressed
during
the
v erticalretrace
(verticalblanking),
and
when
the
3Xzoomisengaged,
over
the
top
third of
the
imageawell.
s Thisis
accomplished
beyxtending
the
duration
othe
f verti calsync
ttohefirstscanned
line,
counting
the
requisite
number
of
lines.
Aloadable counterand
small
a ASM,triggered
by
the
signalVSNCandclocked
bLy INE_SYNC,
produc tehe
signalVERT_SYNC,
which
extendsto
the
firstscanned
line.
0 W
VERT_SYNC
=
0
0
VSNC
TC_LD
1R
TC_EN
VERT_SYNC=
1
0
TC_VS
Picture
20:VERT_SYNCASM
7.1.2 Line
density
ComponentLOAD_CNT2
icasounter
whichreloadsan
e xternalvalueD_IN
whenever
it
runsout,
then
continuesrunning
ttohatvalue.It has
parovision
for
the
zero
count,
and
two
termcountsignals,
fullperiod
and
half
period Itruns
. while
itsTRG
inputishigh.
Thiscomponentisused
tsoetthe
line
density,
by clocking
iwith
t LINE_SYNC.
35
(A=0) W
0
TRG
LOAD_CNT
CNT_EN
(A=1) R
CNT_EN
0
TRG
0
TC
LOAD_CNT
Picture
21:LOAD_CNT2
ASM
36
7.1.3 Line
count
ComponentSCAN_LPFisastopping
counterwhich
irs esetasynchronously
otnhe
rising
edge
oits
fTRG
input.
ItsoutputGATE
remainshig fhor
the
duration
othe
f count;
afterwardsthecounter
sleepsuntilthe
nextreset. Thiscomponentisused
tcoountthe
scanned
lines in
the
evenfield.
(BA)
00 W1
SYNC_LD
CNT_EN
Theasynchronous resetstate
01 R
GATE
CNT_EN
0
TC
10 W2
Picture
2SCAN_LPF
2: ASM
7.2 Horizontalformatting
The
horizontalformatting
circuit(also
iV
n ISION_I N)
doesthese
four
things:
- decidesatwhatpixeldensity
tsoample
- producesthe
sampling
signalfor
the
black
referenc aefatixed position
ilnine
- decideswhere
itnhe
scanned
line
tsotartsampling pixels
- and
how
many
pixelsto
sample
per
line
(i.e.
whent sotop)
37
7.2.1 Pixeldensity
An
ASMand
loadable
a counter,
identicalto
those
i nLOAD_CNT2,generate
clock
a
signalatthe
pixelsampling
frequency,
byreducing the
systemclock
bfactor
ay
dependenton
the
zoom.
7.2.2 Blackreference
sampling
ComponentLOAD_CNSicasounter
which
synchronously loadsan
externalvalue
D_IN,
runsto
thatvalue
and
stops.Ithas
parovision
fo the
r zero
count,
andtwo
termcount
signals,
fullperiodand
half
period.
Itstartswh enitsSYNC_LOAD
inputgoeshigh.
Thiscomponentisused
tgoenerate
the
blackref.s ampling
signal(BPE),
bycounting
off
the
length
othe
f back
porch
itnhe
intervalsof
th seampling
clock.
7.2.3 Startingpixel
A
secondLOAD_CNScountsoff
thedelay
fromBPE
to the
firstpixel(zero,
or
one
third
of
the
line
for
the
3Xzoom).Itraisesthe
signal EN_SAMP,
during
whichpixelsampling
isenabled.
7.2.4 Pixelcount
An
ordinary
counter
countsthe
sampled
pixelsin
th leine,
andlowersthe
EN_SAMP
when
the
fullnumber
ofpixelsisreached.
38
7.3 Auxiliarycomponents
CLK_DELAY t–hiscomponentcreatesburstsof
itsinputclock CLK,
forthe
duration
of
the
inputCTRL,
only
delayed
bfixed
ay number
of
c lock
periods.
CLKisassumed
tboe
caontinuousclock
Thiscomponentisused
tcoreat clock
ae burstdelayedbfyour
clock
periods,
which
ineeded
s tloatch
the
outputfromt he
A/D
converter’spipeline.
HCP(half-clock
pulse) –passesto
Q
the
firsthighhalf-period
oCLK,
f fo llowing
the
rising
edge
oits
finputD,
and
only
that.Itisus ed
tcoonvertthe
term-countsignals
(which
last
faullclock
period)
into
half-period
p ulses.
Thecomponentisasynchronous,
and
usesthreeFF’s(A,Band
Cwhich
) mutually
clea each
r other,
according
ttohis
timing
diagram:
CLK
B Q
signifies
falip -flopicnontinuous
clear
Picture
23:HCPtiming
diagram
39
7.4 Signals
Sync
signalsare
active
low,
inverted
and
used
aas ctive
high
through
the
design.
In
order
to
processstill-frame
outputfromVCRs,
which
lack the
s odd/even
signal,
thissignalis
generated
internally.
LINE_SYNC active
- high delimiter
between
video
lines,
typica lly
4.7
microseconds.
Suppressed
during
the
odd
field.
VSNC active
- high delimiter
betweenfields,
typically
2 30
microseconds.Suppressed
during
the
oddfield.
VERT_SYNC derivative
- oVSNC.
f Extendsfromthe
beginning
of VSNCto
the
first
sampled
line
itnhe
field,
covering
verticalretrac aend
verticalzooming
delay.
Coversup
unused
synchronization
intervalinLINE_SYNC.
SCAN_LINE –active
high
during
each
scanned
line;reflectsth veerticalsampling
density
(every
line
oevery
r third
line,
setby
the zoomlevel).
Itsderivative
SCAN_L1
is
low
during
horizontalsyncs.
Both
signalsare
activ oenly
during
even
fields.
EN_SAMP when
- thissignalishigh,
pixelsampling
othe
fv ideo
line
ipermitted.
s This
signalishigh
inevery
n-th
video
line,
assetby SCAN_LINE,and
coverseither
the
entire
line
othe
r middle
third
oit,
fassetby
the
zoom level.
40
1_7_MHZ clock
- whichcontrolsthe
density
opixel
f samplin ogthe
f videolines.
At108
pixelsper
line,
thisclock
runsat1.83
MHzo5.4
r M
9 Hz,
depending
otnhezoomlevel
(see
clock-reducing
counter).
BP_END the
- single
pulse
indicating
theend
oback
f porch Itsdelayed
. derivative,
LATE_BPE,
isused
tloatch
the
black
reference
leve l.
Unlike
the
restof
thesampling
signals,
theseare
notaffected
btyhe
zoom.
AD_CLK triggering
- signalsentto
the
A/D
converter.Itc omprisesBP_END
and
the
1_7_MHZ
line
sampling
burstcoveredbE
y N_SAMP.
LATE_CLK l–ine
sampling
burst,
delayed
bsyeveralperiods( 4)
of
the
sampling
clock,
to
allow
for
pipeline
delay
itnhe
A/Dconverter.
T hissignalclocksthe
utilization
othe
f
digitized
signal.
DLY_END s–ingle
pulse
indicating
the
end
ohorizontal
f zoo ming
delay.
41
8. DIGITIZING
AND
THRESHOLDING
(schematic
ANALOG_IN)
Video
signalfromthe
camera
idigitized
s with
the AD876
converter
chip.The
circuit
generatesthe
sampling
triggerfor
theconverter,
A D_CLK,and
receives8-bitgrayscale
signal,
VIDEO,
in
return.
Video
signaliscorrectedfor
the
intensity
fluctua tionsby
subtracting
the
black
reference
value
fromit.
Thecorrected
video
ithresholded
s t strictly
ao black
and
white
signal,
C_BIT,
which
iboth
s stored
itnhe
RAMand
passed
t foramecomparators.
The
corrected
video
ialso
s passed
ttohe
back-end FPGA
for
histogram/threshold
calculation,
during
the
even
field.
In
theodd
fie ld,
thesignalTHR_RDYfromback-end
latchesthe
calculated
threshold
into
data
a regist er,
to
buesed
itnhe
nextframe.
Presently,
thesampling
idone
s o8nl1inesof
ever eyven
field
othe
f videosignal,
at108
pixelsper
line.
Thisyields
1a08x81
b/w
digitized imageframe,
atthe
correctNTSC
width-to-heightratio
o4:3.
f
8.1 Black-and-white
inversion
Which
side
othe
f threshold
iconsidered
s active,o fareature,
is
m
a atter
of
convention,
and
can
bseetby
run-time
a parameter.
Thechoice doesnotaffecttheedgedetection,
although
iaffects
t the
sensitivity
othe
f motion
d etector
somewhat.
42
9. SETTING
THETHRESHOLD
AUTOMATICALLY
9.1 Heuristic
procedure
Thisvision
systemoperateson
the
assumption
that the
scene
consists
relatively
of
luminoustargetfeature(s)
and
relatively
dark
unin teresting
background(orthe
reverse).
An
early
and
importantstep
ito
sset
balack-and-w hite
threshold
thatwillseparate
the
featuresfromthe
background,
greatly
reducing
the complexity
othe
f scene.
Setting
thisthreshold
manually
idaselicate
task. The
selection
iguided
s btyheapparent
simplicity of
the
b/w
image,and
once
set,
thethreshold
usual ly
workswellfor
raange
of
similar
images.Itwouldbdeifficultfor
the
navig ator
to
adjustthe
threshold
ifnlight,
and
smallerrorsin
the
threshold
can
alter
theresult dramatically.
Automatic
thresholding
wasimplemented
tm
o ake
the
vision
more
robust.
Finding
the
threshold
followsthisheuristic
proced ure:
a)
Constructthegrayscale
histogramof
the
image. Thisis
satraightforward
pixelcount,
accumulated
iannarray
o256
f grayscale
levels.
b)
Find
the
highestmaximumin
the
histogramand
as sume
thatitislocated
itnhe
middle
of
laarge
"hump"
representing
the
background.
c)
Find
the
lowestminimumon
one
side
othe
f highe stmaximum(in
thiscase,
the
brighter
side).
Setthe
threshold
ttohatgrayscale level:the
area
opposite
(brighter
than)
the
background
hump
representsfeatures.
43
d)
Disregard
threshold
choiceswhich
define
very
a smallfeaturearea
itnhehistogram,
since
they
typically
havenvoisualsignificance.
e)If
thesearch
for
m
a eaningfulminimumfails,
re verse
thegrayscaleandlook
for
featureson
the
opposite
end
othe
f histogramin
th neextvideo
frame
f)
Clear
the
histogram.
The
assumption
here
ithat
s thelowestminimum
give best
s separation
othe
f histogram
into
background
and
featuresof
interest,
and
the successultimately
dependson
the
grayscale
separability
othe
f image.
Picture
24gi vesan
illustration
othe
f procedure.
The
grayscale
version
othe
f image
inot
s currently used
itnhe
later
vision
stages;neither
isthe
value
othe
f threshold.
Forthatreason,
the histogram/threshold
processis
implemented
ihnardware,
onthe
back-end
XC4010
con nected
ttohefront-end
vision
processor
via
dedicated
a data
bus.
While
the
hist ogram/threshold
algorithmiswell
suited
for
implementation
icnode,
running
ion
tt h C
e oldFire
processor
would
have
complicated
the
dataflow
and
slowed
down
thefeatu rerecognition.
44
threshold
Picture
2Sample
4: histogram
9.2 Descriptionothe
f thresholdcalculationonthe
back-end FPGA
The
histogramisstored
isynchronous
an RAMcompon ent,
SYNC_RAM,
which
is
contained
itnhe
schematic
HIST_IO_RAM,
along
with elementswhich
build
(and
clear)
the
histogram.
Thereare256
word-sized
locations in
SYNC_RAM;the
histogramis
maintained by
presenting
the
grayscale
value
ttohe
RAMasthe address,
and
incrementing
the
corresponding
location
boyne(or setting
ito
tzero).
During
the
histogrambuild
(even
field),
histogram addressesare
thevideodata
coming
fromthe
front-end
FPGAthrough
the
busHIST_IO.
D epending
otnhe
nature
othe
f
image,
thesegrayscale
valuesmay
bienverted
bsyu btracting
themfrom255.
During
the
threshold
calculation
(odd
field),
HIST_ADDRisgen erated
internally
bcounter,
ay and
histogramvaluesappearon
the busHIST_OUT.
45
The
threshold
calculation
sweepsthe
histogramtwic e,
byconvention
idnownward
direction
(255
t0owhite
, tbolack).
The sweep
of histogramaddressesisgenerated
by
the
counter
C(P);specific
countvaluesare
latched in
MIN_LOCand
MAX_LOC
registers,
asthe
locationsof
histogramextrema
(s chematic
HIST_DP).
Minima
and
maxima
are
detected
bcyomparing
three
a djacenthistogramvalues,
which
flow
through
the
comparison
registersPL,
Pand
PR during
the
sweep.
These
are
the
relevantcomparisons,
with
black
dotsrepresenting relative
heightsof
theadjacentbarsof
the
histogram:
PRPL
P PRPPL
or (P>
PR) (•P<
PL) left-biased
maximum
or (P>
PL) (•P<
PR) right-biased
maximum
or (P<
PL) (•P>
PR) left-biased
minimum
or (P<
PR) (•P>
PL) right-biased
minimum
We
use
the
left-biasedcomparisons.
Currentextreme are
s latched
into
registers
CUR_MIN
and
CUR_MAX.
Since
waere
interested
itnh gelobalextremes,
locally
found
extremalvaluesmustbe
comparedwith
current largest/smallestvalues.
Allthe
comparison
logic
icontained
s itnhe
schematicHIST _MINMAX.
46
Step
dfrom
) previoussection
iimplemented
s itnhe schematic
HIST_INTEGRAL.
Histogramintegraland
the
featuresintegralareac cumulated
irnegisteredadders,
andthe
featuresintegraliscompared
with
anappropriatef raction
othe
f totalhistogramintegral.
Since
the
search
for
m
a eaningfulminimumcan
fail, the
success/failure
irecorded
s itnhe
flip-flop
MIN_FOUNDand
passed
ttohe
controlASM.
9.2.2 Control
ControlASMisimplemented
itnhe
Abelcodecompone ntHIST_ASM,
shown
iP
n icture
25
(two
pages).
The algorithmisby
itsnature
seq uential,
andcan
breoughly
divided
into
these
sixsteps:initialize
the
addressand
the
com pregisters,
search
for
themaximum,
re-
initialize,
searchfor
theminimum,
notify
fronten odinvert
r thegrayscale,clear
the
histogram.
Individualstatesand
logicare
explain ed
itnhe
ASMchart.
9.2.3 Signals
CP_SYNC_LD s–ynchronously
load
the
value
255
into
the
address counter
C(P).
CLR_REG s–ynchronously
clear
registersPR,
CUR_MAX,
MAX_LO C.
PL_LD,
P_LD,
PR_LD –enable
loading
ocomp
f registersPL,
Pand
PR.
47
CUR_MAX_LD,
MAX_LOC_LD,
CUR_MIN_LD,
MIN_LOC_LD e–nable
loading
of
maximum/minimumregisters.
PR_ASYNC_LD a–synchronously
setregistersPRand
CUR_MIN
to
“i nfinite”
value.
THR_RDY n–otify
thefrontend
thatthe
threshold
iready.
s
INV_GRAY s–etthegrayscale
inversion
for
the
nextframe.
CLR_WE e–nable
writing
zerosinto
the
histogram.
NORM_EN,
FT_EN –enable
the
accumulation
ohistogram
f and
feature integrals.
FFR f–eature
integralismeaningfulrelative
ttohe
en tire
histogramarea.
48
(CBA)
000 WAIT Keepthehistogram
location
at
CP_SYNC_LD C(P)
255
[ =255]
0
TRG Startthethresholdcalculation
PL ->DATA(255)
P; ->PL
011 IN12 Completedinitialization
oPL,
f P,
P_LD PRformax.search.
PL_LD Calculatetheareaothe
f
NORM_EN histogram.
010 S1
Decrementhist.location
untilC(P)=0,
i.e.countdown
untilCP_TC
=1
NORM_EN
1
Search forthemaximumfinished;
TOSTATE starttheminimum search.
CP_TC
IN21
0
Checkcurrent
if valueim as aximum
1 (P_MAXtrue),andiitis
flargerthan
P_MAX& CUR_MAX_LD
NEW_MAX MAX_LOC_LD
earliermaxima
(NEW_MAXtrue).
Savecurrentmaximum valueandits
0 histogram location.
Shift
DATA[C(P)] ->PL ->P ->PR
PR_LD simultaneously
P_LD
PL_LD
Picture
2Control
5: ASMfor
the
histogram/threshol cdalculation,
p.1
49
FROMSTATE
S1
CP_SYNC_L Resethistogram
location
to255
110 IN21
Loadhistogram dataalocation
t 255
PL_LD intoPInitialize
L. PRtoinfinity,clear
PR_ASYNC_LD
MIN_FOUND
101 S2
Decrementhist.location
until
FT_EN
C(P)=MAX_LOC,
i.e.CP_FIN=1
1
CP_FIN
0 Savecurrentminimum
andits
histogram
location
P_MIN& 1 CUR_MIN_LD 1
NEW_MIN MIN_FOUND Seenotebelow
MIN_LOC_LD
0
0
Shift
DATA[C(P)] ->PL ->P ->PR
simultaneously Resethistogram
CP_SYNC_LD
locationto255.
100 CLR
Note:
maifinimumwas
found,signalthatthe CLR_WE
thresholdiready
s forthe Resetentire
front-end. histogram to
0 zero.
CP_TC
If
thesearch
failed,invert
thegrayscaletotry
the 1
oppositesearch itnhenext
frame.
TO
STATE
Picture
2p.2
5,
WAIT
50
10. DIGITALZOOM
Itbecame
very
obviousduring
testflightsthatthe human
navigator
and
thevision
steering
systemneed
twodifferentperspectiveson the
ground
scene.
Ataltitudesof
200-
300
ft,
view
through
thewide-anglecamera
wasadeq uate
forgeneralorientation,
butthe
targetwasimpractically
smallfor
the
vision
syste mto
handle.
Using
narrow-field
a (f=16
mm)
lens,
or
equivalently,flying
close
ttohegrou nd,
producedgood
targetimagesifand
when
the
targetwaseverlocated.
Thecamera
wasfi xed
ttohe
body
othe
f plane,
but
even
with
ainndependently
movable
camera
the
navig ator
would
have
difficulties
spotting
areasof
interestthrough
the
narrow
fiel d.
A
camera
with
the
zoomlens,
or
even
two
different camerason
the
samegimbal,
would
solve
thisdifficulty
athe
t costof
additionalmec hanicalequipment.
However,
since
the
vision
systemoriginally
used
only
one
sixth
othe
f totalimage
information(sampling
every
third
line
othe
f even
field),
there
wasroom for
electronic,
instead
ooptical,
f
zooming.
Onlythe
vision
systemseesthe
effectof the
zoom,
since
the
navigator
currently
receivesno
digitized
image
feedback.
The
digitalzoom,
asimplemented,
is
3aX
zoom.
It amountsto
using
the
fullline
density
of
the
evenfield,
sampling
pixelsatthree
timest he
"no
zoom"rate.In
order
to
maintain
the
same
data
volume,
only
the
centralone-ninth
of the
video
image
iactually
s digitized
and
passed
ttohe
vision
system.
A
6Xzoomcouldbe implemented
buysing
the
linesin
the
odd
field
also, butsome
care
would
have
tboexercised
regarding theend-of-field
communication
betweenprocesses.
51
10.1 Zoom
implementationonthe
FPGA
The
zoomhasbeen
implemented
otnhe
front-endFPGA as
,partof
the
overalldigitizing
circuit.
Thezoomlevelispassed
ttohe
FPGAasa run-time
parameter,
andselection
a is
made
between
two
setsof
five
constants.
These
five constantsdefine
the
resolution
and
framing
othe
f image.
In
order
to
cope
with
high
sampling
frequency,
the FPGA'sclock
rate
iset
s to
33.3
MHz,
and
the
pixelread/write
cycle
wasmade
ashort
s an pdipelined
afeasible
s (see
description
iS
n ection
12).
Picture
2s6howsthe
zo om’sgeometry
and
thesignals
involved.
Refer
back
tS
o ection
7Formatting
, the
i mage
scan,for
details(p.33).
Pictures27
and
28give
aonverview
of
the
formatti ng,
modified
btyhe
zoom.
Picture
27
showsthe
sampling
clock
active
itnhe
centralone- third
othe
f lines
the
ofeven
field.
Greater
magnification
showsthe
sampling
clock
also limited
ttohe
centralone-third of
one
line,
with
the
black
reference
sample
following the
horizontalsync
(Picture
28).
The
non-zero
data
signal(DR)
isdue
tdark
ao objecti tnhe
camera’sfield
ovision.
f
When
the
zoomisengaged,
rotationsof
the
camera
p roduce
larger
displacements in
the
image.
Therefore,
theprocedure
thatcalculatesser vo
duty
cyclesmustalso
take
the
zoom
into
accountand
turn
thecamera
bsymaller
angles (see
the
programmoduleServo.s).
52
Verticalblanking
VERT_SYNC
Area
othe
f video
signal
(even
field)
density; per
field
Line
Samplingrate
5.49
MHz;
signal
1_7_MHZ
DLY_END Samplesper
line
(fixed)
Blackreference
sample; BP_END
Picture
2Formatting
6: and
zoom
53
Picture
2Vertical
7: formatting
(zoom)
ref. sampled
Picture
2Horizontal
8: formatting
(zoom)
54
11. ROUND
ROBINPROCEDUREFORDATASHARING
Presence
otwo
f processes(motion
detection
and
fea turerecognition),
running
on
separate
processors,
makesheavy
demandson
themem ory
containing
the
image
data.
In
thissystem,
accessconflictsand
buslogjamsarea voided
buysing
dual-port
a SRAM
chip
and
round-robin
a data
accessprocedure.
7
The
RAMchip
used
iCY7C007AV,
s an
asynchronous32K
xpart
8 byCypress
Semiconductor.Ithastwo
address/data
ports,
which can
read
simultaneously
fromthe
same
memory
location.
Thepartarbitratesread/wri te
accessconflictsin
hardware,
although
thatfeature
inot
s used
here.
Thechip
al so
has
baank
ohardware
f semaphores,
with
their
ownchip-selectsignalsand
arbitration logic.
Thisfeature
wasessentialin
8
implementing
the
round
robin
procedure.
In
thisscheme,
themotion
detection
(processP1)
r eadsfromonememory
area,
sayM1,
and
writesinto
another(M2).Itswapstheseareas on
each
new
video
frame.
Feature
recognition
(P2)takes
faew
frames'
time
t coomplete
one
calculation.
When
P2
needs
an
update,
itreadsthe
P1'sread
frame,
say M1
(P2
never
writes).
On the
next
video
frame,
P1 readsM2
and
writesto
M3,
thenswa psM2
and
Mu3ntilP2
claims
whichever
of
these
iP1's
s read
frameathe
t moment (see
Picture
29).
55
P1 P1
M1 M2 M1 M2 M1 M2
P1
P2 P2
M3 M3 M3
P2
Picture
29:Round
robin
Addr 1
M1
Addr 2
P1 M2 P2
Addr 3
M3
Picture
3DPRAM’s
0: memory
buffers
Buffers
M1-M3
are
implemented
adistinct
s memory
a reason
the
dual-portmemory
chip,
andwaiting
ieliminated
s completely.
P2 can startreading
the
P1'sread
frame
through
itsown
bus,
andthe
round-robin
motion
is performed
bsywitching
the
starting
addresses
M1-M3
of (see
Picture
30).
Read/writec onflictscannotoccur
onbuffer
access,
only
doublereads,
which
are
permitted
bty he
DPRAM.
56
Atany
moment,each
buffer
isassigned
tone
oth
f ese
three
states:P1
writes,
P1 reads,
P2
reads;any
buffercanbieanny
othem,
f andno two
buffersareever
thesame.
The
record
othe
f currentstate
imaintained
s idedi
an cated
location,
thestatusbyte,
which
is
updated
bPy a1nd
Po2envery
turn
othe
f round
rob in.
Since
Pa1nd
Pa2re
mutually
asynchronous,genuine accessconflictswilloccur
on
the
statusbyte.
These
are
avoided
bpyrotecting
the
s tatusbytewith
the
semaphore:only
one
portcan
hold
the
semaphore
(thisisarbitrated
by the
memory
chip),and
thatportupdates
the
statusbeforereleasing
the
semaphore.
The oth er
processstaysin
polling
a loop
until
accessisgranted,
buttheduration
othe
f busy
wai isno
t more
than
one-byte
a I/O
operation,
which
iinsignificant
s oneither
process or.
11.1 Statusbyte
The
statusbyte,atthe
DPRAMaddress0x08,
contain three
s two-bitfieldscorresponding
to
the
buffer
statesP1W,
P1Rand
P2R,
and
the
valu ieenach
field
ithe
s number
of
the
bufferassigned
ttohatstate.
0 0
0x08 P2R P1R P1W
Maintenance
othe
f statusbyte
ivery
s simple.
On r eset,
itsvalue
iset
s to
0b00100100
(0x24),
which
meansthat:
buffer
- zero iP1's
s writebuffer
buffer
- one iP1's
s read
buffer
buffer
- two iP2's
s readbuffer
57
Processs1wapsthecontentsof
fieldsP1Wand
P1R (interchangesitsworking
buffers).
Processs2wapsthecontentsof
fieldsP1Rand
P2R (releasesthe
buffer
itjustread
and
takesup
the
reading
buffer
of
the
Process1).
Bet ween
updates,
each
processmaintainsa
private
copy
othe
f statusinformation;otherwise, itsworking
buffer(s)
could
change
in
mid-cycle,
with
disagreeable
results.
Picture
3s1 howsthe
allowed
swapsof
the
status
byte
values.
P1
0x24 0x21
P2 P2
0x18 0x09
P1 P1
0x12 0x06
P2
Picture
31:Statusbyte
values
58
11.2 Round Robinonthe
front-endFPGA
The
procedure
bwy hichthe
two
processesshare
buff ersof
image
data
itnhe
DPRAM
hasalready
been
described
earlier.
Thissection
d ealswith
the
implementation
othe
f
round
robin
otnhe
FPGAside,
asthe
componentROUN D_ROBIN.
The
privatecopy
othe
f statusbyte
residesin
the register
SBYTE_REG,which
iread
s
fromand
written
ttohe
DPRAMaddress0x08.
Notice the
peculiar
ordering
oinput
f bus
leads,
which
accomplishesthe
swapping
ofields
f P1 Rand
P1W.
Currentaddressesfor
thethree
image
buffersresid ietnhree
countersdrivenbtyhe
LATE_CLK(see
schematic
RR_ADDRESS).
FieldsP1Ran P
d 1Ware
used
toperate
the
multiplexer
which
selectsthe
currentbuffer
fo read
r owrite
r operations.
The
ASM(see
picture
33)
isstraightforward:itpol lsthe
semaphore
foraccess,
readsand
writesthe
statusbyte,
then
releasesthe
semaphore Picture
. 3s4howsthe
timing
diagram.
SEM_IN r–ead
value
othe
f semaphore.
SEM_OUT w
– ritten
value
othe
f semaphore.
RR_SE s–emaphore
enable
(the
semaphore’schip
select).
SEM_WE,
SEM_OE,
SEM_D_TSB c–ontrolsignalsfor
the
semaphoreI/O.
RR_CE c–hip
enable
forthe
regular
RAMarea.
SB_WE,
SB_OE control
- signalsfor
the
RAMI/O.
STAT_A_TSB a–ddressTSB
controlfor
theentire
round
robin
se quence.
59
Picture
3Semaphore-protected
2: update
othe
f stat usbyte
60
(CBA)
000 WAIT
0
TRG Start
theupdate
othe
f status
byte
001 WSEM
SEM_WE
RR_SE Tentatively
write
zero
into
the
SEM_OUT=0 semaphore
RSEM011
SEM_OE Readbackthe
semaphore,to
checkif
RR_SE the
write
succeeded
1
SEM_IN Poll
semaphore
until
zero
iread
s back
0
111 WR
Pause oneclockperiod
taovoid
a
glitchotnhe
OEsignal
101 RSB
SB_OE
RR_CE Read
the
status
byte
100 WSB
SB_WE
RR_CE Write
back
the
status
byte
withfields
P1R
andP1Wswapped
110 WW
Pause oneclock
periodtaovoid
a
glitchotnhe
WEsignal
010 QSEM
SEM_WE
RR_SE Release
the
semaphore
bw
y riting
one
SEM_OUT=1 into
it
DONE
Picture
33:Round-robin
ASM
61
WAIT WSEM RSEM WSEM
-- RSEM
WR-- RSB WSB WW QSEM WAIT
TRG PulltheASM
out ofwait
RR_SE_ Semaphoreenable&
controls
Semaphorevalueonwrite
RR_CE_ RAMenable&
controls
SB_OE_
Picture
34:Round-robin
timing
SB_WE_ p.
62
Address
TSB
62
11.3 RoundRobinonthe
MCF5307processor
Implementation
othe
f round
robin
procedure
icnode isstraightforward:
the
flowchartis
almost
identicalto
the
ASMchartin
Picture
33. The
procedure
iinvoked
s once
ienach
passof
the
featurerecognition
loop
(see
module
ma in.c).
Itpollsthe
semaphore
for
accessand
readsthe
statusbyte
when
the
accessi granted;
s swapsthefieldsP1Rand
P2R,
updatesthe
statusbyteand
releasesthe semaphore.
The new
P2R(in
the
localcopy
of
the
statusbyte!)
isused
tsoelectthe
starting addressof
the
read
buffer,and
that
addressismadeavailablettohe
vision
code
itnhe globalvariable
IMAGE_FRAME.
The
subroutine
round_robin
icontained
s itnhe
prog rammodule
RoundRobin.s,
along
with
the
subroutine
config_cs,which
configuresthe leftport(ColdFire
side)
of
the
DPRAM.
Note
otnhe
DPRAMaddresseson
MCF5307:
Thehardwa re
iconfigured
s stohatthe
ColdFire
processor
usesitschip
select f4or
the
semaphore
bank
othe
f DPRAM,
and
the
chip
selectf5or
the
regular
storagearea.
ColdFi rechip
selectsare
assignedblocksof
9
addressspace
MB
2 in
size; consequently,
thebase
addressesfor
the
semaphor esand
the
regular
RAMbecome0xFF800000
and
0xFFA00000
re spectively,eventhough
they
are
contiguousin
the
DPRAM’saddressspace
(see
Pi cture
4o8pn98).
. ColdFire
generatesthe
proper
addressin
the
lower
15
bits, and
the
high
bitsserve
only
taoctivate
the
rightchip
select.
63
12. THEPIXELREAD/WRITECYCLE
The
pixelread/write
cycle
icentral
s to
theearly vision
processing:
itstoresthe
digitized
B/Wimageand
performsthe
motion
detector'sframe comparison.
Thissection
describes
the
cycle
suitablefor
the33.3
MHzsystemclock
an tdhe
5.4
MHzpixelsampling
rate.
Eventswithin
the
cycle
are
sequenced
btyhe
state machine
PIXEL_CYCLE,
which
is
clocked
btyhe
systemclock
and
runsonefullseque nce
per
period
othe
f sampling
clock.
The
pixelcycle
accessestwo
memory
buffers,
P1Ran d P1W;corresponding
addressesof
the
pixelin
these
two
buffersare
determined
btyh reound
robin
algorithm.Pixel
calculation
itriggered
s btyhe
delayed
sampling
cl ock,LATE_CLK,
whichalso
incrementsthe
bufferaddresses.
The
pixel's
readaddressiscalculated
during
the high
time
othe
f sampling
clock,
andthe
value
istable
s otnhesignalM_BIT
one
clock
perio ldater;thisisthe
pixel'svalue
from
the
previousframe.
Thethresholded
signal,
C_BIT, becomesavailable
around
the
same
time.
Both
bits are
presented
ttohe
comparator/accumulator
(see
Se ction13.3,
the
description
othe
f FPGAdesign,
p.70,
fordetails) and
the
incremented
value
othe
f
motion
vector
isclockedionne
period
later.
The
addressisnow
switched
ttohe
pixel'swrite
ad dress,
andotnhe
rising
edge
oWE,
f
one
clock
after
theaddressswitch,
thenew
pixelv alue
iwritten
s ttohe
write
buffer.
The
64
entire
read/writecycle
takesfive
clockcycles:at 33.3
MHz,
thisissufficiently
fastto
complete
allpixelprocessing
athe
t sampling
rate.
Inaddition,
there
are
RAM-controlling
signalsin
t he
cycle:outputenable,write
enable
and
the
three-state
outputbuffer
on
the
zero
bito the
f data
line.
Allof
theseareactive
low.
Picture
3s7howsthe
timing
diagramof
the
rea d/writecycle,
onecolumn
per
state.
Picture
3s5howsthe
read/write
cycle
driven
btyhe 33.3
MHzclock,
andrunning
athe
t
fullsampling
rate
othe
f 3X
zoom.
Picture
3Pixel
5: cycle
65
(CBA)
000 WAIT
0
TRG
001 CALC
ZB_OE_
ZB_CS_
011 SW
CALC_EN
ZB_OE_
ZB_CS_
111 WR
CALC_EN
ZB_WR_BUF
ZB_CS_
110 DONE
ZB_WR_BUF
ZB_WE_
ZB_CS_
Picture
36:PIXEL_CYCLE
ASM
66
WAIT CALC SW WR DONE WAIT Picture
37:Pixelcycle
timing
TRG PulltheASM
outofwait
C-BIT Databitfrom
videostream
M-BIT Databitfro m
memory
CALC_EN Enablethebitcomparison
ZB_WR_BUF 0-access
P1R;1 -access
P1W
ZB_TSB_ Open
TSBtowriteC -BIT
ZB_CS_ CSforthepixelcycleI/O
67
13. FRAMECOMPARISON
AND
THEMOTION
VECTOR
13.1 Methodology
Motion
detection
itnhissystemislimited
tloinea displacements
r of
the
entire
scene,
since
waere
interested
only
idnetecting
changesd ue
ttohe
movementsofthe
plane
(ego-
motion).
Driftdetectionwould
perhapsbe
more
a ac curate
term.
Formula
used
tcoalculate
the
motion
vector
isasf ollows:
∑ x (∆pix)
∑ pix
∆=
i i
Thisisessentially
the
formula
for
the
dipole
mome ntof
the
displacement,
with
the
previousframe’spixelscounting
athe
s negative
ch arge,
andthe
currentframe
as
positive.
Summationsare
over
the
entire
image, pix
ithe
s pixelvalue
(zero
oone),
r ∆pix
isthe
difference
betweenconsecutive
frames,and xi isthe
i-th
pixel’sposition.
The
normalization
constantin
the
denominator
issimply the
number
of
black
pixelsin
the
image.
Motion
detection
itnhe
vision
oinsects
f with
comp osite
eyesutilizesthe
principle
of
consecutive
activation/deactivation
oreceptors;
f d irection
omotion
f idetermined
s btyhe
10 11
pattern
oneural
f wiring
betweenadjacentreceptors (eyeletsin
the
composite
eye).
Our
detector
hasno
pixeladjacency
information,
an cdannotdetectlocalmotion
within
the
image.Instead,
itobtainsan
integrated
value of
spatialdistancesbetween
activated/deactivated
pixels.
Forsimple
driftmoti on,
thisisan
adequate
motion
vector,
68
with
the
caveatthatthe
detector
issensitive
tao ppearance
onew
f objectsin
the
periphery,
which
iinterprets
t asmotion.
13.2 Computation
Pixelsare
processed
irnealtime,
andeachpixelc ycle
containsthe
following
steps:
corresponding
- pixelfrompreviousframe
iread
s f romthe
DPRAM
bufferP1R;
current
- and
previouspixelare
presented
ttowoc omparators,which
calculate
the
componentsof
the
motion
vector;
current
- pixelisstored
itnhe
DPRAM
buffer
P1W.
Each
b/w
pixelisstored
itnhe
zero-th
bitof
bay te,
which
makesaddressing
simpler
and
faster.
Higher
bitsof
these
bytesare
notused.
13.3 Designcomponents
line-in-frame
counter counts
- scanned
video
lineswithin
one
frame.
Use dathe
s vertical
(Y)
coordinate
othe
f currentpixel.
pixel-in-line
counter counts
- the
sampling
clock
(LATE_CLK),
starting
a the
t beginning
of
each
video
line.
Usedathe
s horizontal(X)
coo rdinate
othe
f currentpixel.
pixel-in-frame
counter counts
- the
sampling
clock
(LATE_CLK),
starting
a the
t
beginning
ofaframe.
Itresetsto
the
starting
ad dressof
the
frame
itnhe
SRAM,
and
its
value
iused
s athe
s addressof
the
SRAM
byte
that containsthe
currentpixel.
69
COMP_ACCUM comparator/accumulator;
- thiscomponentadds/subtr actsthe
value
on
the
inputbusNUM[31:0]
to/fromthe
currentvaluei intsinternalregister.
Thecurrent
value
ialways
s availableotnhe
outputbusSUM[31: 0].
Thesign
othe
f operation
dependson
the
valuesofCBIT
and
MEM,asfollows:
CBITMEMoperation
0 0 none
1 0 add
0 1 sub
1 1 none
Thisoperation
idesigned
s tcoapture
the
differenc esin
pixelsof
adjacentframes.Itis
enabled
btyhe
ENsignal.
ASYNC_CTRL
resetsthe
re gister
value
tzoero
asynchronously,
andnoperationstake
place
while ASYNC_CTRL
ihigh.
s
black
pixelcounter counts
- the
sampling
clock
(LATE_CLK),
starting
a the
t beginning
of
farame,
only
iC_BIT
f ishigh
otnhe
rising
edg oethe
f clock.
Countofblack
pixelsin
one
frame.
13.4 Signals
the
currentpixelof
the
currentframe.
70
M_BIT current
- pixelof
the
previousframe,
retrieved
fr omDPRAM
andcompared
with
the
C_BIT
to
detectmotion.
CALC_EN e–nable
signalfor
the
comparison;outputof
the
P IXEL_CYCLE
sequencing
ASM.
71
14. WRITING
THEMOTION
VECTOR
TODPRAM
Atthe
beginning
othe
f odd
field,
vectorcomponent and
s the
normalization
constantare
written
iD
n PRAM,
atthe
address0x0C,
asthree
lon gwordsin
the
big
endian
order.
ComponentVECT_OUT
handlesthatprocedure.
VECT_OUT
hasanaddresscounter,
w
a ord
counter
and bayte-in-word
counter.
The
latter
two
countersoperate
the
multiplexerswhich selectthe
proper
byte
foroutput,
and
the
whole
procedureconsistsof
satraightforwardd ouble
loop,
corresponding
ttohree
wordsand
four
bytesperword.
TheASM
chartissho wn
iP
n icture
39.
TRG t–he
trigger
signal.
…_CNT_EN,
…_CNT_LD –controlsignalsfor
the
counters
WE_OUT,
TSB DPRAM
- controlsignals
DONE e–nding
signal;signalto
the
nextstage
tporocee d.
72
Picture
38:Vector
output
73
(BA)
00 WAIT
ADDR_CNT_EN
ADDR_CNT_LD Initializetheaddress
and
WORD_CNT_LD wordcounters
0
TRG Startthevectoroutput
1
01 W
WORD_CNT_EN
BYTE_CNT_LD Counttheword,restartthe
bytecount
11 B1
WE_OUT Putout
bayte
10 B2
ADDR_CNT_EN
BYTE_CNT_EN
Incrementthebyte’s address,
countthebyteitnheword
0
BYTE_TC Finishedwith
oneword?
1
0
WORD_TC Finishedwith
allthewords?
1
DONE
Picture
39:VECT_OUTASM
74
15. PARAMETERSOF
THEFRONT-ENDFPGA
The
componentPAR_IO1
handlesthe
parameter
logisti c.
Currently,
there
are
three
byte-
size
parametersallocatedttohe
front-endFPGA,
st arting
athe
t DPRAM
address0x09:
- b/w
threshold,
(0 2–55)
- one
byte
obitwise
f parameters:
o bit0 b/w
- inversion;zero
standsfor
darkfeatures,
one for
brightfeatures
o bit1 zoom
- level:zero
for
no
zoom,
one
for
3Xzoom
- mailbox,
written
tD
o PRAM
on
eachframe:
o bit0 z–oomlevelindicator:zero
for
no
zoom,
one
for
3 X
zoom
Parameterscan
baeddedaneeded,
s byfairly
a stra ightforward
extension
othis
f
component.
Defaultvaluesof
the
parametersare
contained
int he
circuit,
asbyte-size
constants.
On
the
firsthigh
TRG
after
power-up,
thatison
the
f irstodd
field,
defaultparametersare
written
tD
o PRAM
atconsecutive
addresses.
On subse quentTRGs,
each
parameter’s
addressispresented
otnhe
busADDR_OUT,andthe
c orresponding
parameter
selection
signalgoeshigh,
fortheduration
otwo
f clocks.
T hisallowsthe
currentparameter
values
to
breeadfromDPRAM
and
latched
into
parameter
re gistersin
the
circuit.
The mailbox
parameter
iswritten
tD
o PRAM
on
each
frame,
tobe read
btyhe
microprocessor.
The
ColdFire
programupdatesthe
parametersin
the DPRAM
during
theIRQ5
handler,
and
readsthe
mailbox.
Thismechanismallowsfor
c hangesin
the
parametersto
bm
e ade
75
atrun
time,
e.g.
by
operator
commands,
aswellas for
the
FPGA
to
send
signalsto
the
ColdFire.
Currently,
themailboxiused
s tnootify the
ColdFire
when
the
zoomcommand
hasbeen
radioed
ttohe
front-end
FPGA.
SequencerASM
for
PAR_IO1
ishown
s iP
n icture
40. Itssubcomponent,INIT_BOX,
raisesPAR_INIT
once
after
the
power-up,
anditsAS M
isshown
iP
n icture
41.
TRG t–he
trigger
signal;high
once
perframe,atthe
s tartof
odd
field
TRG_MACH d–erivative
trigger,
produced
btyhe
initializatio cnomponentINIT_BOX.
ADDR_CNT_EN,
ADDR_CNT_LD s–ignalsthatcontrolthe
addresscounter
PAR_INIT h–igh
onfirstoccurrence
othe
f TRG;passesthe
d efaultparameter
values
onto
the
PAR_IO
bus.
A_TSB,
D_TSB,OE_OUT,
WE_OUT R
– AM
controlsignalsinvolved
ipnarameterI/O
to
and
fromthe
DPRAM.
PAR_SEL0,
PAR_SEL1,PAR_SEL2 p–arameterselectors;enable
latching
othe
f
corresponding
parameterin
the
appropriate
data
reg ister
in
thecircuit.
PAR_IO d–ata
buswhich
carriesthe
parameters.
76
(CBA)
000 WAIT
0
TRG_MACH
1
001 W1
PAR_SEL0
A_TSB
0
PAR_INIT
WE,
D_TSB OE
011 A1
A_TSB
0 0
PAR_INIT PAR_INIT
1 1
D_TSB OE D_TSB OE
010 W2 111 W3
PAR_SEL1 PAR_SEL2
A_TSB WE
A_TSB
D_TSB
0
PAR_INIT
1 101 A3
A_TSB
WE,D_TSB OE D_TSB
DONE
110 A2
A_TSB
TO
STATE
WAIT
77
(BA)
00 SLEEP
Theresetstate
0 st
TRG Waitforthe1 TRG
1
PAR_INIT_SET RaisethePAR_INITsignal
onceafterthereset
11 NEXT 01 WAKE
0
PAR_INIT_CLR 1 TRIG_MACH Triggeronecycleoparameter
f
TRG
updates
Lowerthe
PAR_INITsignal 10 WAIT
andwaitfornext
TRG
Receivethesignalthatthe
0
DONE parameterupdateifinished
s
Picture
41:INIT_BOX
ASM
78
Picture
42:FPGA
parameters
79
16. THE
IRQ5/PARALLELPORT
COMPLEX
A
tightly
coupled
hardware/software
subsystem,
cent eredaround
theInterrupta5nd
the
parallelportof
the
MCF5307,
coordinatesthe
back- end
dataflow
in
the
vision
system.
Herewdeescribe
thatsubsystem.
16.1 IRQ5handler
Early
itnhe
odd
field,
after
the
motion
vector
has been
written
ttohe
DPRAM,
back-end
FPGAgeneratesanIRQ5,
as
haardware
signalto
the MCF5307.
Whenthe
processor
enterstheIRQ5
handler,the
interruptisacknowled ged
bhandshake
ay otnhe
parallel
port,
viatwo
signals,
ACKand
RDY
(bits14
and15) .
The
motion
vector
isread
fromDPRAM
(itwaswritte bnefore
theIRQ5,so
there
ino
s
accessconflict),and
iisnormalized
t bdyividing by
the
black
pixelcount.
Thisoperation
isperformed
hereand
notin
the
front-end
FPGA,wh ere
the
vector
isgenerated,
because
of
long
integer
divisions.
Subjectto
some
size
restrictions,
thevector
istr anslated
into
incrementsin
the
servo
cycle'spulse
widths,
andthese
incrementsare
used to
update
the
currentpulse
widths.
Notice
thatthe
desired
position
othe
f camera
ias lwaysknown
ttohe
vision
system(in
the
formof
calculated
pulse
widths),
butthatthe actualposition
may
notbeknown
irneal
time.
80
16.2 Duty-cycle
generator
The
generator
of
the
pulse
modulated
servo
signals resideson
the
back-endFPGA.It
receivesthe
pulse
widthsfromthe IRQ5
handler,and
producesthe
corresponding
waveforms.
Pulse
widthsare
passed
a14-bit
s number on
sthe
parallelport,in
protocol
a
synchronized
btyheACK/RDY
signals.
16.3 Servomotionfeedback
When
the
pulse
width
changes,
theservosstartmovi ng
into
the
new
position.
Theservo
circuit,
builtaround
PIC16F877
a microprocessor,
d etectsthe
pulse
changeand
beginsto
monitor
the
angle
othe
f servo
shaft,
asan
analog signal.Itassertsthe
servo-move
signal,
which
remainshigh
untilthe
servoshave
stopped.
In
thisfashion,
theinstantaneousinformation
abou the
t camera
position
idecoupled
s
fromthe
vision
system.The
vision
merely
issuesth deesired
position,
andreceives
confirmation
when
thatposition
ireached.
s
While
the
servo
motorsare
moving
ttoheir
newposi tion,
theIRQ5
inot
s being
generated,
since
the
motion
detector
would
countera ctthe
displacementmotion,
leading
to
unsteady
movementofthe
camera.
16.4 Displacement
vector
When
iisnot
t communicating
with
theIRQ5
handler, the
duty-cyclegenerator
waitsfor
the
displacementvector
transfer,
initiated
btyhe MCF5307.
The displacementvector
is
81
being
calculated
btyhe
Process2,
andwhen
ready, itistransferred
ttohe
back-endFPGA
in
the
same
way
athe
s motion
vector.
In
thistransfer
sequence,however,
theservo-motio snignal(theresponse
tothe
new
displacementvector)
ispassed
back
ttohe
MCF5307, forcing
the
Process2into
busy
a
waituntilthe
servoshavestopped
moving.
Thismi ghtappear
wastefulatfirst,
butitis
easy
tsoee
thatProcess2really
mustpause
during the
servo
motion.
Thesnapshotfor
feature
recognition
mustnotbe
taken
untilthe
cam era
hasmoved
ttohe
new
location.
Otherwise,
thechange
inthe
image
would
notbe
reg istered,
andthe
nextcycle
ofeature
f
recognition
would
end
uwp orking
with
stale
data.
16.5 Saccadic
blanking
On
the
face
oit,
fthissuppression
oimage
f sampli ng
during
camera
motion
resemblesthe
12
phenomenon
osaccadic
f blanking
ihnuman/animalvis ion.Itiswell
knownthatthe
sensitivity
othe
f optic
nerve
isuppressed
s while saaccade
(raapid
eyemovement)
isin
progress,
andiisplausible
t thatthe
purpose
otf hissuppression
ito
spreventvisual
confusion
ibniologicalsystemsaswell.
Interestingly,
there
isome
s question
whether
thes uppression
othe
f opticnerve
signalis
triggered
btyhe
blurring
oretinal
f image
oby
ra signalfromtension
sensorsin
the
eye
13
muscles.In
robotic
a systemitismuch
easier
to
detects ervo
motion
than
image
blur,
and
the
choice
omechanism
f isobvious.
82
16.6 Descriptionothe
f IRQ/PP
circuit
onthe
back-endFPG A
The
circuitdesign
icontained
s itnhe
schematicDU TY_CYC_PP.
Control
of
the
process
iscarried
outby
the
ASM
componentIRQ_PP1,
descri bed
itnhe
previoussection
(see
also
ASM
chart,
Picture43,
andtiming
diagrams,Pi ctures44
and
45).
The
data
path
leadsfromthe
parallelportto
two
1 4-bitregisters,
which
hold
the
current
pulse-width
values.
These
valuesare
itnurn
availa ble
ttohe
square-pulsegenerators.
The
pulsegeneratorcomponent,
CYCLE_GEN,
contains faixed-value
counter,
which
measuresthe
2m
0 period
s othe
f servo’sduty
cycle and
, loadable
a counter,which
measuresthecurrentpulse
width.
Asimple
two-sta te
ASM
switchesbetween
highand
low
signallevels.
The
servo-move
signalpassesthrough
the
component DIP_FILTERwhicheliminatesthe
short(lessthan
CLK
a cycle)
dipsin
the
signal(n oise).
ACK,
RDY h–andshakesignalson
the
parallelport:
ACK pin
- 14,
output
RDY pin
- 15,
input
IRQTRG s–ignalfromthe
front-endFPGA
to
starttheIRQ5 communication
athe
t start
of
odd
field
(see
Picture18).
83
IRQ h–ardwarerequestforInterrupto5C
n oldFire.
SERVO_MOVE t–he
cleaned-up
servo
motion
signal;also
sentout to
pinon
0the
parallelport.
Pins1-13
are
allinputpins
LATCH1,LATCH2 r–egister-enable
signalsto
capture
the
pulse
widt vhalues.
PWM1,
PWM2 g–enerated
servo
duty
waveforms.
84
(DCBA)
0000 P2
0
0
IRQTRG RDY
1 1
LATCH 1
1110 SVS
PIN0_TSB
0
RDY ACK
1 0
SM
LATCH 1 1
0 0
RDY SM
1 1
1101 P4
LATCH 2 PIN0_TSB
ACK
0
RDY
1
1001
IDLE2
Picture
43: 0
RDY
IRQ_PP1 ASM 1
85
WhileRDY=0,poll PollforRDY=1C.F.
; DropIRQ5, PollRDY=1, PollRDY=0, Picture
44:
IRQTRG=1;start acknowledges IRQ5 pollRDY=0 latch
value1 latch
value2 Communication
with
the
IRQ5handler
P2 P2 IRQ IRQ IRQ IDLE1 IDLE1P0 P0 P1 P1 P2
SignaltostarttheIRQ5exchange,
IRQTRG onceperframe
IRQ5,throwsColdFireintothe
IRQ5 handlerroutine
RDY
Acknowledgmentsignalgenerated
ACK by
theASM
Latch thevalueopnarallelport
LATCH
1 intoregister1
Latch thevalueopnarallelport
LATCH
2 intoregister2
86
While IRQTRG=0, PollRDY=0, PollRDY=1; Poll
SM=1,servos
are Poll
SM=0, end
oservo
f PollRDY=1 PollRDY=0,
poll
RDY=1, latchvalue
2 servomotion moving motion(usually
long
a interval) endtransfer
latchvalue
1
P2 P3 P3 TSB TSB SVS SVS SVS SVM SVM SVM P4 P4 IDLE2 IDLE2
P2
RDY
ACK
LATCH
1
LATCH
2
PIN0_TSB
SM
Notes:
PIN0_TSB
1) reversesthe
sense
pin
ofon
0the PP.
When
low,pin
0is
the
input
bitzero.
When high,
pinoutputs
0 the
SM(servo
move)signa to
lC.F.
SM
isthe
2)
filt eredandsynchronized
inputfrom
the
camera
servob oard.
Whenhigh,
atleastone
servomotorisinmo tion.
Picture
45:Communication
with
Process2
87
17. AUXILIARYFEATURES
The
systemhasseveralauxiliary
features,
which
do notpertain
tiots main
function,
but
which
are
necessary
for
the
deploymenton
flying
a airplane.
17.1 Serialcommunicationwiththe
MCF5307
The
ColdFire
processor
hastwo
built-in
UARTs,
and the
SBC5307
board
hastwo
serial
ports.
One portisconfigured
and
active,
andthe other
can
beasily
madesaowell.
s
There
isasmallI/O
utility
library,
which
allows transmission
obyte
f strings,
and
transmission/reception
oindividual
f bytes.
17.2 Diagnostic
datalogging
Lower
bank
othe
f SBC5307'sDRAM
isorganized
aas four-megabytecircular
buffer,
to
be
used
for
logging
dataduring
the
system'soperat ion.
Thisbuffer
isinitialized
upon
board
reset,
sothe
systemshould
notbe
casually
r eset(or
turned
off!)
before
the
diagnostic
data
are
retrieved.
Autility
library
a llowsfor
recording
obytes
f and stringsof
bytes,
andfor
dumping
the
buffer'scontentsto
the serialport.
Data
shouldbceaptured
to
faile
using
terminal
a software.
Note:pressthe “D”
key
tsotartthe
data
dump
ttohe
terminal.
17.3 Soft
restart
ofthe
visionprogram
The
non-maskableInterrupti7used
s tiomplementa softrestartof
the
vision
programon
the
MCF5307.
WhenIRQ7
ireceived
s (as
haardware signal),
thecorresponding
handler
cleansup
the
cache
and
the
parallelport,
releases allheap
allocations,
resetsthe
stack
and
88
returnsto
the
main
entry
point.In
thisway,
thev ision
algorithmcan
bpeulled
outof
some
confused
statewithoutresetting
theentire
Co ldFire
board.
17.4 Radiocontrols
The
systemcontains
daecoder
(implemented
oanX
nC 9572
CPLD)
whichacceptsfour
PWM
signalsfrom
raadio
receiver,
andconvertsthe minto
four
logic
signals.
The
decoder
usesthe
flightcontrol'spower
supply
and receivesno
inputfromthe
restof
the
system.
Itspurpose
ito
simplementradio
control over
the
following
functions:
power
- on/off:circuitry
unrelated
tm
o anualcontr olsof
the
plane
can
bsehutoff
from
the
ground
ianenmergency.
Anelectronic
switchi attached
s ttohe
battery
pack
for
that
purpose.
reset:
- vision
boards(and
possibly
other
componen ts,
in
the
future)can
breesetto
a
clean
state
icnase
oirreparable
f malfunction.
soft
- restart:the
vision
systemcan
bberoughtba ck
tiotsclean
state
withoutresetand
the
consequentlossof
diagnostic
data.
TheFPGA
part of
the
systemisresetwithoutharm.
The
vision
can
also
bheeld
irnestartcontinuously in
,which
case
thecamera'sgaze
is
fixed,
thetracking
istopped
s and
ndoataare
logg ed.
Thisisusefulto
keepthe
camera
fromturning
around
aimlessly
during
takeoffand
la nding.
zoom:
- the
zoomof
the
vision
systemcan
bteurned on
ooff
r by
the
operator.
89
(BA)
00 WAIT
Waitfortheriseothe
f
0 radiosignal
CHNL
Loadthe1.5mcounter
s
LD
01 RUN
If
thecounterexpires
1
CHNL
&TCN whilesignalisup,decode
tologic1
0
0 1
ZERO CHNL If
thesignaldropswhile
counterisstillrunning,
decodetloogic0
ONE
11 WCH
Waitforthesignalto
dropbeforeresettingfor
thenextcycle
0 1
CHNL
Picture
46:Radio
decoder
ASM
(module
CHANNEL)
17.4.1Radiodecoder’ssignalsinthe
schematic
CHANNEL
CHNL p–ulse-width
modulated
signalfromthe
radio
recei ver.
LD s–ynchronousload
signalfor
the
1.5
mcounter.
s
TCN t–he
counter’sexpiration
signal(termcount).
ZERO/ONE –clear/setthe
logic
output.
90
18. FEATURERECOGNITION
ONTHEMCF5307
PROCESSOR
Functioning
othe
f programsrunning
otnhe
ColdFire processor
isbestunderstood
in
termsof
the
time
periodsinvolved.
Onecomponent, centeredaround
the
motion
vector,
runsin
response
toIRQ5,
which
iissued
s btyheFP GA
once
per
video
frame.Ithas
already
been
describedearlier,
in
Section
1o6tnh IeRQ5/parallelportsubsystem.
The
other
component,
feature
recognition,
runsin
t he
backgroundrelative
ttohe
interrupts.Ithasno
time
requirementsimposed
on itby
the
inputsignal,
only
thegeneral
consideration
thatfasterprocessing
leadsto
bette tracking.
r Herewdeescribe
that
procedure.
The
feature
recognition
consistsof
the
following
c omputations:
- the
thresholded
black-and-white
image
iscanned
s fo edges.
r Apixelisdefined
aan
s
edge
pointif
itisblack
icnolor,
andhasbetween 2and
black
7 neighbors.
Theoutput
of
edge
detection
ianother
s b/w
image:edgesaret raced
ibnlack,
andtheredundant
interior
of
the
featureshasbeen
removed.
- segmentation:edge
pointsare
logically
grouped
int coonnected
threadsor
loops,
and
eachconnected
componentisassumed
troepresenta distinctfeature(object)
in
the
scene.
Theoutputis
caollection
oarrays,
f each
c ontaining
the
edge
coordinatesof
14
one
feature.
We use
two-pass
a algorithmdescribed by
Lumiaeal.
t (Algorithm
3
in
the
reference).For
each
line,
adjacentpointsi tnhatand
the
previouslineare
91
labeled
abelonging
s ttohe
same
component,andthe labeling
equivalencesare
resolved
bmy eans
daof
epth-firstgraph
search.
H ere
isasimple
example:
Atthe
pixelX,
the
equivalence
olabels
f and bw illbe
recognized,
andchangesin
labeling
carried
outin
the
second
(bottomup)
pass The
. procedure
workswellfor
simple
scenes,
butthe
connectednessproblemiscom binatorialin
nature,
andthe
equivalence
search
willeventually
getbogged
down in
compleximages.
- insignificantly
smallfeaturesare
removed,
andan invariantsignature
icalculated,
s
which
iused
s foractualrecognition
othe
f object in
the
scene.
We use
thesecond
momentsaboutthe
principalaxesof
inertia,
which are
invariantunderrotation
and
translation
and
relatively
easy
tobtain.
Thecalc ulation
involvesone
square
rootper
feature,
butrequiresno
adjacency
information
abou the
t feature
points.
Asa
preliminary,
thisstep
also
calculatesthefeature’ center
s of mass.
- targetrecognition:signaturesof
the
objectsin
th secene
arecompared
ttohose
oaf
selected
targetfeature.Ifthe
targetisfound,
th deisplacementvector
isthedifference
between
itscenter
of
massand
the
centerpointof
t he
image.If
thetargetislost,
an
error
isreturned.
92
18.1 Main
datastructuresinthe
MCF5307
code
DigitizedImage:
IMAGE_FRAME starting
addressof
theread
buffer
P2 Ron
DPRAM
FRAME_WIDTH image
dimensionsin
pixels
FRAME_HEIGHT
Motion/DisplacementVectors:
XS_VECT_ADDR DPRAM
addressof
the
vector
component s
X_MVECT,
Y_MVECT localcopy
othe
f motionvector
NORM_MVECT normalization
constantof
the
motion
ve ctor
X_DVECT,
Y_DVECT the
displacementvector
Semaphores:
These
variablescarry
messagesbetween
(andwithin) the
two
main
processeson
the
MCF5307.
They
are
integers,
setto
one
andcleared to
zero.
DISP_VECT_AVAIL feature
recognition
announcesthat naew
displacementvector
isavailable
TARGET_AVAIL main
processannouncesthat
taargetf eature
has
been
selected
for
tracking
Post-Segmentation
Description
oImage
f Features:
Each
individualconnected
edge
istored
s istruc
an ture
othe
f typeFpoint.
Thestructure
containsthe
pointcount,
arraysof
coordinate
valu es,
center
of
massand
the
second
93
moments.
These
structuresareallocated
dynamically asthefeaturesemerge
fromimage
analysis.
compResult sa-tructureothe
f type
ComponentList*,
holdsthe overallresultof
the
image
segmentation.
Itcontainsthe
featureco unt,
andaanrray
opointers
f to
structuresof
typeFpoint,
which
contain
the
detail of
seachfeature.
targetFeature the
- feature
thatisbeing
tracked
istored
s itnh isFpointstructure.
Fpointand
ComponentListdata
typesare
specified
i tnhe
headerfile
FeaturePoints.h
Parametersof
the
ServoDuty
Cycles:
See
the
data
section
othe
f programmodule
Servo.s for
description
and
currentvalues.
94
19. INITIALIZATION
OFTHESBC5307
BOARD
Pre-initialization
conditions:
The
entirecode
(initandfunctional,
ca.45K
in
si ze)
residesin
flash
ROM,
and
the
flash
ROM
isassociated
with
the
globalchip
select.
Vect or
table
and
initialization
code
are
linked
to,
and
reside
at,
zero
address;vector
base register
(VBR)
pointsto
zero.
Functionalcode
ilinked
s ttoheupper
bank
oDRAM
f and
, residesin
theflash,
justabove
the
initcode.
Seethe
linker
scriptsbc5307vis.ld for
the
detailsof
the
linking
procedure.
Initialization
sequence:
on
-resetexception,
initialPCand
SPare
loaded fromROM,
aslongwordsataddresses
zero
and
four.
PCpointsto
start-up
code
iR
n OM.
ColdFire
- processor
isconfigured:cache
idisabl
s ed
and
turned
off;SIMand
upper
bank
of
DRAM
areconfigured;chip
selectsare
configured and
the
systemispulled
outof
the
globalchip
select
contents
- of
the
flash
ROM
arecopied
tuopperDRA M,
VBRissetto
the
base
oupper
f
DRAM
program
- controljumpsto
DRAM
starting
point
chip
- selectzero
ireconfigured
s ttohe
top
oad
f dressspace(to
getthe
flashmemory
out
of
the
way)
lower
- bank
oDRAM
f isconfigured
control
- jumpsto
main
and
startsrunning
function alcode
95
20. CHARACTERISTICSOF
THECAMERA/SERVOSYSTEM
(GIMBA L)
A
technicaldescription
othe
f camera
isnotavail able,
butby
measuring
the
screen
image
of
an
objectof
known
size,
we have
determined
the following:
the
- angle
corresponding to
one-pixel
a displacement(the
cameraratio)
is0 .62
degrees/pixel,
atourgiven
imageresolution,
the
- camera’sfield
ica.
s 60degreeswide
andca. 47
degreeshigh.
The
servo
motorswhichmove
the
camera
have
anangu lar
motion
range
oca.
f 90
degrees.
Their
duty
cyclei20
smilliseconds,
with the
recommended
high
time
ranging
fromt1ms.
2oThese
duty
cyclesaregeneratedby the
back-end
FPGA.
The
camera’splatformisdriven
blever
ay mechanis m(see
Picture
47),
anditsangular
motion
inot
s entirely
linear
relative
ttohe
motio onthe
f servos.
However,within
the
above
high-time
range
the
relationship
ireasonabl
s lyinear,and
the
platformcoversan
angle
oca.
f 46
degreesineachcoordinate
directio n.
In
thecurrentcircuitdesign,
theto
12 msrange
correspondsto
the
range
o2048
f clock
ticksof
high
time,
or
45.3
ticksper
degree
opla
f tformmotion.
Combined
with
the
above
camera
ratio,
thisyields28.3
ticksper
one
pixel of
displacement,
theconstantof
proportionality
betweenmotion/displacementvectors and
the
motor
movement.
96
Picture
4View
7: of
thecameragimbal
97
0x00 0x00
ROM
Low Semaphores
DRAM
0x08 Status
byte
0x09
FPGA
parameters
0x400000
Codeanddata
0x0C
Heap
High
DRAM Motion
vectors
Stack
0x800000
0x20
BufferM0
0x10000000 Config.
0x2710
BufferM1
0xFF800000 Semaphores
BufferM2
0xFFE00000
High
ROM
Memory
mapothe
f ColdFireboard FPGA/DPRAMmemory
map
Picture
48:Memory
maps
98
J11
J8 J9
J6
33.333 MHz
RST
J10
XC4010 FPGA XC4010 FPGA
back end front end
CY7C007AV
J5 J4 J3 DPRAM
J7
AT17LV J2
front
J1
2.4576
AU2904 AT17LV MHz
back SW
LM317T POWER
(3.3V) LM317T ZOOM
(5V) MAX622
5ACPA XC9572 RESET
LM1881
AD876AR NMI
RCA video
jack
Picture
49:Layoutof
theFPGA
board
99
4
+12V
Switch
+12V
+7.5V +7.5V Battery
GND pack
serial
Diagnostic
Power
on/off
logic GND
comm. 2
SBC5307 FPGA
board and
Camera
gimbal
4
5
4 Dutycyclesand
servo
positions
+7.5V
+7.5V Servoboard
Dutycyclesand
servo-move +7.5V +7.5V
Video
signal
fromthe
camera
3
3
3 Radio
3
Picture
50:Airplane Radio TV
system’soverview receiver transmitter
+12V
100
REFERENCES
1
R.W.
Rodieck,
“The
FirstStepsin
Seeing,”
Sinaue r,
1998.
2
David
VandenBout,
“The
PracticalXilinxDesigne Lab
r Book,
Version1.5,”
Prentice-Hall,
1999.
3
“ColdFire
Microprocessor
Family
Programmer’sRefer ence
Manual,”
Motorola,
1997
(MCF5200PRM/AD)
4
Leon
WCouch,
. “Digitaland
Analog
Communication
S ystems,”
third
edition,
MacMillan,
1990,
Chapter 5-9.
5
Data
sheetfor
the
videosync
separatorLM1881,Na tionalSemiconductor,February
1995.
6
Data
sheetfor
theA/Dconverter
AD876,
Analog
Dev ices,
December
1997.
7
Data
sheetfor
the
CY7CnnnAV
Dual-PortStatic
RAM Cypress
, Semiconductor,
December 7,
2000.
8
“Understanding
AsynchronousDual-PortRAMs,”
tech nicalnote,
Cypress
Semiconductor,rev.
November7,
1997.
9
MCF5307
User’sManual,
Motorola,
1998
(MCF5307UM /AD)
10
NicolasFranceschini,
“Early
Processing
oColour
f and
Motion
iMosaic
an Visual
System,”NeuroscienceResearch,
Supplement2,pp.
S 17-S49,
Elsevier,
1985.
11
Jean-Marc
Pichon,
Christian
Blanes,
NicolasFrance schini,
“VisualGuidance
oaf
Mobile
Robot,”
SPIE,
v.1195,
1989.
12
Brian
AWandell,
. “Foundationsof
Vision,”
Sinauer 1995.
,
13
S.
Yuand
T.S.Lee,"WhatDo
VN1 euronsTellUsab outSaccadic
Suppression?,"
Journalof
NeuralComputing,2000.
14
RonaldLumia,Linda Shapiro,
OscarZuniga,
“ANew Connected
Components
AlgorithmforVirtualMemoryComputers,”
Computer Vision,
GraphicsandImage
Processing, 22287-300
, (1983).
101
CURRICULUM
VITAE
DankoAntolovic
Born
i1n955,
inZagreb,Croatia
Education:
• B.S.
Chemistry,
University
oZagreb,
f Croatia(197 8)
• M.S.
Chemistry,
JohnsHopkinsUniversity,Baltimore MD
, (1981)
• Ph.D.
Chemistry,
Johns
HopkinsUniversity,Baltimor e,
MD (1983)
• M.S.
Computer
Science,
Indiana
University,Blooming ton,IN
(2001)
Work
experience:
• 25yearsin
systemsanalysisand
computer
software development
Platforms:
desktop,
servers,
mainframesand
superc omputers
Applications:
Robotics,
Computer
Vision,Industria Programming,
l Solarand
GasCombustion
Modeling,
QuantumChemicalModeling, Molecular
Modeling
• 10yearsin
scientificresearch
• 5yearsin
teaching
athe
t undergraduate
level
• 5yearsin
running
software
a developmentbusiness
102