ARTICLE IN PRESS

Biosystems Engineering (2003) 85 (4), 449–454 Available online at www.sciencedirect.com

doi:10.1016/S1537-5110(03)00098-9
PA}Precision Agriculture

Fall Armyworm Damaged Maize Plant Identification using Digital Images

D.G. Sena Jr1; F.A.C. Pinto1; D.M. Queiroz1; P.A. Viana2
1
Department of Agricultural Engineering, Universidade Federal de Vi-cosa, Vi-cosa, MG 36571-000, Brazil; e-mail of corresponding author:
facpinto@ufv.br
2
Embrapa Milho e Sorgo, Sete Lagoas, MG 35701-970, Brazil; e-mail: paviana@cnpms.embrapa.br

(Received 15 August 2002; accepted in revised form 15 May 2003; published online 21 June 2003)

The objectives of precision agriculture are profit maximisation, agricultural input rationalisation and
environmental damage reduction, by adjusting the agricultural practices to the site demands. The fall
armyworm (Spodoptera frugiperda) is one of the most important maize pests in Brazil and the use of insecticide
is the main control method. It is believed that site-specific control can be implemented by using a machine
vision system. The objective of this work was to develop and evaluate an algorithm at simplified lighting
conditions for identifying damaged maize plants by the fall armyworm using digital colour images. Images of
damaged and non-damaged maize plants were taken in eight different stages and in three different light
intensities. The proposed algorithm had two stages: the processing and the image analysis. During the first
stage, the images were processed to create binary images where the leaves were segmented from the other
pixels. At the second stage, the images were subdivided into blocks and classified as ‘damaged’ or ‘non-
damaged’ depending on the number of objects found in each block. The algorithm correctly classified 94
72%
of 720 images.
# 2003 Silsoe Research Institute. All rights reserved
Published by Elsevier Science Ltd

1. Introduction of damaged plants, visually identified by experts.

Visual diagnoses are widely used in agriculture fields.
Thus, machine vision systems have been proposed for
site-specific management of weed (Steward & Tian;
1998), plant nutrition (Borhan & Panigrahi, 1999), plant
disease (V!ızh!anyo! & Felfoldi, 2000) and water stress
(Kacira et al., 1999). In agriculture, the challenges for
developing a machine vision system are greater than in
the industry. The main problems are the great variation
in light intensity and the shadows on the images
(Steward & Tian, 1998), the variability in shape,
dimensions and colour of the agricultural product and
the work environment for the equipments (Pinto et al.,
2001).
The fall armyworm (Spodoptera frugiperda) is one of
the most significant maize pests in Brazil (Cruz, 1995;
Fancelli & Dourado Neto, 2000). The main control
system is the use of insecticides that frequently damage
the natural enemies population (Cruz et al., 2000). The
techniques of integrated pest management (IPM)
recommend the use of chemicals based on the number
Feeding in the whorl of maize, the fall armyworm
produces characteristic holes in the leaves, whereas
other insect damage usually appears at the edges of the
leaves since it is eaten from the edge to the leaf centre.
One of the biggest concerns is pest resistance to the
chemicals. This results in a greater number of applica-
tions being required; increased product dosage; or the
use of an alternative product that is often more toxic
and expensive. Consequently, this is likely to increase
costs and the environmentally damaging effects (e.g.
causing a reduction in natural predators). Omoto et al.
(2000) accomplished a study about the susceptibility of
S. frugiperda populations to the most used insecticides in
Brazil and the results showed that it is necessary to
implement a resistance management programme for this
pest.
The maintenance of refuges within the crop fields,
without chemical application, has been suggested as a
way of responsibly managing the use of toxic chemicals
to preserve an environmental balance. The site-specific
IPM could theoretically create these refuges (Midgarden

1537-5110/03/$30.00 449 # 2003 Silsoe Research Institute. All rights reserved

Published by Elsevier Science Ltd
 ARTICLE IN PRESS
450 D.G. SENA JR. ET AL.
et al., 1997) where it would be possible to preserve the
susceptible genotypes and natural predators whilst
reducing the use of insecticides (Weisz et al., 1996).
Precision agriculture can also help the rational use of
chemicals by applying them at the right time and
amount on pre-selected targets (Concei-ca* o, 2000) since
the pests usually show a grouped spatial distribution
(Viana, 2000). Weisz et al. (1996) showed this potential
on the site-specific management of potato pests, redu-
cing the use of insecticides between 40 and 70%.
Midgarden et al. (1997) showed the advantage of site-
specific management over traditional IPM for maintain-
ing low resistance levels of Colorado potato beetle
(Leptinotarsa decemlineata) and greater natural enemy
populations in commercial potato fields.
It is believed that the fall armyworm shows char-
acteristics that could allow the site-specific control with
a machine vision system; the most useful is that at the
larval stage there is no movement from one plant to
another. Although the pest damage is typical, some
difficulties are expected, such as early detection due to
the reduced size of the holes and possible confusion with
image segmentation errors. The identification of the
damaged maize plants could generate data for applica-
tion maps or real-time spraying. Both methods show
potential for reducing the use of insecticides, environ-
mental damage, mortality of natural predators, and the
development of pest resistance.
The main focus of this work was on vision develop-
ment. Thus, the objective of this work was to develop
and evaluate an algorithm at simplified light conditions,
to enable maize damaged plants by the fall armyworm
to be identified using digital colour images. The
algorithm was developed in modules that can be
improved independently to adapt for different field
conditions.
2. Material and methods
The images used for this work were of 30 plants
cultivated on pots in a greenhouse at Embrapa Corn &
Sorghum, Sete Lagoas, MG, Brazil. Three HT971011
hybrid seeds were sown into each of 30 pots to increase
chances of at least one germinating. Then once seedlings
emerged, these were twined down, leaving one plant per
pot. Fifteen days after emergence, 15 plants were
randomly chosen and infested with ten fall armyworms
(S. frugiperda) hatched 1day before and 15 left healthy.
Images of the damaged and healthy plants were taken
under three light intensities and at eight different dates
after infestation, ending in a final number of 720 images.
The images, 1392 pixels horizontally by 1039 pixels
vertically with 24-bit depth, were captured using a red,
Table 1
Each charged couple device (CCD) gain and time of exposure in
the red, near-infrared and green-blue camera channels
Channel Gain, dB Exposure, ms
Red 15
0 123
000
Near infrared 12
9 122
625
Green-blue 10
2 122
625
*
The NIR channel was not used in this work.
green, blue (RGB) digital camera, MS3100 (Duncan
Technologies, Inc., CA, USA), connected to a frame
grabber, PCI 1424 (National Instruments Corp, TX,
USA). A 17 mm focal length lens was used with a fixed
f-stop of 3
5. The camera was set at 1
0 m from the
highest plant leaves. The spatial resolution was,
approximately 0
3 mm pixel1. The gain and time
exposure of each charged couple device (CCD) is shown
in Table 1.
A metal structure covered with a white canvas was
built to control the light conditions. The illumination
system comprised ten halogen lights: four of 150 W and
six of 50 W. The lights were set to a diffused illumina-
tion. Three levels of illumination were produced by
keeping the four 150 W bulbs on and adding two, four
or six 50 W bulbs, respectively. Although the intensity of
the lighting system was not measured, the halogen light
was used because of its spectral distribution all over the
visible and near-infrared range.
The images were processed using Image Processing
MATLAB1 (MathWorks Inc., MA, USA) toolbox.
Since the images presented pixels with null values on
their edges, they were resized to 1372 pixels horizontally
by 1019 pixels vertically. The algorithm was developed
in two steps: the processing and the image analysis. The
objective of the image processing was to obtain a binary
image where the plant leaves were segmented from the
rest of the image (soil, pot edges and damage done by
the fall armyworm). The objective of the image analysis
was to classify the images as damaged or non-damaged
plants.
First, the original RGB images were transformed to
256 grey level images by applying the excess green index
and rescaling the pixel values. Then, the monochrome
images were thresholded by the iterative method (Yang
et al., 2001). This method assumes that the optimal
threshold is the average of the mean values for the
foreground and background pixels that is searched
iteratively on the image histogram. The pixels represent-
ing leaves received a value of zero and the others a value
of one. Finally, a 3 by 3 median filter was applied to
reduce the binary image noises due to segmentation
errors. Thus, it was assumed that group of four or less
connected pixels was noise (Fig. 1).
 ARTICLE IN PRESS
MAIZE PLANT IDENTIFICATION
Original image
Excess green index
Grey level image
Iterative threshold
method
Binary image
Median filter 3 by 3
Binary filtered image
Fig. 1. Steps of the image processing algorithm
The original RGB images were transformed into
index monochromatic images using the normalised
excess green index:
2G  R  B
EG ¼
RþGþB
where: EG is the normalised excess green index; G is the
pixel value on the green band; R is the pixel value on the
red band; and B is the pixel value on the blue band.
Since the EG value ranged from –1 to 2, the pixel
values of index images were re-scaled to 256 grey levels
by
p0  pmin
p¼ 255
pmax  pmin
where: p is the pixel value of enhanced image; p0 is the
pixel value of indexed image; pmax is the maximum pixel
value of indexed image; and pmin is the minimum pixel
value of indexed image.
Figure 2 shows the steps of image analysis. The binary
images were subdivided automatically into 12 blocks of
300 pixels by 300 pixels each. These blocks were located
in the centre region of the image with 1200 pixels
horizontally by 900 pixels vertically. Therefore, this
region presented three rows of blocks with four blocks
on each row. The blocks that presented more than 5%
of leaves were selected. A search for connected groups of
pixels with value one, here called objects, was done in
the selected blocks. Then, the number of the objects on
each selected block were counted. Finally, the selected
blocks were classified as a damaged or non-damaged
leaf based on its total number of objects. The image was
assumed to represent a damaged plant when it had at
least one block classified as damaged.
451
Binary image
Image subdivision into
12 blocks
Subdivided image
Blocks with more than
5% of leaf selection
Selected blocks
Object identification
and counting
Classification as
damaged or non-
damaged block
Fig. 2. Image analysis steps for the proposed algorithm
ð1Þ
It was assumed that some segmentation errors always
exist. Therefore, the threshold for the number of objects
necessary to classify the block is crucial to the algorithm
performance. Four threshold values were pre-selected by
using a training set with 90 images. The training set
consisted of images from three groups of ten plants (five
damaged and five undamaged) taken at three dates, each
ð2Þ
group using a different light intensity. Images were
recorded at 17, 20 and 23 days after seedling emergence,
corresponding to 2, 5 and 8 days after infestation by the
fall armyworm.
The algorithm accuracy was assessed by classifying
the whole set of 720 images for each threshold value for
the object number.
3. Results and discussion
Figure 3 shows a sample of the image processing step
and the selected blocks for image analysis. The excess
green image was efficient to enhance the healthy leaves
[Fig. 3(b)], allowing its segmentation from the back-
ground [Fig. 3(c)].
The four pre-selected threshold values for object
numbers were 8, 9, 10 and 11 objects in the block.
Table 2 shows the classification errors for each group
of damaged and non-damaged plants. The higher
 ARTICLE IN PRESS
452 D.G. SENA JR. ET AL.

(a) (b)

(c) (d)

Fig. 3. (a) Original image (changed to its greyscale representation for typographical reasons); (b) image processed with the
normalised excess green index; (c) binary image created with the iterative method; and (d) the selected blocks

Table 2
Classification errors using 8, 9, 10 and 11 objects as a threshold to classify the plants as damaged or non-damaged from a total
sample of 720 images
Damage threshold Misclassification of damaged plants Misclassification of undamaged plants Total plant misclassification
No. % No. % No. %
8 7 1
94 73 20
28 80 11
11
9 9 2
50 46 12
78 55 7
64
10 12 3
33 26 7
22 38 5
28
11 37 10
28 17 4
72 54 7
50

classification errors occurred for the non-damaged plants
with the values of 8, 9 and 10 objects. Applying chemicals
to a healthy plant is a less critical error than an application
being missed. The lower is the threshold value, the lower is
the classification error in the damaged plants and the
greater the classification error in the non-damaged plants.
The best algorithm for overall performance was presented
when using ten objects as threshold value.
Table 3 shows the number of plants incorrectly
classified using ten objects as the threshold and the per
cent of error for damaged, non-damaged and total
number of plants. The algorithm classified damaged
plants as non-damaged only at the first two growth
stages, when the damagesites were few and small. The
greatest errors occurred in the first stage (17 days after
seedling emergence and 2 days after infestation) with
2
2% error. On the other hand, on the group of non-
damaged plants, the greatest errors occurred in the last
stages, when the leaf veins were developed and presented
a bright colour. For this group, the greatest error
occurred at 24 days after seedling emergence with an
error of 3
89%.
 ARTICLE IN PRESS
MAIZE PLANT IDENTIFICATION 453

Table 3
Number of plants incorrectly classified in a total sample of 720 images, per cent of error for damaged, not damaged and total number
of plants using ten objects as threshold, for different stages after seedling emergence and after insect infestation
Stage Misclassification of damaged Misclassification of undamaged Total mean
plants plants error, %
Time after seed- Time after insect No. % No. %
ling emergence, infestation, day
day
17 2 8 17
78 2 4
44 11
11
18 3 4 8
89 2 4
44 6
67
19 4 0 0 1 2
22 1
11
20 5 0 0 1 2
22 1
11
21 6 0 0 0 0 0
22 7 0 0 2 4
44 2
22
23 8 0 0 4 8
89 4
44
24 9 0 0 14 31
11 15
56
Total 12 3.33 26 7
22 5
28

(a) (b) (c) (d)

Fig. 4. Incorrectly classified blocks (a) and (b) of fall armyworm damaged leaves; and (c) and (d) undamaged leaves

(a) (b) (c) (d)

Fig. 5. Correctly classified blocks (a) and (b) of fall armyworm damaged leaves and (c) and (d) undamaged leaves

Figures 4 and 5 show blocks of damaged and non-
damaged plants incorrectly and correctly classified,
respectively. The image block of Fig. 4(a) shows an
atypical injury, at the edge of the leaf, which is mixed
up with the background. Figure 4(b) shows a plant
presenting too early damage with few visible injuries.
The leaf veins show in Fig. 4(c) were not segmented as
one object, causing the classification error. The centre
of the plant was segmented as more than one object in
the image block of Fig. 4(d), causing the classification
error.
4. Conclusions
An algorithm for identifying damaged maize plants
by the fall armyworm (Spodoptera frugiperda) using
digital colour images was developed and tested. The
 ARTICLE IN PRESS
454 D.G. SENA JR. ET AL.
algorithm was tested using damaged and non-damaged
maize plant images that were taken under different
lighting conditions and stages of maturity. The pro-
posed algorithm performed well, correctly classifying
94
72% of the images.
Acknowledgements
This research was sponsored by the World Bank and
Embrapa through the project PRODETAB 030-01/99.
The authors have been sponsored by the Brazilian
Agencies CNPq and FAPEMIG. All the mentioned
supports are gratefully acknowledged.
References
Borhan M S; Panigrahi S (1999). Multi-spectral imaging
techniques for nitrogen determination in potato leaf. ASAE
Paper No. 99 5005
Concei-ca* o M Z (2000). Manejo integrado em defesa vegetal.
[Integrated management plant protection] In: Manejo
integrado } doen-cas, pragas e plantas daninhas (Zambolim
L, ed), pp 1–80, DFP/UFV, Vi-cosa
Cruz I (1995). A lagarta do cartucho na cultura do milho.
[The fall armyworm on maize production]. Sete Lagoas:
EMBRAPA-CNPMS, (EMBRAPA-CNPMS. Circular
T!ecnica, 21)
Cruz I; Figueiredo M L C; Alberton M (2000). Efici#encia de
novos piretro! ides no controle da Spodoptera frugiperda em
milho e seletividade ao predador Doru luteipes [New
pyrethroid efficiency for Spodoptera frugiperda controlling
in maize and Doru luteipes selectivity]. In: XXIII congresso
nacional de milho e sorgo, resumos expandidos, Uberl#andia,
Associa-ca* o Brasileira de Milho e Sorgo.
Fancelli A L; Dourado Neto D (2000). Produ-ca* o de milho.
[Maize Production]. Livraria e Editora Agropecu!aria
Gua!ıba
Kacira M; Ling P P; Short T H (1999). Non-contact sensing of
plant water stress by ir thermometry and image processing.
ASAE Paper No. 99 5004
Midgarden D; Fleischer S J; Weisz R; Smilowitz Z (1997). Site-
specific integrated pest management impact on development
of esfenvalerate resistance in Colorado potato beetle
(Coleoptera:Chrysomelidae) and on densities of natural
enemies. Journal of Economic Entomology, 4(90), 855–867
Omoto C; Schmidt F B; Diez-Rodriguez G I; Silva R B (2000).
Resist#encia de Spodoptera frugiperda a inseticidas no Brasil:
situa-ca* o atual e perspectivas. [Spodoptera frugiperda chemi-
cal resistance in Brazil: actual situation and prospection]. In:
XXIII congresso nacional de milho e sorgo, resumos
expandidos, Uberl#andia, Associa-ca* o Brasileira de Milho e
Sorgo
Pinto F A C; Sena Jr D G; Queiroz D M; Gomide R L
(2001).Vis*ao artificial na agricultura de precis*ao. [Machine
Vision on precision agriculture]. In: Avan-cos na agricultura
de precis*ao no Brasil no per!ıodo 1999–2001 (Balastreire L A,
ed). ESALQ/USF, Piracicaba
Steward B L; Tian L F (1998). Real-time machine vision weed-
sensing. ASAE Paper No. 98–3033
Viana P A (2000). Manejo de pragas na cultura do milho. [Pest
management on maize crop]. In: XXIII congresso nacional
de milho e sorgo, resumos expandidos, Uberl#andia,
Associa-ca* o Brasileira de Milho e Sorgo
V!ızh!any!o T; Felfoldi J (2000). Enhancing colour differences in
images of diseased mushrooms. Computers and Electronics
in Agriculture, 26, 187–198
Weisz R; Fleischer S; Smilowitz Z (1996). Site-specific
integrated pest management for high-value crops: impact
on potato pest management. Journal of Economic Ento-
mology, 2(89), 501–509
Yang X; Beyenal H; Gary H; Lewandowski Z (2001).
Evaluation of biofilm image thresholding methods. Water
Research, 35(5), 1149–1158