Nonlinearity optimization in

nonlinear joint transform correlators

Leonid P. Yaroslavsky and Emanuel Marom

Three types of nonlinear transformations of the joint spectrum in nonlinear joint transform correlators
~NLJTCs! are investigated with the purpose of achieving the highest discrimination capability in target
location in a cluttered background: logarithmic transformation and the ~1yk!th law transformation in
combination with the limitation of the signal dynamic range and binarization by thresholding. By
computer simulation carried out on a set of test images, it is shown that application of these transfor-
mations in NLJTCs may considerably improve the correlators capacity to locate and recognize properly
small objects on a cluttered background, provided there is proper selection of nonlinearity parameters.
It is also shown that a moderate blur of the joint spectrum in such NLJTCs before nonlinear transfor-
mation is permissible, which simplifies the requirements of correlator optical alignment, the resolution
power of correlator electronic components, or both. 1997 Optical Society of America
Key words: Optical correlators, matched filters, joint transform correlators.

1. Introduction the nonlinear optical media and electronic compo-

Nonlinear joint transform correlators ~NLJTCs! with
input images joint spectra nonlinearly transformed
by computer or in an electronic amplifier before mod-
ulating the output spatial light modulator ~SLM!
~Fig. 1! have been proposed as a tool for real-time
pattern recognition and target location.17 Among
them, the binary joint transform correlators
~BJTCs!2 4 that use a binary SLM at the Fourier
plane have attracted special attention because of
their improved discrimination capability and high
light efficiency because of the availability of binary
SLMs. However, a number of important issues on
the design and implementation of such correlators
have not been addressed as yet:
What type of nonlinear transformation should
one use in joint transform correlators ~JTCs! to en-
sure the highest discrimination capability?
How sensitive is the correlators discrimination
capability to such design factors as the accuracy of
realization of the nonlinear transformation, the lim-
itation of the dynamic range, the resolution power of
The authors are with the Faculty of Engineering, Tel-Aviv Uni-
versity, Tel-Aviv 69978, Israel.
Received 31 July 1996; revised manuscript received 30 January
1997.
0003-6935y97y204816-07$10.00y0
1997 Optical Society of America
nents used, and the accuracy of optical alignment?
Why do BJTCs perform better than conven-
tional ones and how sensitive is their performance to
the binarization threshold?
In this paper, we address these issues. We show
that NLJTCs with a logarithmic nonlinearity ap-
proximate the optimal adaptive correlator ~OPAC!8
that guarantees the highest discrimination capability
for target location in a cluttered background. We
show also that the logarithmic, ~1yk!th law nonlin-
earity within a limited dynamic range and binary
nonlinearity can be regarded as a version of nonlin-
earity that compresses the signal dynamic range and
that JTCs with such nonlinearities may exhibit sim-
ilar discrimination capabilities, provided an appro-
priate selection of the nonlinearity index k, the
dynamic-range limitation and the binarization
threshold, respectively.
In Section 2, we provide a brief review of the rele-
vant theory. In Section 3, we justify the use of a
logarithmic nonlinearity in JTCs and experimentally
investigate their discrimination capability and sensi-
tivity to the limitation of the nonlinearitys dynamic
range and resolution power. In Section 4, from anal-
ogy between the logarithmic and ~1yk!th law nonlin-
earities for k .. 1, we suggest using, in NLJTCs, the
~1yk!th law nonlinearity, demonstrate that the
NLJTCs may perform slightly better than even the
logarithmic JTCs, and investigate, by computer sim-

4816 APPLIED OPTICS y Vol. 36, No. 20 y 10 July 1997


Fig. 1. Schematic diagram of the JTC with a nonlinear transfor-
mation of the joint spectrum.
ulation, a trade-off between the nonlinearity index k
and the nonlinearitys dynamic-range limitation
threshold. Being motivated by the results for the
NLJTCs with the ~1yk!th law nonlinearity and the
limitation of the nonlinearitys dynamic range, in
Section 5 we experimentally ~by computer simula-
tion! investigate BJTCs and show that, with an ap-
propriately chosen binarization threshold, they may
perform nearly as well as NLJTCs with logarithmic
and ~1yk!th law nonlinearities and that they are
rather tolerant to reasonable deviations of the bina-
rization threshold from its optimal value. In the
conclusion, we summarize the discussion.
2. Review of the Theory
We base our analysis on the theory presented in Ref.
8. JTCs represent an attractive implementation of
optical correlators. From the point of view of the
theory, optical correlators can be regarded as a spe-
cial case of localizationrecognition devices that con-
sist of a linear filter followed by a unit for the
localization of the signal global maximum at the filter
output. It was shown in Ref. 8 that, for the problem
of target location in a cluttered background, the re-
quirement for the highest discrimination capability of
such devices is practically equivalent to the require-
ment that the filter provide the maximal ratio of the
signal value in the point of the output, where the
target object is located, to the standard deviation of
the signal in the background part of the output sig-
nal. Since a number of random factors, such as sen-
sor noise, an unknown reference-object position,
variability of the reference-object orientation, size,
etc., are involved in the localization, this requirement
should be satisfied on average over these factors.
The transfer function of the optimal filter H~f! was
found8 to be
RO*~f!
Hopt~f! 5 , (1)
AVimsysAVxo@uB~f!u2#
where RO*~f! is the complex-conjugate Fourier spec-
trum of the reference ~target! object, uB~f!u2 is the
power spectrum of the background component of the
input image ~for object location! or average power
spectrum of objects to be rejected ~in object recogni-
tion!, AVimsys denotes averaging over the imaging-
system sensor noise, and AVx0 denotes averaging over
the unknown coordinates x0 of the target. The filter
described by Eq. ~1!, if realizable, is adaptive to the
background component of the image and therefore
provides the best possible performance for the given
individual input image. We refer to it as the OPAC.
The design of the OPAC requires that one know the
power spectrum of the background objects averaged
over the involved random factors. These data are
not known before the target is located and have to be
determined from the observed input image.
As a zero-order approximation of the background
power spectrum one may use the squared absolute
value of the entire observed image spectrum uIM~f!u2:
AVimsysAVx0uB~f!u2 < uIM~f!u2. (2)
Approximation ~2! is based on the assumption that
the target object size is much smaller than the size of
the entire image ~area of search!.
For a more accurate estimation of the background
power spectrum, one can use the following two mod-
els for the representation of the image background
component:
b~x! 5 w~x 2 x0!im~x!, (3)
b~x! 5 im~x! 2 ro~x 2 x0!, (4)
where im~x! is the observed image, b~x! is its back-
ground component, ro~x 2 x0! is the reference ~target!
object, and w~x 2 x0! is a window function:
H
0 within the target object
w~x 2 x0! 5 , (5)
1 elsewhere
For the model of Eq. ~3!, one can show that, with
the assumption of a uniform a priori distribution of
the object coordinates x0 over the picture area S,
AVimsysAVx0uB~f!u2 < AVimsys@uIM~f!u2 ^ ~uW~f!u2yS!#, (6)
where the symbol R denotes convolution and uWu2 is
the squared magnitude of the window functions Fou-
rier spectrum.
For the model of Eq. ~4!, one can show that, with
the same assumption of a uniform distribution of the
object coordinates x0 over the picture area S,
AVimsysAVx0uB~f!u2 < AVimsys@uIM~f!u2 1 uRO~f!u2#. (7)
10 July 1997 y Vol. 36, No. 20 y APPLIED OPTICS 4817
 With estimations ~6! and ~7! of the background
power spectrum, the optimal filter can be imple-
mented either as
RO*~f!
Hopt~f! }
AVimsys@uIM~f!u2 ^ uW~f!u2#
or as
RO*~f!
Hopt } . (9)
AVimsys@uIM~f!u2 1 uRO~f!u2#
Filtering schemes similar to those shown in expres-
sions ~8! and ~9! were also discussed, although under
different assumptions, in Refs. 9 12.
3. Logarithmic Joint Transform Correlators
For the filter of expression ~9!, signal filtering in a
suboptimal adaptive correlator is described in the
frequency domain by the formula
IM~f! z RO*~f!
OUT~f! 5 . (10)
AVimsys@uIM~f!u2 1 uRO~f!u2#
Let us now show that, with a logarithmic nonlinear
transformation of the joint spectrum, the NLJTC of
Fig. 1 approximates this filter. With the logarithmic
nonlinearity, the transformed joint power spectrum
OUTNLJTC at the output of this nonlinear device can
be written as
OUT~f!NLJTC 5 log@uIM~f! 1 RO~f!u2#
5 log@uIM~f!u2 1 uRO~f!u2 1 IM~f! z RO*~f!
1 IM*~f! z RO~f!#.
Since the size of the reference object is usually much
smaller than the size of the input image, we can
assume that, for the majority of the spectral compo-
nents,
uIM~f!u 1 uRO~f!u .. uIM~f!u z uRO~f!u.
2 2
With this assumption, OUTNLJTC ~f! is approximately
equal to
OUTNLJTC~f! < log@uIM~f!u2 1 uRO~f!u2#
IM~f!* z RO~f!
1 optical system is usually much higher than that of
uIM~f!u2 1 uRO~f!u2
IM~f! z RO*~f!
1 .
uIM~f!u2 1 uRO~f!u2
When a JTC configuration is utilized, the two last
terms displaying the correlation function are readily
separated. The last term of expression ~13! resem-
bles expression ~10! when it corresponds to the OPAC
with estimation of the background image component
power spectrum by expression ~7! but without aver-
aging AVimsys. Therefore, one can conclude that the
nonlinear joint transform with the logarithmic non-
linearity correlator can be regarded as an approxi-
mation to the OPAC and therefore promises an
4818 APPLIED OPTICS y Vol. 36, No. 20 y 10 July 1997
improved discrimination capability. Note that aver-
aging AVimsys can be implemented by a kind of
smoothing ~for instance, by a linear blur! of the in-
volved signal.
To verify this conclusion, we conducted computer-
(8)
simulation experiments using sixteen 128 3 128
pixel fragments of a 512 3 512 pixel satellite photo-
graph of an urban area ~the same images were used
in Ref. 13! as a set of test images and a small circular
spot with a diameter of approximately 5 pixels as the
target object. The target object was embedded
within each test image in such a way that a sector of
5 3 5 pixels at the center of each image was substi-
tuted for by the target image. Arrangement of the
input images and the target object, which was used as
an input for the JTC, is shown in Fig. 2~a!. Pairwise
arrangement of the test images allowed us to make
experiments with two images of the set in parallel.
To reduce boundary effects we inscribed the input
images and the test image into a uniform background
with a gray level equal to the average gray level of the
images. Experiments with this set of test images
were aimed at investigation of the discrimination ca-
pability of adaptive nonlinear correlators in images
with the same target and different backgrounds. To
verify the results obtained we also performed an ex-
periment with stereoscopic images @Fig. 2~b!#. In
this case, the target object was a 21 3 21 pixel frag-
ment of one image @shown in the box in the lower-
image illustration in Fig. 2~b!# and the test image in
which this fragment was to be located was the second
~upper! image.
When designing nonlinear correlators one should
also take into account that the dynamic range of
(11) SLMs is always limited. This feature can be de-
scribed as a pointwise nonlinear transformation of
the form
H
x x # lim
LDR~ x! 5 . (14)
lim otherwise
(12)
The parameter lim defines the degree of the dynamic-
range limitation.
An important practical issue in the design of the
JTCs of Fig. 1 is the required resolution power of the
TV camera that reads out the joint spectrum. It is
well known that the spacebandwidth product of an
electronic imaging devices such as TV cameras.
This casts a restriction on the spacebandwidth prod-
(13) uct of NLJTCs. On the other hand, the estimation
of the background-image power spectrum required
for the implementation of an OPAC according to ex-
pression ~7! assumes that the input-image power
spectrum is smoothed by convolution with an appro-
priate window function, as was mentioned above.
Experiments reported in Ref. 13 provide evidence
that appropriate spectrum smoothing may provide
substantial improvement of the nonlinear correla-
tors discrimination capabilities. In the setup shown
in Fig. 1, the limited resolution power of the TV cam-
era ~as reported in Ref. 11! causes smoothing of the

Fig. 2. ~a! Example of the input image of the JTC. ~b! Stereo-
scopic images used in the experiments.
joint spectrum. Averaging the joint-spectrum image
power spectrum uIM~f! 1 RO~f!u2 by convolution with
the point-spread function of the TV camera results in
smoothing both the @uIM~f!u2 1 uRO~f!u2# and the
IM~f!RO*~f! terms in Eq. ~11!. Whereas the former
terms smoothing is what one needs for the spectrum
estimation and may increase the correlators discrim-
ination capability, the latter may have a negative
effect. Therefore, one might expect that there exists
an optimal smoothing degree. The investigation
Fig. 3. Frequency response of the blur operator ~blr! used in the
experiments.
into this issue was also included in the computer
simulation.
The computer model implemented the following
signal transformation:
coroutput
5 uIFT(CONV$LDR@log~uFT~corinput!u2!#, h~blr!%)z2.
(15)
Here, coroutput and corinput are the output and in-
put images, respectively, of the correlator, FT and
IFT are direct and inverse discrete Fourier trans-
forms, respectively, that were used as approxima-
tions to the optical Fourier transform, LDR is the
dynamic-range limitation transformation @Eq. ~14!#,
CONV$ . . . , h~blr!% is a ~blur!fold convolution operator
with a point-spread function h. Frequency re-
sponses of the convolution operator for values of the
blur parameter of blr 5 0, 1, . . . , 11 are shown in Fig.
3.
The parameter lim of the dynamic-range limitation
was selected as the ~0.1lim21!th fraction of the signal
maximum with lim 5 1, 2, . . . , 12. Thus, the case in
which lim 5 1 corresponds to no dynamic-range lim-
itation, and the case in which lim 5 12 corresponds to
limitation at the level of ~10211!th of the signal max-
imal value.
Two performance measures of the correlators dis-
crimination capability were computed in each exper-
iment: the ratio of the objects signal maximum
~exclusive of the output-signal dc component! to the
output signals standard deviation over the back-
ground area in the correlation plane ~SNRV!, and the
ratio of the objects signal maximum to the highest
signal maximum over the background area in the
correlation plane ~SNRM!,14 both exclusive of the
output-signal dc component.
The experimental results are presented in Figs. 4
10 July 1997 y Vol. 36, No. 20 y APPLIED OPTICS 4819
Fig. 4. Plots of the average SNRV and SNRM at the output of the
logarithmic JTC versus the limitation threshold.
and 5. The plots in Fig. 4 represent values of the
SNRV and SNRM for the logarithmic JTC as func-
tions of the dynamic-range limitation parameter lim
averaged over the set of test images. They show
that the logarithmic JTC is not very sensitive to the
limitations up to ~1027! of the entire dynamic range of
the joint spectrum.
Similarly averaged plots in Fig. 5 show how the
discrimination capability of the logarithmic JTC de-
pends on the blur of the joint spectrum. One can
observe the optimum degree of blur that tells that,
although the expected gain in the correlators dis-
crimination capability derived from optimal joint-
spectrum smoothing is not high, the discrimination
capability remains high over a rather broad range of
the degree of smoothing. Note that the optimum in
the SNRM is less pronounced than that in the SNRV.
Fig. 5. Plots of the SNRV and SNRM at the output of the loga-
rithmic JTC versus the blur parameter ~blr! in the Fourier plane.
4820 APPLIED OPTICS y Vol. 36, No. 20 y 10 July 1997
Fig. 6. Illustration of the similarity between the logarithmic and
the ~1yk!th law nonlinearities.
In two of 16 test images no optimum was observed,
and the SNRM monotonically, although very slowly,
decreased with the increase of the blur parameter blr.
One can conclude from these data that the require-
ment of the resolution power of the TV camera is not
very critical: The spatial bandwidth of the camera
may be 1.52 times less than that of the optics with-
out noticeable losses in the correlators discrimina-
tion capability.
4. Nonlinear Joint Transform Correlators with the
~1yk!th Law Nonlinearity
The distinctive feature of the logarithmic signal
transform is that it compresses the signals dynamic
range. Similar compression can be also achieved by
the ~1yk!th law nonlinearity
F~ . . . ! 5 ~ . . . !1yk, (16)
where k .. 1. This similarity is illustrated in Fig. 6
where the logarithmic and ~1yk!th law nonlinearities
are plotted together after normalization by a con-
stant. Therefore, one can expect that NLJTCs with
the ~1yk!th law nonlinearity and k .. 1 will perform
nearly as well as does the logarithmic JTC.
The simulation experiments with NLJTCs with
the ~1yk!th law nonlinearity were carried out with
the same computer model as that for the logarithmic
nonlinearity, except the logarithmic transformation
was substituted for by the ~1yk!th law transformation
and spectrum smoothing was not applied:
coroutput 5 uIFT(LDR$@uFT~corinput!u2#2k%)u2. (17)
The corresponding averaged experimental data for
the SNRV are plotted in Fig. 7~a! for the set of test
images of Fig. 2~a! and in Fig. 7~b! for the stereoscopic
images of Fig. 2~b! as functions of the nonlinearity
index k for the dynamic-range limitation parameter

Fig. 7. Average SNRV ~SNRVav! at the output of the JTC with the
~1yk!th law nonlinearity versus the nonlinearity index k for ~a! the
set of test images and ~b! the set of stereoscopic images ~SNRVstereo!.
lim 5 1, . . . , 10. The results for the SNRM ~not
shown! are similar. These results show that
NLJTCs with the ~1yk!th law nonlinearity may
have a considerably improved discrimination capabil-
ity in comparison with that of the JTC without non-
linear transformation of the joint spectrum ~case of
k 5 1!. In terms of the SNRM, the observed aver-
aged gain is exceeded 3 times. Comparison of the
corresponding data for the NLJTCs with the ~1yk!th
law nonlinearity and the logarithmic JTC shows that,
with an appropriate selection of the nonlinearity in-
dex k and the dynamic-range limitation threshold
lim, the former performs slightly better.
As for the logarithmic JTC, the discrimination
capability of the NLJTC with the ~1yk!th law nonlin-
earity is not sensitive to the limitation threshold,
Fig. 8. Plot of the SNRV at the output of the BJTC ~SNRVbin!
versus the fraction of the joint-spectrum energy under the bina-
rization threshold for the set of test images.
provided there is a proper selection of the nonlinear-
ity index of k .. 1.
A trade-off exists between the nonlinearity in-
dex k and the dynamic-range limitation parameter
lim: A higher degree of dynamic-range limitation
requires lower values of k. With this trade-off, the
discrimination capability remains practically the
same.
The discrimination capability of the NLJTC with the
~1yk!th law nonlinearity does not depend noticeably
on k, provided that k exceeds a minimal value deter-
mined by the dynamic-range limitation level.
5. Binary Joint Transform Correlators
Many nonlinear media are binary, that is, practically
they can be in only two states: transparent or
opaque. This feature can be described as hard lim-
iting ~binarization!:
LDRh~x! 5 H0
1
x , lim
otherwise
. (18)
From the experiments with the ~1yk!th law nonlin-
earity, one can conclude that even the simple
dynamic-range limitation alone may substantially
improve the NLJTC discrimination capability, pro-
vided the limitation threshold is properly chosen.
This fact allows us to assume that BJTCs with the
hard limitation according to Eq. ~18! may also have a
sufficiently high discrimination capability. The sim-
ulation results confirmed this conjecture. The sim-
ulation was carried out with the same set of test
images according to the model:
coroutput 5 uIFT(LDRh$@uFT~corinput!u2#%)u2. (19)
The simulation results are plotted in Fig. 8. The
graphs represent the SNRV for all 16 images of the
test set as functions of the fraction of the joint-
10 July 1997 y Vol. 36, No. 20 y APPLIED OPTICS 4821
spectrum energy under the binarization threshold.
They demonstrate the variability of the SNRV for
different images, and at the same time they clearly
show that, for all the test images, there exists an
optimal value of the binarization threshold for which
the correlators discrimination capability reaches lev-
els close to those achievable for the logarithmic and
~1yk!th law NLJTCs. This optimal value corre-
sponds to the binarization threshold in the range of
approximately ~2 3 1024!th to ~9 3 1024!th fraction of
the entire energy of the joint spectrum. This range
is of the same order of magnitude as that of the ratio
7.5 3 1024 of the area occupied by the target object
~5 3 5 pixels! to the area of the input image ~128 3
256 pixels!, which one would expect. One can also
see that the discrimination capability of the BJTC is
relatively tolerant to reasonable deviations of the bi-
narization threshold from its optimal value. Note
that a similar conclusion can be reached from the
results reported in Ref. 4. The achievable SNRV
and SNRM for BJTCs are lower than those in the
optimal NLJTC but still are much higher than those
for the matched filter @lim 5 1, k 5 1 in Fig. 7~a!#.
6. Conclusion
We have shown that different types of nonlinear
transformations of the joint spectrum in JTCs can be
used successfully for substantial improvement of the
correlators discrimination capability, provided an
appropriate choice of the transformation parameters:
logarithmic transformation, the ~1yk!th law transfor-
mation in combination with limitation of the dynamic
range, and thresholding. We also have demon-
strated that a moderate blur of the joint spectrum
before its nonlinear transformation in NLJTCs is
admissible and may even slightly improve their dis-
crimination capability, that all types of NLJTCs dis-
cussed are tolerant to reasonable deviations of
nonlinearity parameters @the limitation of the dy-
namic range for logarithmic NLJTCs, the nonlinear-
ity index k and the limitation of the dynamic range
for NLJTCs with the ~1yk!th law nonlinearity, and
the binarization threshold for BJTCs#. This per-
mits a substantial weakening of the requirements on
the electronic components of NLJTCs.
Direct correspondence to L. Yaroslavsky, Faculty of
Engineering, Department of Intedisciplinary Stud-
4822 APPLIED OPTICS y Vol. 36, No. 20 y 10 July 1997
ies, Tel-Aviv University, Tel-Aviv 69978, Israel.
E-mail address: yaro@eng.tau.ac.il.
L. P. Yarosslavsky is on leave from the Institute of
Information Transmission Problems, Russian Acad-
amy of Sciences, Bolshoy Karetney 19, Moscow
101447, Russia.
References
1. B. Javidi, Nonlinear joint power spectrum based optical cor-
relation, Appl. Opt. 28, 2358 2367 ~1989!.
2. B. Javidi and C.-J. Kuo, Joint transform image correlation
using a binary spatial light modulator at the Fourier plane,
Appl. Opt. 27, 663 665 ~1988!.
3. F. T. S. Yu and T. Nagata, Binary phase-only joint transform
correlator, Microwave Opt. Technol. Lett. 2, 1517 ~1989!.
4. W. B. Hahn, Jr., and D. L. Flannery, Design elements of
binary joint transform correlation and selected optimization
techniques, Opt. Eng. 31, 896 905 ~1992!.
5. P. Refregier, V. Laude, and B. Javidi, Nonlinear joint-
transform correlation: an optimal solution for adaptive image
discrimination and input noise robustness, Opt. Lett. 19, 405
407 ~1994!.
6. H. Inbar, N. Konforti, and E. Marom, Modified joint transform
correlator binarized by error diffusion. I. Spatially constant
noise-dependent range limit, Appl. Opt. 33, 4434 4441
~1994!.
7. H. Inbar and E. Marom, Modified joint transform correlator
bynarized by error diffusion. II. Spatially variant range
limit, Appl. Opt. 33, 4444 4451 ~1994!.
8. L. P. Yaroslavsky, The Theory of Optimal Methods for Local-
ization of Objects in Pictures, Vol. 32 in Progress in Optics
Series ~Elsevier, Amsterdam, 1993!, pp. 147201.
9. B. Javidi and J. Wang, Optimum filter for detection of a target
in nonoverlapping scene noise, Appl. Opt. 33, 4454 4458
~1994!.
10. A. Mahalanobis, B. W. K. Vijaya Kumar, S. Song, S. R. F. Sims,
and J. F. Epperson, Unconstrained correlation filters, Appl.
Opt. 33, 37513759 ~1944!.
11. H. Inbar and E. Marom, A priori and adaptive Wiener filter-
ing with joint transform correlators, Opt. Lett. 20, 1050
~1995!.
12. H. Inbar and E. Marom, New interpretations of Wiener filters
for image recognition, J. Opt. Soc. Am. A 13, 13251330
~1996!.
13. L. P. Yaroslavsky, Optical correlators with ~2k!th law non-
linearity: Optimal and suboptimal solutions, Appl. Opt. 34,
3924 3932 ~1995!.
14. The SNRM is sometimes called the peak-to-clutter ratio ~see,
for instance, Ref. 4!.