AUDITING: A JOURNAL OF PRACTICE & THEORY

Vol. 24, No. 1

May 2005
pp. 7781

Applying Digital Analysis Using

Benfords Law to Detect Fraud:
The Dangers of Type I Errors
Richard Cleary and Jay C. Thibodeau
SUMMARY: At first glance the application of digital analysis using Benfords Law holds
great promise as a fraud detection process. However, a closer look at the underlying
statistical assumptions reveals that auditors seeking to use Benfords Law must be
aware of the costs of the potential Type I errors that can occur during the analysis stage.
For example, statistical considerations indicate that there is a far greater chance of
making a Type I error if the Benfords Law analysis is completed on a digit-by-digit
basis, as compared to the test-by-test basis typically employed by statisticians. In this
paper, we explain the merits of each choice in terms of statistical concepts and practical
audit process considerations.
Keywords: digital analysis; Benfords Law; fraud detection; Type I Errors; and Com-
puter Assisted Auditing Techniques (CAATs).

INTRODUCTION

I
n recent years, auditors have employed more sophisticated statistical techniques, such as digital
analysis using Benfords Law, as part of their fraud detection processes. Benfords Law proposes
a probability distribution for first, second, and other digits of numbers in data sets describing the
sizes of similar phenomena as long as the sizes span multiple orders of magnitude. There is much
empirical evidence suggesting that the frequencies of first and second digits of a data set that
contains credible numbers will indeed correspond to a Benfords Law probability distribution (e.g.,
Nigrini 1996; Nigrini and Mittermaier 1997). Given its potential to identify data points (e.g., transac-
tion amounts) that contain characteristics associated with fraudulent activity, digital analysis using
Benfords Law holds great promise as a fraud detection process (Coderre 1999).
The increased use of sophisticated statistical tools, like digital analysis using Benfords Law, has
been facilitated by the increased use of Computer Assisted Auditing Techniques (CAATs) as part
of the audit process. For example, an auditor seeking to apply the Benford command using ACL
(http://www.acl.com) need only identify the appropriate data field (e.g., invoice amount) within the
appropriate data file (e.g., client accounts receivable file) in order to successfully run the command.
The auditor would then have to consider whether additional audit procedures should be conducted

Richard Cleary and Jay C. Thibodeau are Associate Professors at Bentley College.

The authors gratefully acknowledge the comments from two anonymous reviewers and Bill Messier (the editor). We also
thank Jim Hunton and participants at the 2004 AAA Midyear Auditing Conference, the University of Florida Statistics
Seminar, and the University of Massachusetts Statistics Seminar for their helpful comments and suggestions.
Submitted: December 2003
Accepted: June 2004

77
78 Cleary and Thibodeau

on any field that did not conform to the Benfords Law probability distribution. It is precisely at this
stage of the process where the auditor must be aware of the costs of possible Type I errors, especially
considering the nature of a standard audit software package such as ACL.1
Standard audit software packages that feature digital analysis using Benfords Law may be
based on statistical assumptions that dramatically increase the likelihood of a Type I error. Given
their potential impact on the audit decision process, these assumptions should be more transparent.
For example, in ACL, the Benford command automatically produces an analysis completed on a
digit-by-digit basis rather than on a test-by-test basis. The need for greater transparency is
particularly salient when considering the statistical literature, which suggests that the chance of
making a Type I error is significantly greater when the Benfords analysis is completed on a digit-
by-digit basis. This is very similar to the multiple comparisons problem encountered in traditional
statistical methods such as Analysis of Variance (Ott 1993).
The purpose of this paper is to illuminate the statistical assumptions that are germane to an
auditors decision process when completing a digital analysis using Benfords Law. To do so, we first
provide a brief description of Benfords Law and a review of the literature seeking to apply Benfords
Law in various auditing contexts. We then provide a description of the Benfords Law analysis stage,
including a discussion of the possibilities for Type I errors. In this section, we also describe the
various statistical assumptions and their importance to the audit process. The final section provides
conclusions and implications to practicing auditors.

BACKGROUND LITERATURE
Benford (1938) became convinced that more numbers have small leading digits, like 1 or 2, than
large leading digits. He researched this assertion by studying many lists of data, such as the areas of
rivers, and the atomic weights of the elements. These empirical studies led Benford to propose that in
many real world applications the first digits d follow the probability distribution:
P(d = m) = log10 ((m + )/ m), for any d in the set {1, 2, 3,,9}.
This probability distribution gives P(d = 1) = log 2 = .301, P(d = 2) = log(3/2) = .176, on up to
P(d = 9) = log(10/9) = .046. In addition to Benfords empirical work, there is a sound theoretical
basis for this distribution (Hill 1995).
Benfords law does not apply universally. For instance, when data are all about the same
magnitude, Benfords distribution will not hold. The heights of adults, measured in any units, will not
follow Benfords law because very few adults are two times as tall as any other, and none are ten
times as tall as another. For example, if the units used were inches, the first digits would nearly all be
fives, sixes, and sevens (but the choice of unit is not important). The incomes of individuals,
however, and many other financial characteristics will span many orders of magnitude so that the
largest is perhaps thousands of times as big as the smallest. Importantly, Benfords Law is not likely
to apply to measurements where human intervention has occurred (e.g., rent expense based on a
contractual lease) or where random numbers have been applied (e.g., phone numbers). In general, for
data fields that cover several orders of magnitude, it is reasonable to expect that Benfords Law will
apply.
In auditing, Benfords law has been shown to be applicable in a number of auditing contexts,
including external (Nigrini and Mittermaier 1997; Nigrini 1999a; Tapp and Burg 2001), internal
(Nigrini 1999b), and governmental (Nigrini 1996; Wallace 2002) auditing contexts. To date, the
extant literature has primarily focused on properly identifying and defining the data sets that would
be appropriate candidates for a digital analysis using Benfords Law. There has been very little

1 Among internal auditors, ACL is the current market leader in fraud detection/prevention and continuous monitoring audit
software (McCollum and Salierno 2003). In addition, each of the Big 4 audit firms uses ACL in their fraud practices.

Auditing: A Journal of Practice & Theory, May 2005

Applying Digital Analysis Using Benfords Law to Detect Fraud: The Dangers of Type I Errors 79

discussion however, about the importance of analyzing the statistical output from a properly
conducted digital analysis using Benfords Law. This apparent void in the literature is surprising
because the underlying statistical assumptions made in an application of Benfords law are important
in determining the correct inference. A closer look at the analysis stage is therefore warranted.2

BENFORDS LAW ANALYSIS STAGE

At the analysis stage of a simple first digit analysis using Benfords Law, an auditor would
essentially be testing the following statistical null hypothesis:
H0: First digits in a data set are distributed according to Benfords Law.
The corresponding practical statement is that no fraud has taken place. Rejecting this null
hypothesis would mean that the digits are not behaving as one might expect. There are at least four
possible explanations:
1. The data really do follow Benfords Law, but due to random chance this particular set of
observations does not. This is the classical Type I error.
2. The assumption of many orders of magnitude is not met.
3. There is a reasonable explanation for an overabundance of some particular first digit. For
instance, if a firm has an arrangement to make a daily transfer to a vendor of $450, the first digit
4 might be overrepresented.
4. Some of the entries are fraudulent. This is the practical alternative hypothesis.
As such, when we reject the statistical null, the presence of fraudulent entries is just one of the
possible explanations. From an audit perspective, rejecting H0 means that additional work needs to
be completed in order to determine which explanation is appropriate, given the client context.
Of particular interest in the present case is the test statistic, or set of test statistics, that should be
used to carry out the test of the null hypothesis. Statisticians presented with this case would likely
select the classical Chi-squared (2) Goodness of Fit test (Keller and Warrack 2003). In this case, we
can compute the number of observations that we expect for each first digit within a data set of this
size if the null hypothesis is true, and compare these to the number actually observed, summarizing
the results in the calculated 2 statistic:
2 calc = ((observed expected)2/expected),
where the sum is over each of the admissible digits or digit-combinations. Large values of this
statistic lead us to reject the null since the observed values are then far from what we expected. We
compare this calculated value to tabulated values to find the p-value for the test (i.e., the probability
of a 2 calc value at least this large if the null hypothesis is in fact true). We reject the null if this p-
value is smaller than some designated probability of type I error; usually if p < .05.
Notice that this case illustrates an overall test of the fit of the data to the distribution proposed by
Benfords Law. It corresponds to checking for deviation from Benfords Law with a single test for
the entire data set. Rejecting the null does not say which digits are overrepresented or underrepresented.
An alternative testing approach would be to examine the digits; one at a time. For first digit testing,
this yields a set of nine different tests. For each d in the set {1,2,3,,9} we test:
H0,d: The first digit d appears as often as we would expect according to Benfords Law.
These tests obviously carry far more detailed information than the overall test. However, there is
a potential consequence in that repeating a testing procedure nine separate times greatly increases the
probability of at least one Type I error, when the data actually follow the Benfords Law distribution.

2 The possibility of Type I errors and the resulting audit consequences has been studied previously in a generalized manner
(e.g., Elliott and Rogers 1972; Beck and Solomon 1985).

Auditing: A Journal of Practice & Theory, May 2005

80 Cleary and Thibodeau

In fact, if the test for each individual digit is carried out at significance level of 0.05 (so that the null
is rejected when the p-value is less than 0.05), then the probability of at least one Type I error in a
battery of nine tests is about 1 (.95)9 = .37.3 An auditor using this digit-by-digit approach will
thus see a false alarm when searching for fraud roughly seven times more often than an auditor
using only the overall test. An important (and potentially scary) byproduct of frequent false alarms is
that the output of a digital analysis may lose relevance for practicing auditors if it almost never
uncovers an actual fraudulent entry.

CONCLUSIONS AND IMPLICATIONS

We suggest the following two courses of action for practitioners. First, consider doing both a
test-by-test and a digit-by-digit analysis when using Benfords Law. The Chi-squared goodness of fit
test is standard in many spreadsheet and statistical packages and the output from ACLs Benford
command could easily be adapted and tested in this manner. Second, on large data sets, consider
doing a Benford analysis of both the first digit, and the first two digits, of a particular set of data.
While examining the results of the 90 statistical tests produced in the two-digit analysis means an
even greater chance of discovering a Type I error than in the one-digit test, the resulting detail will
allow a practitioner to more easily determine which frequently occurring numeric amount is the
cause for the deviation from the expected Benford distribution, where the actual proportion exceeds
the expected proportion.
The situation faced by practicing auditors in this case is very similar to what is faced by
statisticians using an Analysis of Variance (ANOVA) to decide if the mean value of some variable is
the same across several different populations. The overall ANOVA null hypothesis, carried out using
an F-statistic, is that all of the means are equal (Ott 1993). Rejecting this null does not give any
indication of which populations have the largest or smallest mean values, or which pairs of popula-
tions have significantly different means. In the ANOVA setting, statisticians have a choice of several
methods for making these important distinctions. Some are conservative, and carefully control the
risk of Type I error. Others more aggressively look for differences and thus increase the chance of
Type I error. Auditors testing a distribution should be aware of the merits of the testing method
employed and choose accordingly.
The importance of this issue is heightened when considering that the Benford command in
certain auditing software packages (i.e., ACL) automatically supplies the digit-by-digit analysis
without even providing an option for an overall chi-squared test. The individual hypotheses H0,d are
tested using a normal distribution approximation (z-test). Auditors using Benfords Law or other
statistical tests to find evidence of fraudulent entries should be aware of the benefits of the test-by-
test and digit-by-digit approaches. Using an overall test-by-test approach makes it relatively
easy to control the probability of Type I error, but the results may not be as informative in the case
where fraud actually has occurred. Practitioners interested in applying a single test should consult a
standard business statistics book for detailed guidance (e.g., Keller and Warrack 2003, 535). Using a
digit-by-digit approach increases the chances of a Type I error, but also increases the chances of
finding actual fraudulent entries. Perhaps the most prudent approach would be to begin the analysis
stage with an overall analysis using a Chi-squared test, and then follow up with a digit-by-digit
analysis only if there is an indication of possible fraud in the overall analysis.

3 This calculation is not exact because it assumes that each of the nine tests is independent of the others. The authors have
completed a simulation study suggesting that the calculation is not very sensitive to this assumption.

Auditing: A Journal of Practice & Theory, May 2005

Applying Digital Analysis Using Benfords Law to Detect Fraud: The Dangers of Type I Errors 81

REFERENCES
Beck, P., and I. Solomon. 1985. Sampling risks and audit consequences under alternative testing approaches.
The Accounting Review 60 (October): 714723.
Benford, F. 1938. The law of anomalous numbers. Proceedings of the American Philosophical Society 78:
551572.
Coderre, D. 1999. Computer-assisted techniques for fraud detection. The CPA Journal 69 (August): 5759.
Elliott, R., and J. Rogers. 1972. Relating statistical sampling to audit objectives. Journal of Accountancy 134
(July): 4655.
Hill, T. 1995. The significant digit phenomenon. American Mathematical Monthly 102 (4): 322327.
Keller, G., and B. Warrack. 2003. Statistics for Management and Economics. Pacific Grove, CA: Brooks/Cole-
Thomson.
McCollum, T., and D. Salierno. 2003. Choosing the right tools. Internal Auditor 60 (August): 3243.
Nigrini, M. 1996. A taxpayer compliance application of Benfords Law. The Journal of the American Tax
Association 18 (Spring): 7291.
, and L. Mittermaier. 1997. The use of Benfords Law as an aid in analytical procedures. Auditing: A
Journal of Practice & Theory 16 (2): 5267.
. 1999a. Ive got your number. Journal of Accountancy 187 (May): 7983.
. 1999b. Adding value with digital analysis. Internal Auditor 56 (February): 2123.
Ott, R. 1993. An Introduction to Statistical Methods and Data Analysis. Belmont, CA: Duxbury Press.
Tapp, D., and D. Burg. 2001. Using technology to detect fraud. Pennsylvania CPA Journal 71 (Winter): 2023.
Wallace, W. 2002. Assessing the quality of data used for benchmarking and decision making. Journal of
Government Financial Management 51 (Fall): 1622.

Auditing: A Journal of Practice & Theory, May 2005