Byte-oriented Efficient Implementation of the NIST Statistical Test Suite

A. Suciu, K. Marton, I. Nagy, I. Pinca

Technical University of Cluj-Napoca, Romania {Alin.Suciu, Kinga.Marton}@cs.utcluj.ro, {Iszabela.Nagy, john_john_2402}@yahoo.com

Abstract Randomness comes in many flavours and has countless applications in various domains which state different quality requirements for the outcome of random number generators, therefore decisions on the suitability of a randomness generator for a specific application has to be made as a result of thorough analysis including the essential part of statistical testing. But as systems tend to consume increasingly larger volumes of random data, having high throughput random number generators is imperative, but not sufficient, because between generators and the application requiring randomness the data can flow with a speed limited by the performance of the statistical testing units interposed. Furthermore statistical tests can assess only certain features of random sequences based on the statistical properties of true random sequences but can not ensure perfect randomness, hence several statistical tests have to be applied in order to increase the confidence in the selected generator. As a result there is a stringent need for high performance test suites for assessing the quality of the generated random sequences. Our work enrols in this direction presenting a performance efficient version of the well known NIST statistical test suite for random and pseudorandom number generators based on a paradigm shift towards byte stream processing mode inside the tests. Experimental results show significant performance improvements of up to 13 times in average compared to the original version. Keywords random number generator; statistical test suite; efficient implementation

I. INTRODUCTION Randomness is an essential input for applications requiring unpredictability, irreproducibility, uniform distribution or other specific properties of random number sequences. While unpredictability is highly desired for the output of random number generators, the quality of produced sequences must not be left a matter of chance, as the quality of randomness determines the suitability of the generator for a specific application. Therefore the selection of a generator has to rely on a thorough analysis of the randomness source and the randomness properties of the output sequences. Statistical testing plays a fundamental role in this analysis measuring certain properties of random number sequences in terms of probability, and based on the likely outcome of the tests applied on perfect random sequences can highlight possible deviations from randomness. Yet no finite set of statistical tests can be considered complete and consequently can not ensure perfect randomness, but in order to increase the

confidence in the selected generator different statistical tests are applied on the generated sequences. As we earlier mentioned, each test focuses on one specific property that a perfectly random sequence exhibits and verifies whether the generated sequence satisfies this property in order to be called random from the perspective of the current test. Depending on the requirements a certain domain (or application) imposes on the random number generator, different randomness tests that assess the desired properties are grouped together, forming batteries of statistical tests, or test suites, for providing a more general view on the suitability of the tested generator for a specific application. There are several well known batteries of statistical tests for random number generators such as the Diehard test suite [5], ENT [4], Test01 [5], but the most popular is the NIST statistical test suite [1] developed by NIST and applied as a standard stage in assessing the outcome of random number generators. Applying several batteries of statistical tests on large sequences of random numbers is a very time consuming operation; therefore, in order to satisfy the increasing demand for large volumes of random data, the development of high throughput random number generators must be combined with high performance statistical test suites. Unfortunately the most popular batteries of tests are not focused on efficient implementation so there is a stringent need for providing highly efficient statistical tests and our efforts on improving the NIST test suite aims to address this need. The paper is structured in 5 sections. Section 2 briefly presents the original NIST test suite describing the included tests. Section 3 points out the main drawbacks in the NIST implementation and introduces the optimization steps followed by experimental results, in section 4, demonstrating the effectiveness of our approach. Section 5 presents final conclusions and further work. II. THE NIST STATISTICAL TEST SUITE The NIST statistical test suite [1] is the most popular battery of statistical tests for assessing the quality of random number generators. Designed to be used for cryptographic applications that impose the most exacting requirements on the quality of random number sequences, the test suite initially consisted of 16 statistical tests implemented sequentially in the C programming language.

Recently, two tests - the Discrete Fourier Transform (Spectral) test and the Lempel-Zip Compression test - were disregarded due to problems identified by NIST. Each test can be applied on input data, usually binary files, containing random number sequences and the results indicate the degree to which properties of the input data deviates from properties expected from perfect random sequences expressed by means of P-values. A. Frequency (Monobit) test - NIST test 1 The first test verifies whether zero and one bits appear in the tested sequence with approximately the same probability. This simple test can reveal the most obvious deviations from randomness hence further tests depend on this result. B. Frequency test within a block - NIST test 2 As a generalization of the first test, the focus is on the frequency of zeros and ones within M-bit blocks (M=1 in case of the first test) revealing whether zeros and ones are uniformly distributed throughout the tested sequence. C. Runs test - NIST test 3 In order to determine whether transitions between zeroes and ones in the sequence appear as often as expected from a random sequence, this test counts the total number of runs of various lengths. A run consists of an uninterrupted sequence of identical bits. D. Longest run of ones in a block test - NIST test 4 The tested sequence is processed in M bit blocks with the aim of revealing whether the length of the longest run of ones in a block is consistent with the length expected from a random sequence. Any irregularity in this length would imply inconsistence in the longest run of zeros as well. E. Binary Matrix Rank Test - NIST test 5 With the aim of identifying linear dependencies among fixed length subsequences, this test computes the rank of disjoint sub-matrices extracted from the original sequence. F. Non-overlapping template matching test - NIST test 7 If a given pattern occurs more often than expected from a random sequence it reduces unpredictability, hence the test searches for occurrences of a given m-bit non-periodic pattern. G. Overlapping template matching test - NIST test 8 Similar to the above test, but extends the search criteria to overlapping patterns. H. Maurers universal statistical test - NIST test 9 A perfect random sequence can not be further compressed; therefore the purpose of this test is to determine if the sequence can be compressed without information loss. I. Linear complexity test - NIST test 11 The aim of the test is to determine the linear complexity of the LFSR (Linear Feedback Shift Register) that could generate the tested sequence. If the complexity is not sufficiently high, the sequence is considered non-random.

J. Serial test - NIST test 12 In order to verify the uniformity of templates the test counts the occurrences of every possible m-bit overlapping patterns in the sequence. A high level of uniformity patterns occur with the same probability indicates that the sequence is close to random. For m=1 the test is equivalent to the frequency test. K. Approximate entropy test - NIST test 13 The purpose of the test is to compare the frequency of overlapping patterns of two consecutive lengths, m and m+1, against the expected frequency in a true random sequence. L. Cumulative sums test - NIST test 14 In a random sequence zeros and ones occur with the same probability, therefore the sum of all adjusted (-1, +1) digits in the sequence has to be close to zero. This test applies the above principle to successively larger subsequences verifying whether each subsequence satisfies the requirement. M. Random excursions test - NIST test 15 Similar to the Cumulative sum test, but here the results are based on the number of visits to a particular state between two zero values in the partial sums of successively larger subsequences. If the results show deviations from values expected from a perfect random sequence then the tested sequence is non-random. N. Random excursions variant test - NIST test 16 The goal of the test is to determine the frequency of partial sums of successively larger subsequences across all partial sums. III. THE OPTIMIZATION STEPS There are several limitations we encountered using the original NIST test suite, especially when applied to large volumes of random number sequences, which caused a significant setback in performance and motivated our efforts to eliminate the drawbacks and provide an important performance boost. In the following we point out the main limitations and give an insight of the optimization steps. A. Paradigm shift towards byte processing mode The original NIST implementation stores and processes the input sequence, after reading it from the input file, as an array of bits, a very time consuming transformation process. On the other hand, our design introduces byte processing mode, where operations on bit level are performed with the help of special precomputed lookup tables. B. Allowing the processing of very large files A direct consequence of the storage mode (array of bits) and the limitation in the variable definition in the original version, sets a very severe limit on the maximum file size that can be processed in one execution, as the total count of bits in the input file can not exceed the maximum value of an int type variable represented on 32 bits, this gives a theoretical maximum size of 256 MB, but as practice has shown us, this value generally reduces to 198 MB in the original NIST implementation.

In contrast, our implementation allows the processing of files of sizes up to 1GB and eliminates the need of introducing the input file size expressed as number of bits (necessary in the original NIST implementation). C. Removing redundant operations The unnecessary redundancy in bit count operations was removed and each test was adapted to process the input files, according to the paradigm shift, in byte mode. IV. EXPERIMENTAL RESULTS In order to benchmark the enhanced version of the NIST test suite we turned to comparatively test both the original NIST STS version 1.8, and the improved version with input files of different sizes, measuring the processing time without taking into account the time needed for reading the input files. The benchmark was performed on a system with Intel Xeon E5405 processor at 2 GHz and 4 GB of RAM, running Windows 2008 Server, the 64 bit edition. Each test was run 5 times for each of the 10 various size input files in order to obtain an average execution time, and thus a more accurate result. Both implementations are strictly sequential and the performance boost is due to following the optimization steps described in section 3 combined with efficient implementation. The comparative results for each of the improved tests are presented in the following, where our improved version is called SeqNIST. A. Frequency (Monobit) test In the original version of the Monobit test the proportion of zeroes and ones for the entire sequence is determined by computing the sum of all adjusted (-1, +1) digits in the sequence. Based on the fact that the total count of ones is complementary to the total count of zeros in the sequence, our design computes just the number of ones by using lookup tables for byte processing mode. Fig. 1 shows the comparative results of the Frequency (Monobit) test. The improvement ratio is similar to the results produced by the Frequency test within a block. Input files are selected with sizes up to 198 MB as the original version ceases to work on larger files.

B. Frequency test within a block In case of the frequency test within a block, alongside the introduction of byte processing mode for determining the proportion of zeros and ones inside M-bit blocks (M multiple of 8 for byte processing mode), similar to the global frequency test (above), the improved version also allows the use of larger block sizes, instead of the restricted block size of the original version caused by a limitation in the variable definition. C. Runs test The method applied in order to determine the number of runs in the input sequence is to count the passes from zero to one and vice versa. The original version performs this task by comparing every adjacent two bits, whereas the enhanced version uses byte processing mode resulting in performance improvement shown in Fig 2. Tests were run on input files of up to 43 MB in size as the original test ceases to work on larger files.

Figure 2. Comparison of execution time for the Runs test

D. Longest run of ones in a block test The execution time for the longest run of ones within M-bit blocks improved test has known a significant increase in performance compared with the original version. Fig. 3 shows the results for 8-bit blocks. Input files are selected with sizes up to 199 MB as the original version ceases to work on larger files.

Figure 3. Comparison of execution time for the Test for the longest run of ones in a block

Figure 1. Comparison of execution time for the Frequency (Monobit) test

E. Binary Matrix Rank Test The test proceeds in checking for linear dependences among fixed length substrings by forming 32x32 bit matrices from the input sequence and then computes the rank of each matrix.

Instead of the traditional matrix operations approach, our design represents a matrix by a simple array of 32 unsigned long values, each of them representing a row in the traditional matrix and computes the rank of this matrix using logic operations on bits. Fig. 4 presents the comparative results. The maximal input file size allowed by the original version is 197 MB.

template matching test respectively. The original version ceases to work on files larger than 198 MB.

Figure 6. Comparison of execution time for the Overlapping template matching test

Figure 4. Comparison of execution time for the Binary matrix rank test

F. Non-overlapping template matching test and overlapping template matching test For determining the number of pre-defined m-bit target substrings in the sequence the original version uses an m-bit window, and in case of non-overlapping matching when the pattern is found the window slides m bits otherwise only one bit, on the other hand, when overlapping templates are considered, the window slides only one bit regardless of the result of matching.

G. Maurers universal statistical test A sequence that can be significantly compressed contains repeating patterns, and in order to determine the number of bits between matching patterns the test breaks the sequence into an initialization segment and a test segment consisting of several L-bit non-overlapping blocks and computes a table that has as indices all the possible values of L bits length, and as values the last positions on which that values where found in the input sequence. In the NIST implementation the base 10 numerical value of each block of length L is computed by performing multiplication and addition for each bit. In our implementation L was chosen to be 8 and the transformation to the base 10 numerical value is already done, by storing each byte of the sequence in an unsigned char variable. Fig.7 presents the comparative results on input files for which the NIST implementation allows L to be 8.

Figure 5. Comparison of execution time for the Non-overlapping template matching test

The improved version uses an 8-bit window where the template is compared with the bytes of the input sequence that successively form a value of short int type which in nonoverlapping template matching gets shifted left by 1 bit if the pattern is found and by 8 bits otherwise. On the other hand, overlapping template matching test shifts this value always by just one bit. Each time enough space is available a new byte is added to the variable. Fig. 5 and Fig. 6 show the comparative results for non-overlapping template matching test and overlapping

Figure 7. Comparison of execution time for the Maurers universal statistical test

H. Linear complexity test By adapting the Berlekamp-Massey algorithm, which is applied in order to compute the linear complexity of an LFSR that could have produced the tested sequence, to byte processing mode introduced an improvement level shown in Fig. 8.

J.

Cumulative sums test The test computes the partial sums of successively larger subsequences of adjusted (-1, +1) digits in the sequence, and then chooses the largest absolute value from them (z). The test is applied on the sequence for both forward and reverse direction, the improved version computes both values in a single traversal. The maximal input file size allowed by the original version is 198 MB (see Fig. 11).

Figure 8.

Comparison of execution time for the Linear complexity test

I. Serial test and Approximate Entropy test Determining the frequency of each and every overlapping mbit pattern across the entire sequence implies the transformation of every m-bit block into its corresponding decimal value. The paradigm shift towards byte processing results in a significant performance boost as shown in Fig. 9 where m=9 is used. The maximal input file size allowed by the original version is 198 MB. Similar optimizations are used in the Approximate entropy test, where the frequency of overlapping patterns of two consecutive lengths, m and m+1, has to be determined. Fig. 10 presents comparative results for m=8. The original version ceases to work on files larger than 200 MB.

Figure 11. Comparison of execution time for the Cumulative sums test

K. Random excursions variant test In order to determine the frequency of partial sums of successively larger subsequences across all partial sums the original version first traverses the input sequence to compute the number of cycles and the partial sums in the sequence and then it traverses the partial sums for computing several statistical counts. The improved version performs the above operations in a byte oriented manner and in a single traversal of the input sequence. Fig. 12 presents the comparative results for input file size limited by the original version to up to 43 MB.

Figure 9. Comparison of execution time for the Serial test

Figure 12. Comparison of execution time for the Random excursions variant test

Figure 10. Comparison of execution time for the Approximate entropy test

V. CONCLUSIONS AND FURTHER WORK The purpose of this paper is to present the improvements applied to the NIST statistical test suite for assessing the quality of random number generators and to emphasize the efficiency of the improved version displaying the benchmark results that demonstrate significant performance improvements.

TABLE I PERFORMANCE IMPROVEMENTS Test Test name No. 1 Frequency (Monobit) test 2 Frequency test within a block 3 Runs test 4 Longest run of ones in a block test 5 Binary Matrix Rank Test 7 Non-overlapping template matching test 8 Overlapping template matching test 9 Maurers universal statistical test 11 Linear complexity test 12 Serial test 13 Approximate entropy test 14 Cumulative sums test 15 Random excursions test 16 Random excursions variant test Average speedup Speedup 9.82 9.63 5.84 6.51 7.92 3.13 15.15 12.80 3.92 48.73 54.16 3.31 1.26 6.09 13.45

In order to take the improvements to an even higher level we are currently working on introducing parallelism and making the tests more suitable for execution on todays multicore systems. ACKNOWLEDGMENT This work was supported by the CNMP funded CryptoRand project, nr. 11-020/2007 and PRODOC POSDRU/6/1.5/S/5 ID 7676. REFERENCES
[1] A. Rukhin, J. Soto, J. Nechvatal, M. Smid, E. Barker, S. Leigh, M. Levenson, M. Vangel, D. Banks, A. Heckert, J. Dray, S. Vo, A Statistical Test Suite for Random and Pseudorandom Number Generators for Cryptographic Applications, NIST Special Publication SP80022rev1, revised August 2008, Available at: http://csrc.nist.gov/publications/nistpubs/800-22-rev1/SP800-22rev1.pdf. G. Marsaglia, DIEHARD: a battery of tests of randomness, http://www.stat.fsu.edu/pub/diehard/. P. LEcuyer, and R. Simard, TestU01: A C library for empirical testing of random number generators, ACM Trans. Math. Softw. (TOMS) Vol. 33, No. 4, Article 22, 2007. J. Walker, ENT -A pseudorandom number sequence test program, Fourmilab. Available at: http://www.fourmilab.ch. J.-S. Coron and D. Naccache, An Accurate Evaluation of Maurer's Universal Test, Selected Areas in Cryptography - SAC '98, Lecture Notes in Computer Science, Vol. 1556, Springer-Verlag, pp. 57-71, 1999.

[2] [3]

After introducing the context in which randomness testing play a very important role we stressed out the need for high efficiency test suites and presented the optimization steps applied to the NIST statistical test suite resulting in a higher performance sequential version which was further subject to comparative testing. The results show that the execution time was reduces up to approximately fourteen times on the average for the described system.

[4] [5]