DatacompressionusingZopfli

JyrkiAlakuijala,Ph.D.andLodeVandevenne,M.Sc. GoogleInc. Abstract We measure theperformance of the Zopfli compression algorithm and compare it with other implementations of deflate compression. We show that Zopfli has the highest compression density of all deflate compatible algorithms we compared, on four compression corpora we used for testing. Zopfli uses significantly more time in compression,butdecompressionspeedof Zopfligeneratedoutputiscomparablewithotheralgorithms.

Introduction
Zopfli is a new deflate compatible compressor that wasinspired bycompressionimprovements developed originally for the lossless mode of WebP image compression. Beingcompatiblewith deflate makes Zopfli compatible with zlib and gzip. Most internet browsers support deflate decompression, and it hasawiderangeofotherapplications.ThismeansthatZopflicompatible decompressionisreadilywidelyavailable. In this study we compare the compression density of the Zopfli compressor with the compression densityofzlib[1],themostcommondeflateimplementationusedtoday,aswellas two lesser known deflate implementations, 7zip [2]andkzip[3].Weusefourdatacompression corpora [47] to measure the compression density. We also measured the compression and decompression speeds forone testcorpus.Intheconclusionwesuggestapotentiallyimportant usecaseforthenewZopflidatacompressionalgorithm. Data compression works by eliminating statistical redundancy from the data. The redundancy can be, for example, in the form of some symbols occurring more often or sequencesofsymbolsrepeating. There are many benefits to higher compression density. The smaller compressed size allows for storing more items in lessspace, faster data transmission, and lowerwebpageload latencies. Furthermore, thesmallercompressedsizehasadditionalbenefitsinmobileuse,such as lower data transfer fees and reduced battery use. The higher data density is achieved by

using more exhaustive compression techniques, which make the compression alotslower,but thedecompressionspeedisnotaffected.

Methods
Wechoseseveralsetsoffiles(corpora)forrunningthecompressors: a webcentric benchmark by downloading the homepages of the 10000 most popular websites as given in the Alexa Internet directory [4]. 9148 pages were successfully loadedtoformourcorpus. The Calgary Corpus is a collectionofsmalltextandbinarydatafiles,commonlyusedfor comparingdatacompressionalgorithms.[5] Canterbury Corpus, a compression corpus designed for lossless data compression. It wassuggestedasareplacementfortheolderCalgaryCorpus.[6] enwik8 [7] has been developed as a large text compression benchmark, consisting of 100millionbytesofEnglishWikipedia. For running the benchmarks, we used an Ubuntuderivative Linux operatingsystemwith kernel 3.2.5 (x86_64) on Dell Precision T3500 Intel Xeon CPU X5650running at 2.67 GHz. The versions of the various software used in experiments are Zopfli (https://code.google.com/p/zopfli/source/browse/ revision acc035299f8d), gzip1.4,7Zip(A)[64] 9.20,andkzip(release20091108).Thecompilerweusedisgccversion4.6.3.WerunZopfliand kzipwithdefaultarguments,gzipwith9,and7zipwithmm=Deflatemx=9.

Results
Compression results (Table 1) indicate that Zopfli produces the most dense output, but is slowest (Table 2) of allthealgorithmswetested.Uncompressiontime(Table3)isunaffectedby theselectionofthecompressionalgorithm.

Table 1. Compressed data size for the four file corpora and for common compression algorithms along with Zopfli. The output produced by Zopfli is 3.78.3 % smaller than thatofgzip9. Benchmark Corpussize gzip9 7zip kzip Zopfli

Alexatop10k Calgary Canterbury enwik8

693108837 3141622 2818976 100000000

128498665 1017624 730732 36445248

125599259 980674 675163 35102976

125163521 978993 674321 35025767

123755118 974579 669933 34995756

Table 2. Compression times for enwik8. Zopfli is 81 times slower than the fastest measuredalgorithmgzip9. Compressionalgorithm Compressiontime

gzip9 7zipmm=Deflatemx=9 kzip Zopfli

5.60s 128s 336s 454s

Table 3. Uncompression times for data thatwerecompressedwithdifferentalgorithmsare tested with running gzip d for the compressed enwik8 corpus. Weobtained the run times from 9 runs, and chose the median time. The difference between fastest and slowest are within2.5%. Compressionalgorithm Uncompresstimeforgzipdofenwik8

gzip9 7zipmm=Deflatemx=9 kzip Zopfli

934ms 949ms 937ms 926ms

Discussion
Zopfli gives smallerdeflatecompatibleoutputsizethangzip(3.78.3%smaller),7zip,andkzip, with more CPU time used at compression phase. This makes Zopfli ideal for uses where the cost of CPU is small in relation to the output size. Such use could includedensercompression ofstaticcontentformakingwebsitesfaster. Zopfli and gzip compresstogzipformat,whereaskzipand7zipcompresstozipformat. This may alter the sizes slightly as the container format has slightly different overhead. For enwik8,theheaderoverheaddifferenceisbelow0.0001%inrelationtotheoutputsize. 7Zip can operate with the deflate format, but it can read and write several otherarchive formats, and achieve higher compression ratios. In this study we only measured deflatecompatiblecompression. We could achieve faster results with gzip and other algorithms by specifying lower compression density options. In this study we are interested on finding the smallest possible

compressed size, and because of this we have only run every algorithm with maximum compression options. Zopfli also can run even longer to achieve slightly higher compression density,butwechosetorunitwithdefaultsettings. In the light of the results wepresented, we recommend Zopfli for compression ofstatic content and other content where data transfer or storage costs are more significant than the increase in CPU time. To our knowledge Zopfli typically produces the highest compression densityofanydeflatecompatiblealgorithm. Zopfli is opensourcedathttps://code.google.com/p/zopfli/.We inviteeveryonetotryitout, andhopethatitwillfindmanypracticaluses.

References
1. 2. 3. 4. 5. 6. 7. 8. 9. http://en.wikipedia.org/wiki/Zlib http://en.wikipedia.org/wiki/7Zip http://www.advsys.net/ken/utils.htm Alexatop10kcorpus: https://code.google.com/p/httparchive/source/browse/trunk/lists/Alexa10K.txt Calgarycorpus:http://www.datacompression.info/Corpora/CalgaryCorpus.zip Canterburycorpus:http://corpus.canterbury.ac.nz/resources/cantrbry.zip enwik8corpus: http://mattmahoney.net/dc/text.htmlhttp://mattmahoney.net/dc/enwik8.zip P.Deutsch,RFC1952GZIPfileformatspecificationversion4.3, http://www.ietf.org/rfc/rfc1952.txt P.Deutsch,RFC1951DEFLATECompressedDataFormatSpecificationversion1.3, http://www.ietf.org/rfc/rfc1951.txt