Preprint

Compressed Sensing : The Emerging Sampling Paradigm.

Pradosh K. Roy FIETE , C.Eng Asia Pacific Institute of Information Technology , Panipat , 132103 India roypkin@yahoo.com Ankit Kundu , Flat 2C, Tower 3 , South City Gardens , Calcutta 700053, India ankitkundu931@yahho.com Keywords : Compressed Sensing , Basis Pursuit , Sparse Approximation , Sparse Recovery , L 1 minimization

I.

Introduction

A signal f(t) is known as bandlimited , if its Fourier Transform is zero outside a finite interval i.e. F() = 0 for > and its Energy is finite i.e. ,

Band-limitedness, if only approximately , is a realistic assumption reflecting natural properties of responses of dynamical systems [1]. Conventional approaches to sampling the band limited signals or images in the time or space domain follow the Whittaker-Nyquist-Kotelnikov -Shannon Theorem which specifies the number of measurements required to reconstruct the signal .The theorem mathematically establishes that signals, images, videos, and other data can be exactly recovered from a set of uniformly spaced samples taken at the so-called Nyquist rate of twice the highest frequency present in the signal of interest. Signal processing has moved from the analog to the digital domain , ridden the wave of Moore's law , h eralded the era of digital data , on the basis of this theorem. However , phenomenal growth in digital data , aptly termed as digital data deluge by Richard Baraniuk , has been witnessed in 21 st Century. An IDC report published in 2010 found that 0.8 Zeta Bytes of digital data generated in 2009 , would be 35 Zeta Bytes in the year 2020. It is needless to say that the digital data is predominantly sensor data [2]. Successfully navigating the data deluge calls for fundamental advances in the theory and practice of sensor design; signal processing algorithms; wideband communication systems; and compression, triage, and storage techniques [3]. In response to the resulting challenge signal -processing researchers have spent the last several decades creating powerful new theory and technology for digital data acquisition (digital cameras, medical scanners), digital signal processing (machine vision; speech, audio, image, and video compression), and digital communication (high-speed modems,Wi-Fi) that have both enabled and accelerated the information age [3]. The key factor which has enabled mapping of high-dimensional raw sensor data to an extremely lowdimensional subset is the observation that many natural signals are sparse or compressible in the sense that they have concise representations when expressed in a transformed domain or basis. However , these socalled lossy compression techniques require acquisition fulfilling the Nyquist criterion. An acquisition protocol which performs as if it were possible to directly acquire just the important information about the signals/imagesin effect, not acquiring that part of the data that would eventually just be thrown away by lossy compression has recently been proposed by Donoho , Cand s , Romberg and Tao [5][6] . The neologism Compressed Sensing[CS] was coined by David L. Donoho of Stanford University in 2006 to represent this sensing or sampling paradigm . CS is a mathematical framework which predicts that high dimensional signals which allow a sparse representation by a suitable transformation , can be recovered from what was earlier considered highly incomplete linear measurements , by using efficient algorithms. CS explores methodologies from various other fields viz. applied harmonic analysis , functional analysis , numerical linear algebra , optimization theory , random matrix theory , probability theory and computer science. The objective of this article is to introduce the key mathematical concepts , exemplified by the sparse recovery of a speech signal. Readers may also refer to the excellent tutorials and reviews on the theory of Compressed Sensing [4][8][9][10] for exploring this novel research area. IEEE Signal Processing Magazine March 2008 may be also be referred for a comprehensive understanding of the theory.

II.

Transform Coding

The most popular technique for signal compression is Transform Coding which typically involves in finding a suitable basis e.g. Discrete Cosine Transform [DCT] , Discrete Wavelet Transform [DWT] etc. , providing sparse or compressible representations . Transforms by themselves ,it should be recalled , do not provide any compression. However, by reallocation of the energy in the data, transforms provide the possibilities for compression. By a sparse representation, we mean that for a signal of length n , we can represent it with nonzero coefficients; by a compressible representation, we mean that the signal is well-approximated with only k nonzero coefficients. Both sparse and compressible signals can be represented with high accuracy by preserving only the values and locations of the k largest coefficients of the signal. The approximate coefficient series is inverse transformed to yield an approximate representation of the original signal. This process called sparse approximation, forms the foundation of transform coding schemes that exploit signal sparsity and compressibility , including jpeg, jpeg2000, mpeg, and mp3 standards for the compression of image and audio signals [4]. More explicitly , a signal x is compressible if its sorted coefficient magnitudes ] in transform domain follow the power law decay i.e. The larger q is, the faster the magnitudes decay, and the more compressible a signal is. As for example , Fig 1 illustrates compressibility of a speech signal that follows in the DCT basis. Mean Square Errors of the order of O(10-4)between the true signal and its k term approximations i.e. by preserving only the k coefficients with largest magnitudes, also known as hard thresholding , shown in Fig. 2, confirms that the signal is sparse and compressible.

Fig1 Power Law Decay of Sorted DCT Coefficients.

Fig. 2 MSE between Original and the k term Approximate Signal.

III. Let

Theory of Compressed Sensing be the x sparse signal of interest i.e.

,
Compressed Sensing (CS) , the emerging sampling paradigm , asserts that the signal could be recovered from very few nonadaptive linear measurements is the sensing matrix] , by solving a convex optimization problem [5][6]. Since x is k sparse, x must belong to one of nCk subspaces in . Similarly, y must belong to one of mCk subspaces in . For almost all with m k + 1, an exhaustive search through the subspaces can determine which subspace x belongs to and thereby recover the signals sparsity pattern and values. Therefore, in principle, a k sparse signal can be recovered from as few as m = k + 1 random samples [11]. However the exhaustive search is computationally intractable for even moderately large values of k and n. The theory of CS exhibit recovery methods via the following L1 norm minimization or Basis Pursuit , which is numerically stable and robust against noise while requiring a number of measurements comparable to k :

(3).

In other words , CS predicts unique and stable signal recovery from incomplete measurements y . In contrast with transform coding, here we do not operate on the signal x directly, but rather only have access to the CS measurement vector y [9]. Fundamental questions pertaining to necessary and sufficient conditions for solving the underdetermined linear system of equations (3) , summarized by Gitta Kutyniok [10] are : (i) (ii) (iii) (iv) What are suitable signals and sparsity models ? What are suitable sensing matrices ? How can the signal be algorithmically recovered ? When and with which accuracy can the signal be recovered ?

Necessary conditions for existence of sparse solution are : (i) sparsity, a prerequisite for the signal of interest, and (ii) incoherence, which pertains to the sensing matrix . Whereas sparsity expresses the idea that the information rate of a signal may be much smaller than suggested by its bandwidth as explained ear lier , incoherence extends the duality between time and frequency and expresses the idea that the objects having a sparse representation in must be spread out in the domain in which they are acquired , just as a Dirac or spike in the time domain is spread out in frequency domain [7]. Intuitively , incoherence could also be correlated to the time frequency uncertainty principle of Werner Heisenberg . The coherence between the sensing basis and the representation basis is defined as : and measures the largest correlation between any two elements of sensing matrix and . is bounded by and the . Compressed sensing is concerned with low coherence pairs i.e. sensing matrix representation basis must be incoherent. Random matrices are largely incoherent with any fixed representation basis such as DCT and DWT. If is an orthobasis selected uniformly at random , then the coherence between and [7]. Therefore , the best choice for are random matrices such as i.i.d Gaussian , Bernoulli and the Identity Matrix. Stability and uniqueness of the solution of the compressed sensing problem (3) is ensured by Reduced Isometry Property [RIP] , which measures the degree to which each subset of k column vectors of is close to being an isometry. The sensing matrix has RIP of order k , if there exists a such that | | ,

Where the set contains all signals x that are k-sparse . An equivalent description of the RIP is to say that all subsets of k columns taken from are in fact nearly orthogonal [7].

IV.

Results

A speech signal of 250ms duration sampled at 8kHz and its sparse recovery using Basis Pursuit with 93.75% compression [k= 128 in DCT domain] are compared in Fig 3. Sensing matrix used was i.i.d Gaussian (0,0.02). It can be easily verified that MSE between the original and k term approximation as well as between the original and the sparse recovered signals , using inverse DCT and Basis Pursuit respectively , both are of the order of 10-4 and almost identical at any level of compression , thereby validating the accuracy of the recovered signal.

Fig 3. Original and Sparse Recovered Speech Signals.

V.

Conclusion

The potential applications of sparse recovery is being explored in real world signal processing , data acquisition , channel coding , medical imaging etc. Several future research directions and open questions have been suggested by Bruckstein , Donoho and Elad [8]. In conclusion we can say that removing the Nyquist barrier in the resolution limited applications can improve the user experience, increase data transfer, improve imaging quality and reduce exposure time in other words, make a prominent impact on the analog-digital world surrounding us as foreseen by Baraniuk [3] , in near future.

References.
[1] Papoulis,Athansios., 1977, Signal Analysis, McGraw Hill Inc.,Singapore,p.183-220 [2] Gantz , John and David, Reinsel , The Digital Universe Decade Are You Ready? , IDC White Paper, May 2010; http://idcdocserv.com/925 [3] Richard G Baraniuk , More is Less : Signal Processing and the Data Deluge , Science, vol.331, p.717-719, 11 February , 2011 [4]Baraniuk, Richard G. , Davenport, Mark A., Duarte, Marco F., Hegde , Chinmay, [Eds.] , An Introduction to Compressive Sensing , 2012,Rice University , Houston, Texas, http://cnx.org/content/col11133/1.5/ [5]Donoho,D.L.,Compressed Sensing , IEEE Transactions on Information Theory , vol. 52, pp.1289-1306., 2006. [6]Candes, E., Romberg, J., Tao, T., Robust UncertaintyPrinciples: Exact Signal Reconstruction from Highly Incomplete Frequency Information, IEEE Transaction on Information Theory, vol. 52, pp. 489509 , 2006 [7] Candes , Emmanuel G., Wakin , Michael B., An Introduction to Compressive Sampling , IEEE Signal Processing Magazine , March 2008 , p.21-30. [8] Bruckstein , Alfred M., Donoho, David L.,Elad , Michael , From Sparse Solutions of System of Equations to Sparse Modeling of Signals and Images , SIAM Review , vol 51 , no.1, 2009 , p.34 -81. [9] Marco F. Duarte , Yonina C. Eldar , Structured Compressed Sensing : from Theory to Applications , IEEE Trans. Signal Proc., vol. 59, issue 9 , p.4053-4085 ,2011. [10] Gitta Kutyniok , Theory and Applications of Compressed Sensing , GAMM Mitteileungen, 10 July, 2013 available at http://www.math.tu-berlin.de/fileadmin/i26_fg- kutyniok/Kutyniok/Papers/SurveyCS.pdf [11] Goyal, Vivek K.,Fletcher , Alyson K., Rangan , Sundeep , Compressive Sampling & Lossy Compression , IEEE Signal Processing Magazine , March 2008 , p. 48-56