DESIGN AND ANALYSIS OF LDPC CONVOLUTIONAL CODES

A Dissertation

Submitted to the Graduate School of the University of Notre Dame in Partial Fulllment of the Requirements for the Degree of

Doctor of Philosophy

by

Arvind Sridharan, M. S.

Daniel J. Costello, Jr., Director

Graduate Program in Electrical Engineering Notre Dame, Indiana February 2005

c Copyright by Arvind Sridharan 2005 All Rights Reserved

DESIGN AND ANALYSIS OF LDPC CONVOLUTIONAL CODES

Abstract by Arvind Sridharan Low-density parity-check (LDPC) block codes invented by Gallager in 1962 achieve exceptional error performance on a wide variety of communication channels. LDPC convolutional codes are the convolutional counterparts of LDPC block codes. This dissertation describes techniques for the design of LDPC convolutional codes, and analyzes their distance properties and iterative decoding convergence thresholds. The construction of time invariant LDPC convolutional codes by unwrapping the Tanner graph of algebraically constructed quasi-cyclic LDPC codes is described. The convolutional codes outperform the quasi-cyclic codes from which they are derived. The design of parity-check matrices for time invariant LDPC convolutional codes by the polynomial extension of a base matrix is proposed. An upper bound on free distance, proving that time invariant LDPC codes are not asymptotically good, is obtained. The Tanner graph is used to describe a pipelined message passing based iterative decoder for LDPC convolutional codes that outputs decoding results continuously. The performance of LDPC block and convolutional codes are compared for xed decoding parameters like computational complexity, processor complexity, and decoding delay. In each case, the LDPC convolutional code performance is

Arvind Sridharan at least as good as that of LDPC block codes. An analog circuit to implement pipelined decoding of LDPC convolutional codes is proposed. The distance properties of a permutation matrix based (time varying) ensemble of (J, K ) regular LDPC convolutional codes is investigated. It is proved that these codes (for J > 2) have free distance increasing linearly with constraint length, i.e., they are asymptotically good. Further, the asymptotic free distance to constraint length ratio for the convolutional codes is several times larger than the minimum distance to block length ratio for corresponding LDPC block codes. Iterative decoding of terminated LDPC convolutional codes, based on the ensemble mentioned above, is analyzed, assuming transmission over a binary erasure channel. The structured irregularity of the codes leads to signicantly better convergence thresholds compared to corresponding LDPC block codes. At the calculated thresholds, both the bit and block error probability can be made arbitrarily small. The results obtained suggest that the thresholds approach capacity with increasing J .

To all my teachers

ii

CONTENTS

FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

vi ix x

ACKNOWLEDGMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . .

CHAPTER 1: INTRODUCTION . . . . 1.1 Digital communication systems 1.2 Channel models . . . . . . . . . 1.3 Channel codes . . . . . . . . . . 1.3.1 Block codes . . . . . . . 1.3.2 Convolutional codes . . 1.4 Channel decoding . . . . . . . . 1.4.1 Optimal decoding . . . . 1.4.2 Iterative decoding . . . . 1.5 Outline of the thesis . . . . . . CHAPTER 2: BACKGROUND . . . . 2.1 LDPC block codes . . . . . . 2.2 Iterative decoding techniques 2.3 LDPC convolutional codes . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

. . . . . . . . . . . . . .

1 1 4 7 7 10 13 13 14 15 17 17 20 24

CHAPTER 3: LDPC CONVOLUTIONAL CODES CONSTRUCTED FROM QUASI-CYCLIC CODES . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Code construction . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Construction of quasi-cyclic LDPC block codes . . . . . . 3.1.2 Construction of LDPC convolutional codes . . . . . . . . . 3.2 Properties of constructed codes . . . . . . . . . . . . . . . . . . . 3.2.1 Relation between the block and convolutional Tanner graphs

30 31 31 36 42 42

iii

3.3

Quasi-cyclic block codes viewed as tail-biting codes . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Girth . . . . . . . . . . . . . . . . . . . . . . 3.2.4 Minimum distance . . . . . . . . . . . . . . 3.2.5 Encoding . . . . . . . . . . . . . . . . . . . Performance results . . . . . . . . . . . . . . . . . .

3.2.2

convolutional . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

43 44 46 47 49

CHAPTER 4: A CONSTRUCTION FOR TIME INVARIANT LDPC CONVOLUTIONAL CODES . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Code construction technique . . . . . . . . . . . . . . . . . . . . . 4.2 Choosing the base matrix . . . . . . . . . . . . . . . . . . . . . . 4.2.1 Girth properties . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Optimizing thresholds . . . . . . . . . . . . . . . . . . . . 4.3 Distance bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . CHAPTER 5: IMPLEMENTATION OF LDPC CONVOLUTIONAL CODE DECODERS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Decoder description . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Comparison of message passing decoding of block and convolutional codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Analog decoding . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.1 Analog decoding of LDPC convolutional codes . . . . . . . CHAPTER 6: DISTANCE BOUNDS FOR LDPC CONVOLUTIONAL CODES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 An LDPC convolutional code ensemble . . . . . . . . . . . . . . . 6.2 A lower bound on row distance . . . . . . . . . . . . . . . . . . . 6.2.1 Row distance . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.2 Probability for a single block of constraints . . . . . . . . . 6.2.3 Lower bound on row distance . . . . . . . . . . . . . . . . 6.3 A lower bound on free distance . . . . . . . . . . . . . . . . . . . 6.4 Results and discussion . . . . . . . . . . . . . . . . . . . . . . . . CHAPTER 7: CONVERGENCE ANALYSIS ON THE ERASURE CHANNEL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1 Tanner graph description . . . . . . . . . . . . . . . . . . . . . . . 7.2 Termination of LDPC convolutional codes . . . . . . . . . . . . . 7.3 Decoding analysis on the erasure channel . . . . . . . . . . . . . . 7.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . CHAPTER 8: CONCLUDING REMARKS . . . . . . . . . . . . . . . . .

55 56 60 61 62 63

67 68 73 77 79

87 88 92 93 95 102 107 111

117 118 119 134 140 147

iv

APPENDIX A: MAXIMUM OF THE FUNCTION U ([1,L] ) . . . . . . . . BIBLIOGRAPHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

150 152

FIGURES

1.1 1.2 1.3 1.4 1.5 2.1

Block diagram of a digital communication system. . . . . . . . . . The binary erasure channel . . . . . . . . . . . . . . . . . . . . . . The binary symmetric channel . . . . . . . . . . . . . . . . . . . . Illustration of the additive white Gaussian noise channel model. . A rate 1/2 convolutional encoder with memory 2. . . . . . . . . . The Tanner graph of the parity-check matrix of Example 2.1.1. The Tanner graph consists of 20 symbol nodes, each with J = 3 edges and 15 constraint nodes, each with K = 4 edges. . . . . . . . . . Illustration of the J imenez-Zigangirov method to construct the syndrome former of a b = 1, c = 2 LDPC convolutional (4, 3, 6) code . The Tanner graph for the convolutional code of Example 2.3.2. . . Tanner graph for a [155,64,20] QC code. . . . . . . . . . . . . . . Tanner graph for a [21,8,6] QC code. . . . . . . . . . . . . . . . . Tanner graph for the rate 1/3 convolutional code of Example 5. . A 12-cycle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A systematic syndrome former based encodes for the R = 1/3 code of Example 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The performance of convolutional LDPC codes versus the corresponding QC block LDPC codes for (3,5) connectivity. . . . . . . The performance of convolutional LDPC codes versus the corresponding QC block LDPC codes for (5,7) connectivity. . . . . . . The performance of convolutional LDPC codes versus the corresponding QC block LDPC codes for (3,17) connectivity. . . . . . . Performance of regular (3, 5) QC block codes with increasing circulant size m. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2 5 5 6 10

19 26 28 35 36 42 45 48 50 52 53 54

2.2 2.3 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9

vi

4.1

Performance of the codes in Examples 4.1.1-4.1.3 on the AWGN channel. The performance of a QC code with N = 1600 obtained from the convolutional code of Example 4.1.3 is also shown. . . . Existence of six cycle with polynomials of weight 3 or more. . . .

60 61

4.2 5.1 5.2 5.3

The sliding window decoder for I iterations of the code of Example 5. 70 Continuous decoding of the code of Example 5 with I iterations. . Performance comparison of LDPC block and convolutional codes on the AWGN channel for the same processor complexity and decoding delays. The block code is decoded with a very fast decoder. . . . . Serial decoding of LDPC block codes using I decoders. . . . . . . Transistor circuits for implementing the symbol (SYM) and constraint (CHK) node updates. . . . . . . . . . . . . . . . . . . . . . The rotating ring decoder . . . . . . . . . . . . . . . . . . . . . . Bit error rate curves on the AWGN channel with rotating ring decoders (Type c)) of dierent window sizes. The ML performance is also shown. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bit error rate curves for dierent decoder types with W=9. Type a) is the usual sliding window decoder. Type b) is a rotating ring decoder with channel values reloaded at each time. Type c) is a standard rotating window decoder (no reloading) . . . . . . . . . Decoding results for the time discrete decoder (left two columns) and the time continuous decoder (right two columns) for one received block of length 55 bits. Each row represents a dierent test case. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

76 77 78 80

5.4 5.5 5.6 5.7

82

5.8

83

5.9

85

5.10 Probability distributions of output LLRs for analog and digital decoders. L(z ) is the decoder output and p(L(z )) is the probability distribution function of the random variable L(z ). . . . . . . . . . 6.1 6.2 6.3 6.4 6.5 6.6
syndrome former for a code in CP (3, 6, M ).

86 89 91 94

. . . . . . . . . . . . . .
T

Illustration of length L = 3 segment of

Permuting rows of a syndrome former H in CP (3, 6, M ). . . . . . HT [1,L] .

. . . . . . . . . . .

Set of parameters 3 and 3 for calculating the function F3 ([1,3] , [1,3] )103
(L) Lower bound on row distance to constraint length ratio d as a 6M function of L. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

112 113

Maximizing weight vector [1,L] for dierent L. . . . . . . . . . . .

vii

7.1 7.2 7.3 7.4 7.5 7.6 7.7 7.8 7.9

Tanner graph connections of the 2M symbol nodes at time t. . . . The matrix H T [L+1,L+3] . . . . . . . . . . . . . . . . . . . . . . . . . The matrices resulting from row operations on A. . . . . . . . . . The matrices C 3 and C 2 . . . . . . . . . . . . . . . . . . . . . . . Tanner graph of a terminated convolutional code obtained from C (M, 2M, 3). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

118 123 125 130 133 138 139 143

Tanner graph connections of the M parity-check equations at time t.119

Illustration of the messages (a) to a symbol node and (b) to a constraint node for the case J = 3. . . . . . . . . . . . . . . . . . The rst level of computation trees for t = 1, 2, 3 with J = 3. . . . Illustration of the tunneling eect and the breakout value . . . . .

viii

TABLES

3.1 4.1 6.1 7.1

EXAMPLES OF CODES CONSTRUCTED FROM (PRIME) CIRCULANT SIZES. . . . . . . . . . . . . . . . . . . . . . . . . . . . BEST THRESHOLDS OBTAINED FOR BASE MATRICES OF DIFFERENT SIZES . . . . . . . . . . . . . . . . . . . . . . . . .

34 64

DISTANCE BOUND FOR LDPC BLOCK AND CONVOLUTIONAL CODES FOR DIFFERENT (J, K ). . . . . . . . . . . . . . . . . . 115 THRESHOLDS FOR THE ENSEMBLE CP (3, 6, M ) WITH DIFFERENT L AND FOR THE CORRESPONDING IRREGULAR BLOCK CODES. . . . . . . . . . . . . . . . . . . . . . . . . . . .

140

7.2

THRESHOLDS FOR THE ENSEMBLES CP (J, 2J, M ) WITH DIFFERENT J . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

ix

ACKNOWLEDGMENTS

This dissertation would not have been possible without the help of numerous people over the past ve years. Although, it is only possible to thank a few here, I would like to express my gratitude to all of them. I would like to thank Dr. Costello for being a wonderful adviser. He provided constant support, encouragement, made himself readily available, and always had a patient ear, even for all the nonsense I had to say. Many thanks for the wonderful six months in Germany! Dr. Tanner gave a series of interesting talks in the Fall of 2000. I would like to thank him for these talks and other stimulating discussions which has resulted in a chapter of this dissertation. Dr. Kamil Zigangirovs help in shaping my research cannot be over emphasized. Kamils down to earth personality and enthusiasm meant that working with him has not only been a great opportunity and learning experience but also great fun. My heartfelt thanks to him also for the the wonderful dinners he cooked and for the demonstrations of the art of drinking vodka, in both Russian and Swedish styles. Dr. Michael Lentmaier, during his time at Notre Dame, taught me the value of being systematic. His clear insight and attitude to research was a wonderful example. Very often he dropped everything he was doing to discuss my research problems. These discussions have been immensely useful.

I would like to thank Dr. Fuja and Dr. Collins for serving on my defense committee, and Dr. Peter Massey, Andrew Schaefer, and Dr. Dmitry Trukhachev for the opportunity to discuss and work together. My friends and colleagues at Notre Dame; Adrish, Ajay, Chandra, Junying, Ching, Ravi, Rajkumar, Tony, Ajit, Hongzhi, Shirish, Vasu, Sridhar, Ali, Xiaowei, Guanjun, Kameshwar, Shiva, Jagadish, to name a few made Notre Dame a pleasant place to work at. Special thanks to Ali for meeting every single inane problem I had about using the computer with a smile and solving it. I would also like to
A thank Sameer for providing the L TEXclass le used to format the dissertation.

I cannot thank the EE department sta enough for all their help. I would like to specially thank Pat for always doing more than was needed. The music concerts, cinema, theater, wonderful library, and sports facilities at Notre Dame meant I always had more to do than time permitted. The opportunity to see several plays of Shakespeare enacted on stage by Actors from the London Stage will remain an unforgettable part of my time at Notre Dame. A signicant reason for the wonderful time I had at Notre Dame was Deepak. During his time at Notre Dame, Deepak was my friend, philosopher, guide, punching bag, and more. I have had countless discussions with him on virtually everything under the sun. I consider myself fortunate to have spent time with him. My parents and Arathi made home a wonderful place to be. They have always provided me with unconditional support, love, and encouragement. Anthony, Clement, and Dilip have taught me much and continue to do so. I am much the better for knowing them.

xi

CHAPTER 1 INTRODUCTION

Today the use of digital cell phones, the internet, and CD and DVD players is ubiquitous. In all of these cases, digitally represented data is either transmitted from one place to another or retrieved from a storage device when needed. For the proper functioning of these systems the transmitted or retrieved data must be suciently error free. This can be accomplished eciently by using channel coding techniques. One such channel coding technique for achieving reliable data transmission or retrieval is to use low-density parity-check (LDPC) convolutional codes. The design of LDPC convolutional codes and an analysis of their properties are the topic of this dissertation. The next section describes a model of a general digital communication system. In Section 1.2, the discrete time model of the communication channels considered throughout this dissertation is described. The next two sections describe the basic concepts of channel coding and decoding. In Section 1.5, an outline of the rest of the thesis is presented.

1.1 Digital communication systems Most generally, a communication system aims at transmitting information from a source to a destination across a channel. The nature of the source and

Binary Information Source Source Source encoder

u

Discrete Channel Channel encoder

v

Modulator

Physical Channel
PSfrag replacements

Binary Information Sink Destination Source decoder

u

Channel decoder

Demodulator

Figure 1.1. Block diagram of a digital communication system.

destination, e.g., computers exchanging messages or a conversation between two humans, is not of importance. Similarly, what the information exchanged represents, e.g., music or conversation, is not essential. The information exchange can take place across two dierent locations, as in the case of telephone conversations, or across both dierent times and locations as in the case of storage and retrieval systems like CDs. Claude Shannon, in his famous paper [1], pioneered the area of information theory. This paper quantied the notion of information and has been the basis of the study and design of communication systems since. One of Shannons results is that it is possible to separate the processing of the source output before transmission into source encoding and channel encoding without sacricing optimality. The general model of a digital communication system based on this separation principle is shown in Fig. 1.1.

Typically, the data generated from most sources has redundancy, the transmission of which can be avoided. For example, in a phone conversation there are often periods of silence in the conversation which need not be transmitted. The function of the source encoder is two fold, one, convert the information from the source into a binary sequence u, the information sequence, and two, remove all redundancy in the source information in obtaining u. Probably the most remarkable result obtained by Shannon in [1] is the noisy channel coding theorem. This theorem states that, for a broad class of channels, reliable transmission is possible as long as the transmission rate is less than the channel capacity. This is achieved by introducing redundancy, by means of a channel code, suited for overcoming the debilitating eects of the channel on the data to be transmitted. This result led to the subject of coding theory, which investigates the use of ecient and practical coding schemes to achieve reliable communication at information rates close to the capacity of the channel. The function of the channel code is to add redundancy in a controlled fashion to the information sequence u and generate the code sequence v to facilitate reliable transmission to the destination. The modulator converts symbols of the code sequence v into a form suited for transmission across the physical channel. The physical channel is the medium across which data is transmitted. It can be, for example, a simple wire, a storage medium like a CD, or the wireless channel from a base station to a cell phone user. The channel, while allowing the transfer of data, corrupts the transmitted signal from the modulator in some fashion. As already noted, Shannon proved that as long as the transmission rate is less than the channel capacity these eects can be overcome.

The blocks after the channel undo the operations performed during data transmission. The demodulator generates a discrete sequence r, the received sequence, from the output of the channel. The channel decoder uses r to generate an estimate u of the information sequence u. Ideally, u is the same as u. The source decoder uses the estimated information sequence u and regenerates the source data in the form required by the destination. In the scope of this work, we are concerned with channel encoding and decoding. We therefore group the blocks in Fig. 1.1 as shown so that we obtain a binary information source, a discrete channel, and a binary information sink. The information sequence1 u is transmitted from the binary source across a discrete channel to the binary sink. The discrete channel corrupts the transmitted code sequence and this can be overcome by appropriately designed coding schemes.

1.2 Channel models In this section we describe three dierent models for the discrete channel, the binary erasure channel (BEC), the binary symmetric channel (BSC), and the additive white Gaussian noise (AWGN) channel. The binary erasure channel is shown in Fig 1.2. In this case, there are three possible channel outputs, the transmitted symbols 0 and 1, or an erasure . The BEC does not cause any errors, a symbol is either received correctly or as an erasure. The probability of a symbol (0 or 1) being received as an erasure, the erasure probability of the channel, is p. The BEC has capacity 1 p bits per channel use.
For an ideal source encoder, the bits of the information sequence are independent and equally likely and we assume that this is the case throughout the dissertation.
1

0 PSfrag replacements

1p p

0 e

p 1 1p Figure 1.2. The binary erasure channel 1

0 PSfrag replacements

1p p p

1 1p

Figure 1.3. The binary symmetric channel

The BSC is shown in Fig 1.3. In this case a 0 can be erroneously received as a 1 or vice versa with crossover probability p. The BSC has capacity 1 h(p) bits per channel use where the binary entropy function h(p) is dened as

h(p) = p log2 p (1 p) log2 (1 p)

(1.1)

The third model we consider, the AWGN channel, has binary inputs as earlier but the outputs are real numbers. The model for the AWGN channel is shown in Fig. 1.4. The modulator maps the bits of v into antipodal waveforms with energy

nt N (0, N0 /2) PSfrag replacements vt {0, 1} Es rt R

Modulator

Figure 1.4. Illustration of the additive white Gaussian noise channel model. Es . The waveforms are represented by their amplitudes Es in Fig. 1.4. The channel adds noise with one-sided power spectral density N0 to the transmitted waveforms. The optimal demodulator generates at each time t the real number

rt = (1 2vt ) Es + nt , vt {0, 1},

(1.2)

where nt is a Gaussian random variable with variance 2 = N0 /2. The received value rt is therefore a random variable with conditional probability density function p(rt |vt ) =
1 2 e(rt (12vt ) Es ) /N0 . N0

(1.3)

The AWGN channel model is a good model for channels where noise in the channel is the result of a superposition of a large number of independent identically distributed noise sources. The capacity of the discrete AWGN channel considered here must be calculated numerically. In this dissertation we will deal mainly with the AWGN channel except for the penultimate chapter where we focus on the erasure channel.

1.3 Channel codes Channel codes introduce redundancy into the information sequence so that it is possible to correct erroneous symbols altered by the channel and thus ensure reliable transmission. For this reason, channel codes are often referred to as error correcting codes. In this section, a brief introduction to error correcting codes is presented. We review the two main classes of codes, block codes and convolutional codes, and associated concepts. A more thorough treatment of block codes can be found in several textbooks [2][3][4][5][6]. Convolutional codes are described in detail in [7][6].

1.3.1 Block codes Consider data transmission over a BSC. A simple way to combat errors in this case is to repeat each information bit N times, a length N repetition code. The decoder uses a majority vote to determine the information bit. The decoding decision is correct as long as the number of errors is less than N/2. Thus by increasing N , arbitrary reliability can be achieved. The code rate R, the number of information symbols transmitted per channel use, for a length N repetition code, is R = 1/N . The code rate of a repetition code tends to zero as N is increased which makes it not very interesting in practice. A more ecient coding scheme is to encode K > 1 information symbols together to generate a codeword of length N . Example 1.3.1 A rate R = 4/7 error correcting code

A codeword v = (v0 , v1 , v2 , v3 , v4 , v5 , v6 ) of length N = 7 is generated for a group of K = 4 information bits u = (u0 , u1 , u2 , u3 ) as

u = (u0 , u1 , u2 , u3 ) v = (u0 , u1 , u2 , u3 , u0 + u1 + u2 , u1 + u2 + u3 , u0 + u1 + u3 ). (1.4) The addition here is over the binary eld GF(2). The code is dened by the 2K = 16 dierent codewords corresponding to the 2K dierent information blocks u. Since there are N = 7 code bits for K = 4 information bits, the code rate is R = 4/7. 2

A code of the form described in Example 1.3.1, where codewords are generated for a block of information symbols, is called a block code. In the dissertation we only focus on linear block codes over the binary eld GF(2). An (N, K ) linear block code of rate R = K/N is a K dimensional vector space of the vector space GF(2)N . The code in Example 1.3.1 is a (7, 4) linear block code. For a linear code, the codeword corresponding to an information block can be conveniently specied by means of matrix multiplication as v 1N = u1K GK N , where G is a generator matrix. In Example 1.3.1, the generator matrix is 1 0 0 0 1 0 1 (1.5)

0 1 0 0 1 1 1 G= 0 0 1 0 1 1 0 0 0 0 1 0 1 1

(47).

Any set of K basis vectors can be used to obtain a generator matrix. The dierent generator matrices specify the same code but the mapping of information blocks to codewords is dierent. A systematic generator matrix is one where the K

information symbols appear as a part of the codeword. The generator matrix in (1.5) is systematic. The Hamming weight of a vector x is the number of non-zero components in x. The Hamming distance between two vectors x and y is the Hamming weight of the vector x y = x + y .2 The minimum distance dmin of a block code C is the minimum of the Hamming distances between two distinct codewords in C . For a linear block code C , the sum of any two codewords is also a codeword, and hence dmin is equal to the minimum Hamming weight over all non-zero codewords in C . The minimum distance of a code plays a signicant role in determining the error correcting capability of the code; a code with minimum distance dmin can correct up to (dmin 1)/2 errors3 on the BSC. The code of Example 1.3.1 has dmin = 3. In general, determining the minimum distance of a code is dicult, especially if the number of codewords is large. A linear block code can also be specied by means of a parity-check matrix. A parity-check matrix H species the dual space of the code so that for any codeword v we have vH T = 0, (1.6)

where H T denotes the transpose of H . Thus, for a generator matrix G of the block code, GH T = 0. For an (N, K ) linear block code of rate R = K/N , the parity-check matrix is an L N matrix, where L = (N K ). The 3 7 matrix 1 1 1 0 1 0 0 H = 0 1 1 1 0 1 0 1 1 0 1 0 0 1

(1.7)
(37)

2 3

Over the binary eld addition and subtraction are identical. z is the largest integer less than or equal to z

is a parity-check matrix for the code of Example 1.3.1. As in the case of the generator matrix, any set of (N K ) basis vectors of the dual space can be used to form a parity-check matrix of an (N, K ) linear block code. Thus, the same code can be described by several dierent parity-check matrices.

1.3.2 Convolutional codes Convolutional codes were introduced by Elias [8] as an alternative to the class of block codes. A code sequence of a convolutional code is obtained as the output of a convolutional encoder, a linear system. Thus, a code sequence is obtained by the convolution of an information sequence with some generating sequence. In a convolutional code, the information sequence is continuously encoded into a code sequence. Further, the output of a convolutional encoder is dependent on information symbols both at the current and previous times. Example 1.3.2 A b = 1, c = 2, m = 2, rate R = b/c = 1/2 time-invariant convolutional encoder. Figure 1.5 shows a rate R = b/c = 1/2 convolutional encoder. At each time instant, the encoder maps b = 1 information symbols, ut = ut , into c = 2 code vt ut
(0) (0)

PSfrag replacements

vt

(1)

Figure 1.5. A rate 1/2 convolutional encoder with memory 2.

10

symbols, v t = (vt , vt ). The code symbols are obtained as

(0) (0) (0) (0)

(0)

(1)

vt vt

= ut

(1)

= ut + ut1 + ut2 .

(0)

(1.8)

The encoder stores information symbols from m = 2 time instants past, and hence we say that the encoder has memory 2. The encoding equations in (1.8) are the same for each time instant, i.e., the encoder is time invariant. It is also worth noting that the convolutional encoder is a causal system. 2

A general rate R = b/c convolutional encoder maps an information sequence

u = . . . , u0 , u1 , . . . , ut , . . . ,
(0) (b1) ()

(1.9)

where ut = (ut , . . . , ut

) and ut GF(2) into a code sequence v = . . . , v0 , v1 , . . . , vt, . . . , (1.10)

where v t = (vt , . . . , vt

(0)

(c1)

) and vt GF(2).

()

For a convolutional encoder without feedback, as in Example 1.3.2, the encoding can be described as

v t = ut G0 (t) + ut1 G1 (t) + + utm Gm (t),

(1.11)

where m is the memory of the convolutional code and Gi (t), i = 0, 1, . . . , m, are binary b c matrices at each t. For a time invariant encoder the matrices Gi (t), i = 0, 1, . . . , m, are the same for all t. As in the case of block codes, the

11

encoding can be described by a matrix multiplication v = uG, where the matrix

G0 (0) . . . G ( m ) m G (1) . . . G ( m + 1) 0 m G= .. .. . . G0 (t) . . . Gm (t + m) .. .. . . .. . .. .

(1.12)

is called a generator matrix of the code. (The blank area of the matrix is assumed to be lled with zeros.) For the convolutional code in Example 1.3.2, we have G0 = (11), G1 = (01), and G2 = (01).4 We note here that, as in the case of block codes, there are several dierent possible generator matrices for the same convolutional code. For a time invariant convolutional code, the information and code sequences are commonly expressed in terms of the delay operator D as u(D ) = + u0 + u1 D + + ut D t + . . . v (D ) = + v 0 + v 1 D + + v t D t + . . . , (1.13)

and we have v (D ) = u(D )G(D ) where the b c generator matrix G(D ) is equal to G(D ) = G0 + G1 D + + Gm D m . (1.14)

In Example 1.3.2, we have G(D ) = [1, 1 + D + D 2 ] = [1 1] + D [1 0] + [0 1]D 2 . For a time invariant convolutional encoder without feedback, G(D ) consists of polynomials. In general, however, G(D ) consists of rational functions.
4

Since the matrices Gi (t), i = 0, 1, 2, are the same for all t we omit the argument t.

12

The minimum of the Hamming distances between two distinct code sequences of a convolutional code is called the free distance dfree of the code. Since a convolutional code is linear, dfree is equal to the minimum Hamming weight over all non-zero code sequences. As with block codes, on a BSC the transmitted sequence can be recovered as long as there are no more than (dfree 1)/2 symbols in error. 1.4 Channel decoding Suppose that the code sequence5 v corresponding to an information sequence u is transmitted over a discrete channel with binary inputs. The function of the channel decoder is to compute an estimate u of the information sequence u based on the received sequence r . Thus, the reliability of a coding scheme is determined not only by the code but also by the decoding algorithm. An optimal decoding algorithm is one that minimizes the probability of decoding error.

1.4.1 Optimal decoding The decoding block error probability PB of a block code is the probability that the information sequence is decoded incorrectly, i.e.,

PB = P (u = u) = r

P (u = u|r)P (r).

(1.15)

Since, P (r) is independent of the decoding algorithm, PB is minimized by minimizing P (u = u|r) or equivalently maximizing P (u = u|r ) for each r . Thus PB
Whenever the distinction between block and convolutional codes is not relevant, we also use the term sequence in connection with an information block or codeword of a block code.
5

13

is minimized by choosing u as

u = arg max P (u|r ). u

(1.16)

Such a decoder is called a maximum a-posteriori (MAP) decoder. If all information blocks u are equally likely, then a MAP decoder is equivalent to a maximum likelihood (ML) decoder which chooses u as

u = arg max P (r|u). u

(1.17)

In the case of convolutional codes, ML decoding can be eciently performed on a trellis [6][7] using the Viterbi [9][10] algorithm. For equally likely information sequences, the ML decoder minimizes the probability of the decoded sequence being in error. An alternative to minimizing the error probability of the complete sequence or block is to minimize the decoding error probability of a single information bit. The probability that an information bit ui is decoded incorrectly is called the bit error probability Pb = P (ui = ui ). A symbol-by-symbol MAP decoder chooses ui as ui = arg max P (ui = u|r ).
u{0,1}

(1.18)

1.4.2 Iterative decoding In his noisy channel coding theorem, Shannon showed that randomly constructed codes with long block lengths, decoded using an ML decoder, achieve reliable communication at rates equal to the capacity of the channel. One main drawback to a practical implementation of such a coding scheme is that ML decod-

14

ing becomes prohibitively complex for long block lengths. Very often sub-optimal decoding algorithms which provide a reasonable tradeo between complexity and performance are used in practice. An attractive way to construct long powerful codes is by concatenating several smaller codes. While optimal decoding of the long code is still too complex, it can be decoded in an iterative fashion. Decoding is carried out by performing optimal decoding separately for the component codes and then exchanging information between the component decoders. The performance of the overall decoder is greatly improved if the component decoders exchange soft reliability values rather than hard decision values. The component decoders typically calculate the a-posteriori probabilities (APP)

P (ui = u|r) =

v C :ui =u

P (v |r), u {0, 1}

(1.19)

using an APP decoder. Note that if the estimated symbol ui is chosen as the argument that maximizes (1.19), then we obtain the MAP decoder. In the last few years, coding schemes like turbo codes [11] with iterative decoding have led to the design of systems that operate at rates approaching the capacity of AWGN channels. In the dissertation we deal with the low-density parity-check (LDPC) codes, invented by Gallager [12], which are another class of codes that can operate close to the capacity limit on many channels.

1.5 Outline of the thesis In the next chapter, we describe some background material on LDPC codes. We also review the iterative decoding algorithms used for decoding LDPC codes.

15

The following three chapters constitute the design component of the thesis. In Chapter 3, a construction technique for LDPC convolutional codes starting from algebraically constructed quasi-cyclic LDPC codes is presented. The next chapter generalizes the code construction technique and a large class of time invariant LDPC convolutional codes are obtained. We indicate techniques to optimize the code design for improved performance. In this chapter, we also present a bound on the free distance of time invariant LDPC convolutional codes which shows that they are not asymptotically good. Chapter 5 begins with a description of the sliding window decoder for LDPC convolutional codes. We then compare the sliding window decoder with the decoder of LDPC block codes. The chapter concludes with a description of a decoding architecture in analog to decode LDPC convolutional codes. Chapter 6 and 7 serve as the analysis component of the thesis. In Chapter 6, we construct an ensemble of LDPC convolutional codes and show that these codes are asymptotically good. This result is analogous to Gallagers result for LDPC block codes. Chapter 7 describes the iterative decoding analysis of the ensemble of convolutional codes of Chapter 6 on the BEC channel. We show that the convolutional nature of the codes leads to signicantly better thresholds compared to the corresponding LDPC block codes. Chapter 8 summarizes the results of the dissertation and has pointers for future work.

16

CHAPTER 2 BACKGROUND

In this chapter we review several concepts and terminology related to LDPC codes which shall be useful in the remainder of the thesis. Section 2.1 begins with a brief historical sketch of LDPC codes. We then dene an LDPC code and point out how it may also be represented by means of a Tanner graph. In the next section iterative decoding of LDPC codes based on message passing algorithms is described. Section 2.3 deals with LDPC convolutional codes. The Tanner graph representation of a convolutional code and the features that distinguish it from the Tanner graph representation of a block code are also pointed out in this section.

2.1 LDPC block codes LDPC codes and iterative decoding were invented by Gallager in 1963 [12]. With the notable exception of Zyablov and Pinsker [13], and Tanner [14], iterative coding systems were all but forgotten until the introduction of turbo codes [11]. In the wake of the discovery of turbo codes LDPC codes were rediscovered by Mackay and Neal [15]. After the rediscovery of LDPC codes the task of analyzing and designing LDPC codes with performance close to capacity has been carried out in earnest, e.g., [16][17]. Perhaps the culmination of this process was the construction of an LDPC code capable of achieving reliable performance within 0.0045 dB of the capacity limit of the binary input AWGN channel [18]. 17

An LDPC block code is a code dened by a sparse parity-check matrix; containing mostly 0s and relatively few 1s. The LDPC codes introduced by Gallager have the additional property that the number of ones in each row and each column of the parity-check matrix is xed. We call such codes regular LDPC codes. Denition 2.1.1 A regular LDPC block (N, J, K ) code is a code dened by an L N parity-check matrix H , L < N , with exactly J ones in each column (J << L), and K ones in each row. It follows from the denition that L = N J/K and the code rate R 1 (J/K ) since H may not have full rank. The fraction of ones in the parity-check matrices of (N, J, K ) codes decreases with increasing block length. When the block length N is not specied we refer to the class of (J, K ) codes. Example 2.1.1 An LDPC block (20, 3, 4) code Consider the parity-check matrix
H= 0 0 0 0 1 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 1 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 1 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 1 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 1

18

Symbol Nodes

PSfrag replacements

Constraint Nodes

Figure 2.1. The Tanner graph of the parity-check matrix of Example 2.1.1. The Tanner graph consists of 20 symbol nodes, each with J = 3 edges and 15 constraint nodes, each with K = 4 edges.

It is easy to check that there are four ones in each row and three ones in each column so that the parity-check matrix denes a (20, 3, 4) code. H has two dependent rows and the rate of this code is R = 7/20. 2

In his landmark paper [14], Tanner described how a code can be represented by means of a bipartite graph. The bipartite graph representation, called a Tanner graph, consists of symbol nodes corresponding to the columns of H and constraint nodes corresponding to the rows of H . There is an edge connecting a symbol node to a constraint node if the entry in the corresponding column and row of H is a one. Figure 2.1 shows the Tanner graph corresponding to the parity-check matrix in Example 2.1.1. There is a one to one correspondence between a parity-check matrix and its Tanner graph representation. LDPC codes by virtue of their sparse parity-check matrices have sparse Tanner graph representations. As we shall see, the Tanner graph provides a convenient setting to describe iterative decoding algorithms.

19

Gallager, in his dissertation [12], showed the existence of LDPC codes with minimum distance increasing linearly with block length N . Specically, Gallager used an ensemble of (J, K ) regular LDPC codes and proved that for large N almost all codes in this ensemble satisfy dmin G (J, K )N provided J > 2. This result parallels the well known Gilbert-Varshamov (GV) bound [19][20] for the ensemble of linear binary block codes. The GV bound shows the existence of block codes of rate R satisfying dmin GV (R)N for large N . Typically, the Gallager coecient G (J, K ) is smaller than the corresponding GV coecient, thus indicating that LDPC code ensembles are not as good as the ensemble of randomly constructed codes. However, it is computationally feasible to eciently decode long block length LDPC codes which is not the case for the class of randomly constructed codes. A linear increase in distance is a remarkable property and does not hold for many classes of codes that are useful in practice. For a code dened by means of a parity-check matrix H it is possible to nd the generator matrix G by means of Gaussian elimination. Encoding can then be carried out as usual by means of matrix multiplication, which has complexity, O (N 2 ), i.e., quadratic in the block length. In [21], a graph based encoding procedure exploiting the sparse parity-check matrix of LDPC block codes was described. In this case encoding complexity can be reduced to O (N + g 2 ) where g is typically much smaller than N .

2.2 Iterative decoding techniques One of the most attractive features of LDPC codes is that they allow for ecient iterative decoding. There are several algorithms known for iterative decoding of LDPC codes; Gallagers bit ipping algorithm [12], the belief propagation (BP)

20

decoder, the min-sum decoder etc. Most of these decoding techniques can be described as message passing algorithms. The operation of a message passing algorithm can be conveniently described on the Tanner graph of a code. In a message passing algorithm, messages are exchanged between nodes along the edges of the Tanner graph and nodes process the incoming messages received via their adjacent edges1 to determine the outgoing messages. A message along an edge represents the estimate of the bit represented by the symbol node associated with the edge. The message can be in the form of a hard decision, i.e., 0 or 1, a probability vector [p0 , p1 ], where pi is the probability of the bit taking the value i, a log likelihood ratio (LLR) log(p0 /p1 ) etc. An iteration of message passing consists of a cycle of information passing and processing. In a message passing algorithm, the outgoing message is calculated based on the incoming messages along the other edges. The exact fashion in which the outgoing message is calculated depends on the message passing algorithm being used. The computations in a message passing decoder are localized, e.g., computations at a constraint node are performed independent of the overall structure of the code. This implies that highly parallel implementations of a message passing decoder are feasible. Further, the computations are distributed, i.e., computations are performed by all nodes in the graph. The number of messages exchanged in an iteration of message passing is dependent on the number of edges in the Tanner graph; a sparser Tanner graph means lesser computations. Thus, it is computationally feasible to use message passing algorithms for decoding long block length LDPC codes. The most widely used message passing algorithm is the belief propagation (BP) algorithm. The processing at the symbol and constraint nodes for a BP algorithm
1

For symbol nodes we also treat the information from the channel as an incoming message.

21

with LLRs as messages is now described. Let ci denote the LLR of the bit associated with the symbol node i, rij denote the message from symbol node i to constraint node j , and qji the message from constraint node j to symbol node i. Let N (i) denote the set of neighbors of the node i and N (i)\j the set of neighbors of node i excluding j . Then the outgoing messages calculated at a constraint node j and a symbol node i are 1+ 1
i N (j )\i i N (j )\i

qji = log rij = ci +

tanh(ri j ) tanh(ri j )

, i N (j ),

(2.1) (2.2)

j N (i)\j

qj i , j N (i),

respectively. An iteration of decoding involves the above computations at all the constraint and symbol nodes, respectively, of the Tanner graph. For a BP decoder with a maximum of I iterations the process is repeated I times. At the start of decoding the outgoing messages from all symbol nodes are initialized to the LLR of the corresponding bit. When the I decoding iterations are complete a nal LLR, ci +
j N (i) qji ,

is computed and used for making a hard decision on the bit

corresponding to symbol node i. The BP decoder is known to be optimal for Tanner graphs with no closed paths. However, almost all interesting LDPC Tanner graphs have closed paths or cycles. Inspite of this, the BP decoder has been observed to work remarkably well in these cases. Low-complexity versions of the BP algorithm that provide good complexity to performance tradeos also exist. One common modication is to simplify the computations at the constraint nodes (2.1). Another possibility is to quantize the LLRs. Other examples of message passing are presented in [16].

22

Gallager in his thesis [12] analyzed an iterative hard-decision decoding algorithm to upper bound the bit error probability for regular LDPC codes on a binary symmetric channel (BSC)2 . This analysis was extended to a broader class of channels and message passing algorithms in [16][17][22]. In particular, density evolution [16] makes analysis of message passing algorithms with messages belonging to a continuous alphabet, as for example in the case of the BP algorithm, feasible. All known techniques for the analysis of message passing algorithms require that the messages exchanged during iterations be independent. It is possible to show the existence of a sequence of codes with increasing block length for which this holds [12][16]. Thus, the analysis is asymptotic in the block length. In the case of nite block lengths, where arbitrary number of independent iterations is not possible, little is known about the performance of message passing algorithms. The concept of a threshold or iterative decoding limit was dened in [16]. The threshold depends on the particular class of codes, the iterative decoding algorithm, and the channel3 . It is the largest channel parameter for which the probability of an incorrect message being exchanged, when decoding a code from the specied class with the given decoding algorithm, approaches zero as the number of iterations tends to innity. In [23], it was observed that LDPC codes with irregular Tanner graph4 structures lead to signicantly better performance on the BEC compared to regular LDPC codes. Following this idea, in [17] density evolution was used to optimize the degree prole of irregular LDPCs and achieve thresholds close to capacity on a variety of channels. It should be noted however that the thresholds, especially
A BSC has two inputs and two outputs identical to the two inputs. The probability that an error occurs is the same for both inputs. 3 Thresholds are known to exist if the channel is physically degraded and the decoding algorithm respects this ordering [16]. 4 A Tanner graph which is not regular is irregular.
2

23

for irregular LDPCs, are channel parameters below which the bit error probability and not necessarily the block error probability can be made arbitrarily small.

2.3 LDPC convolutional codes In this section we turn our attention to LDPC convolutional codes, rst constructed in [24]. We begin with the denition of the syndrome former of a convolutional code. A rate R = b/c convolutional code can be dened as the set of sequences
T v = (. . . , v 0 , v 1 , . . . , v t , . . . ), v t Fc 2 , satisfying the equality vH = 0, where

T T H 0 (0) . . . H m (ms ) s . . T . . . H = . . T HT ( t ) . . . H ( t + m ) s 0 ms .. .. . .

..

..

(2.3)

The innite transposed parity-check matrix H T is also called the syndrome former. The sub-matrices H T i (t + i), i = 1, . . . , ms , are c (c b) binary matrices. The largest i such that H T i (t + i) is a non-zero matrix for some t is called the syndrome former memory ms . As in the case of block codes, for a generator matrix G of the convolutional code we have GH T = 0. The constraint imposed by the syndrome former can be written as
T T vtH T 0 (t) + v t1 H 1 (t) + + v tms H ms (t) = 0, t Z.

(2.4)

24

As we shall see in the next chapter, (2.4) can be used for encoding a convolutional code. For time invariant convolutional codes, the sub-matrices H T i (t+i), i = 1, . . . , ms , are the same for all t. For example, the convolutional code in Example 1.3.2 is dened by the syndrome former with ms = 2, and H 0 (t) = (11), H 1 (t) = (10), and H 2 (t) = (10) for all t. In the time invariant case it is possible to express the syndrome former in terms of the delay operator. The syndrome former for the code of Example 1.3.2 in terms of the delay operator is H (D )T = [1 + D + D 2 , 1]T . Note that we have G(D )H (D )T = 0. Analogous to LDPC block codes, LDPC convolutional codes are convolutional codes dened by sparse parity-check matrices or equivalently sparse syndrome former matrices. Corresponding to denition 2.1.1 of the previous section we have Denition 2.3.1 A regular LDPC convolutional (ms , J, K ) code is a code dened by a syndrome former H T , having exactly J ones in each row, where J << (c b)ms , and K ones in each column. For completely specifying an LDPC convolutional code it is also necessary to dene b and c. With J and K xed, the syndrome former of a convolutional code can be made sparse by increasing one or both of c b and ms . In this work we consider two cases; small xed c b with large ms , and large c b with small xed ms . In [24], the construction of periodically time varying convolutional codes from LDPC block codes was proposed. An example of this construction is given below. Example 2.3.1 A b = 1, c = 2, LDPC convolutional (4, 3, 6) code [24] We start with a 8 4 transposed parity-check matrix of an LDPC (2, 4) block code. This matrix is cut below the diagonal (in terms of c (c b), i.e., 2 1, 25

1 0 0 1 1 0 0 1

0 1 1 0 1 0 1 0

0 1 0 1 0 1 0 1

1 0 1 0 0 1 1 0

1 0 0 1 0 1 1 0 1 0 1 0 1 0 0 0 1 1 1 0 . . . 1 1 1 0 0 1 1 0 1 1 0 1 0 1 1 0 1 0 1 1 1 1 0 1 1

0 1 1 0 0 1 (b)

1 0 1 0 0 1

(a)

PSfrag replacements

1 1 0 1 0 1 1 1 1 0 1 1

0 1 0 1 0 1 1 1

1 0 1 0 0 1 1 0

0 1 1 0 0 1

1 0 1 0 0 1

0 1 0 1 0 1 0 1 1 1

1 0 1 0 0 1 1 0 1 1

0 1 1 0 0 1 1 0 1

1 0 1 0 0 1 1 . . .

0 1 0 1 1 0 0 1 0

(c)

(d)

Figure 2.2. Illustration of the J imenez-Zigangirov method to construct the syndrome former of a b = 1, c = 2 LDPC convolutional (4, 3, 6) code

26

sub-matrices) as illustrated in Fig. 2.3(a). Then the lower part is moved to the right as shown in Fig. 2.3(b). Then, xed c (c b) sub-matrices are appended from the left (Fig. 2.3(c)) to ensure that the matrices H 0 (t)T have full rank. The resulting matrix represents one period and is repeated indenitely (Fig. 2.3(d)), leading to a syndrome former H T of memory ms = 4, and J = 3 ones in each row and K = 6 ones in each column. 2

We now turn our attention to the Tanner graph representation of convolutional codes. The Tanner graph of a convolutional code can be obtained as in the case of block codes. There are symbol nodes corresponding to each row of the syndrome former H T and constraint nodes corresponding to each column of H T . There is an edge connecting a symbol node to a constraint node if the entry in the corresponding row and column of H T is a one. As earlier, there is a one to one correspondence between a syndrome former and its Tanner graph representation. The features unique to convolutional codes make the Tanner graph of a convolutional code dierent in several important ways from that of a block code. We illustrate these dierences by means of an example. Example 2.3.2 Tanner graph representation of a b = 1, c = 2, LDPC convolutional (3, 2, 3) code Consider the time invariant convolutional code with syndrome former matrix 1 D 2 H T (D ) = D D D3 1 the Tanner graph of this convolutional code. 27
3

(2.5)

It is easy to see that this code has ms = 3, J = 2, and K = 3. Fig. 2.3.2 shows

ms

PSfrag replacements

t+1

t+2

t+3

Figure 2.3. The Tanner graph for the convolutional code of Example 2.3.2.

The rst dierence between the Tanner graph of a convolutional code from that of a block code comes from the fact that the convolutional code is innite so that the corresponding Tanner graph is also innite. The code symbols of a convolutional code have a time index associated with them. The convolutional code of this example is a b = 1, c = 3, convolutional code. The symbol nodes corresponding to the c = 3 symbols generated at each time t are shown vertically aligned in Fig. 2.3.2. Similarly, at each time t there are c b = 2 parity-check equations on symbols, the corresponding constraint nodes are shown vertically aligned in Fig. 2.3.2. The indexing of the Tanner graph by time is absent in the case of block codes. The memory of a convolutional code restricts the local structure in the Tanner graph. Specically, the memory determines the maximum separation between two symbol nodes connected by a single constraint node. Two pairs of symbol nodes ms = 3 time units apart (time indices t and t + 3) are shown highlighted in Fig. 2.3.2. On the other hand, in the Tanner graph of a block code two symbol

28

nodes connected by a constraint node can be arbitrarily separated. Note also that the causality of the convolutional code is reected by all symbols being connected only to constraint nodes either at the same or later time instants. Finally, since the code is time invariant the connectivity of nodes in Tanner graph is the same for all t. 2

Typically LDPC convolutional codes have large ms (or c b) so that decoding based on their trellis representation is no longer feasible. However, LDPC convolutional codes by virtue of their sparse syndrome formers have sparse Tanner graph representations. Thus, they can be decoded by means of message passing algorithms. The special features of the Tanner graph structure of a convolutional code pointed out here, as we shall see in Chapter 5, lead to a highly parallel continuous sliding window decoder for LDPC convolutional codes. Further, in Chapter 7 we explain how the structure of the convolutional code Tanner graphs result in better thresholds for LDPC convolutional codes compared to corresponding LDPC block codes.

29

CHAPTER 3 LDPC CONVOLUTIONAL CODES CONSTRUCTED FROM QUASI-CYCLIC CODES

The construction of periodically time varying LDPC convolutional codes has been described in [24]. One big diculty in implementing time varying LDPC convolutional codes is the huge storage requirements for storing a period of the parity-check matrix. For time invariant convolutional codes the connectivity of nodes in the Tanner graph structure at dierent times is identical. This leads to signicantly reduced storage requirements for implementing these codes. In this chapter we describe the construction of time invariant LDPC convolutional codes from algebraically constructed quasi-cyclic (QC) LDPC codes. Section 3.1 begins with a description of the construction technique for the QC LDPC codes. The QC codes are constructed by generalizing the code construction technique of a (155, 64) QC LDPC code by Tanner [25]. This is followed by a description of the design technique for constructing time invariant convolutional codes from the quasi-cyclic codes. Section 3.2 details the properties of the constructed codes - both quasi-cyclic and convolutional, and the corresponding Tanner graphs. The relationship between the QC and corresponding convolutional codes, girth and distance properties, and syndrome former based encoding of the convolutional code are described. The chapter concludes with performance results of the codes with iterative decoding. 30

3.1 Code construction This section describes the means by which the underlying structure of multiplicative groups in the set of integers modulo m may be used to construct lowdensity parity-check codes both block codes and convolutional codes.

3.1.1 Construction of quasi-cyclic LDPC block codes We use the structure of multiplicative groups in the set of integers modulo m to place circulant matrices within a parity-check matrix so as to form regular quasicyclic LDPC block codes with a variety of block lengths and rates. For prime m, the integers {0, 1, . . . , m1} form a eld under addition and multiplication modulo m i.e., the Galois eld GF (m). The non-zero elements of GF (m) form a cyclic multiplicative group. Let a and b be two non-zero elements with multiplicative orders o(a) = K and o(b) = J respectively1 . Then we form the J K matrix P of elements from GF (m) that has as its (s, t)th element P s,t = bs at as follows: 1 a a b ab a2 b P = ... ... ... bJ 1 abJ 1 a2 bJ 1 (Here, 0 s J 1 and 0 t K 1.) The LDPC code is constructed by specifying its parity-check matrix H . Specifically, H is made up of a J K array of circulant sub-matrices as shown below:
There exists such elements if K and J divide (m) = m 1, the order of the multiplicative group.
1

... ... ...

a a

K 1

K 1

...

. . . aK 1 bJ 1

(3.1)

31

I1 Ia I a2 Ib I ab I a2 b H= ... ... ... I bJ 1 I abJ 1 I a2 bJ 1

... ... ...

I aK 1 I aK 1 b ...

. . . I aK 1 bJ 1

(3.2)

where I x is an m m identity matrix with rows cyclically shifted to the left by x 1 positions. The circulant sub-matrix in position (s, t) within H is obtained by cyclically shifting the rows of the identity matrix to the left by P s,t 1 places. The resulting binary parity-check matrix is of size Jm Km, which means the associated code has a rate R 1(J/K ). (The rate may be greater than 1(J/K ) due to linear dependence among the rows of H ; it is easy to see that there are in fact at least J 1 dependent rows in H .) By construction, every column of H contains J ones and every row contains K ones, and so H represents a (J, K ) regular LDPC code. (We observe here that for the case J = 2, our construction yields the graph-theoretic error-correcting codes proposed by Hakimi et al. in [26].) The codes constructed using this technique are quasi-cyclic with period K i.e., cyclically shifting a codeword by one position within each of the K blocks of circulant sub-matrices (each block consists of m code bits) results in another codeword2 . As described in [27] the construction can also be extended to nonStrictly speaking, the word quasi-cyclic means the code has the property that when a codeword is cyclically shifted by K positions another codeword is obtained; to observe this property in the codes constructed above, the bit positions in each codeword must be permuted to a dierent order than the one indicated by the construction.
2

32

prime m. Examples of LDPC quasi-cyclic codes constructed in this manner from a prime m are shown in Table 3.1.1 [28]. Example 3.1.1 A [155,64,20] QC code (m = 31) [25] Elements a = 2, b = 5 are chosen from GF (31); then o(a) = 5, o(b) = 3, and the parity-check matrix is given by I 1 I 2 I 4 I 8 I 16 H= I 5 I 10 I 20 I 9 I 18 I 25 I 19 I 7 I 14 I 28

,
(93155)

where I x is a 31 31 identity matrix with rows shifted cyclically to the left by x 1 positions. The parity-check matrix H describes a (3, 5) regular LDPC code and has rank 91 (determined using Gaussian elimination), so that the corresponding code has rate R = 64/155 .4129 code. The Tanner graph resulting from H is shown in Figure 3.1. The length of the shortest cycle (i.e., the girth) of the Tanner graph is eight. The sparse Tanner graph along with the large girth for a code of this size makes the code ideal for graph-based message-passing decoding. The associated code has minimum distance dmin = 20 (determined using MAGMA) that compares well with the minimum distance dmin = 28 of the best linear code known with the same rate and block length. It can be shown that the Tanner graph of this code has diameter six, which is the best possible for a (3, 5) regular bipartite graph of this size. Also, the Tanner graph has girth eight, while an upper bound on girth is 10 (from the tree bound; see Section 3.2). Example 3.1.2 A [21,8,6] QC code (m = 7). 2

33

TABLE 3.1 EXAMPLES OF CODES CONSTRUCTED FROM (PRIME) CIRCULANT SIZES. Block length N 21 129 155 186 305 755 905 1055 1205 1477 1477 1928 1928 2041 2248 2947 2947 3641 3641 5219 Design Parameters J 2 2 3 5 3 3 3 3 3 3 5 3 5 3 5 3 4 3 5 3 K 3 3 5 6 5 5 5 5 5 7 7 8 8 13 8 7 7 11 11 17 Design Rate rd 1/3 = 0.3333 1/3 = 0.3333 2/5 = 0.4000 1/6 = 0.1667 2/5 = 0.4000 2/5 = 0.4000 2/5 = 0.4000 2/5 = 0.4000 2/5 = 0.4000 4/7 = 0.5714 2/7 = 0.2857 5/8=0.6250 3/8=0.3750 10/13 = 0.7692 3/8=0.3750 4/7 = 0.5714 3/7 = 0.4286 8/11 = 0.7272 6/11 = 0.5454 14/17 = 0.8235 Actual Rate r 0.3809 0.3411 0.4129 0.1882 0.4065 0.4423 0.4022 0.4018 0.4016 0.5727 0.2884 0.6260 0.3771 0.7702 0.3768 0.5721 0.4296 0.7278 0.5465 0.8239 Circulant Size m 7 43 31 31 61 151 181 211 241 211 211 241 241 157 281 421 421 331 331 307

34

code bit nodes (5 rings of 31 each) constraint nodes (3 rings of 31 each)

Figure 3.1. Tanner graph for a [155,64,20] QC code.

Elements a = 2, b = 6 are chosen from GF (7); then o(a) = 3, o(b) = 2, and the parity-check matrix is given by I1 I2 I4 H= I6 I5 I3
(1421)

where I x is a 7 7 identity matrix with rows cyclically shifted to the left by x 1 positions. The resulting Tanner graph has girth twelve and is shown in Figure 3.2. (Figure 3.2 shows both the ring-like structure characteristic of these constructions and a attened representation that will be useful in what follows.) The associated code has minimum distance dmin = 6 and is a (2, 3) regular LDPC code. The best [21, 8] linear code has dmin = 8. Example 3.1.3 A [5219,4300] QC code (m = 307). Elements a = 9, b = 17 are chosen from GF (307) (note that 307 is a prime); then o(a) = 17, o(b) = 3, and the parity-check matrix is given by 2

35

Figure 3.2. Tanner graph for a [21,8,6] QC code.

H = I 17 I 289

I1

I9 I 153 I 145

I 81 I 149 I 77

I 115 I 113 I 79

I 114 I 96 I 97

I 105 I 250 I 259

I 24 I 101 I 182

I 216 I 295 I 103

I 102 I 199 I6

I 304 I 256 I 54

I 280 I 155 I 179

I 64 I 167 I 76

I 269 I 275 I 70

I 272 I 19 I 16

I 299 I 171 I 144

I 235 I4 I 68

I 273

I 36 , I 305

where I x is a 307 307 identity matrix with rows cyclically shifted to the left by x 1 positions. H is a 921 5219 matrix and describes a (3, 17) regular LDPC code with minimum distance upper bounded by 24 (see Section 3.2). 2 These examples show that the construction technique described above yields codes with a wide range of rates and block lengths.

3.1.2 Construction of LDPC convolutional codes An LDPC convolutional code can be constructed by replicating the constraint structure of the quasi-cyclic LDPC block code to innity [29]. Naturally, the parity-check matrices (or, equivalently, the associated Tanner graphs) of the convolutional and quasi-cyclic codes form the key link in this construction. Each circulant in the parity-check matrix of a QC block code can be specied by a unique polynomial; the polynomial represents the entries in the rst column

36

of the circulant matrix. For example, a circulant matrix whose rst column is [1 1 1 0 1 0]T is represented by the polynomial 1+ D + D 2 + D 4 . Thus, the Jm Km binary parity-check matrix of a regular LDPC code obtained from the construction described above can be expressed in polynomial form (with indeterminate D ) to obtain the following J K matrix: .

H (D ) =

a1

D D

a2 1

... ... ...

D D

aK 1 1

D b1 ... Db
J 1 1

D ab1 ... D ab
J 1 1

a2 b1

aK 1 b1

... Da
2 bJ 1 1

...
K 1 bJ 1 1

. . . Da

(J K )

Since all the circulant sub-matrices in the parity-check matrix of the QC LDPC code are shifted identity matrices, H (D ) is comprised solely of monomials; the exponent of D indicates how many places the identity matrix was shifted to form the corresponding circulant sub-matrix. H (D ) is the parity-check matrix of a corresponding LDPC convolutional code in polynomial form. (The indeterminate D is now interpreted as the delay operator in the convolutional code.) We note here that the parity-check matrix of the QC code, when written in circulant form, is over the ring F2 [D ]/ D m + 1 , i.e., the polynomial ring F2 [D ] modulo the ideal D m + 1 , whereas the parity-check matrix of the convolutional code is over the rational eld F2 (D ).) In all cases that were examined, the rate of the LDPC convolutional codes obtained from the QC codes was equal to the design rate of the original QC code, i.e., R = 1 (J/K ). This rate is slightly less than the rate of the original QC code. The syndrome former memory of the convolutional code so obtained satises ms m.

37

Example 3.1.4 A rate R = 2/5 LDPC convolutional code. From the [155, 64, 20] QC code in Example 3.1.4, we can obtain a rate R = 2/5 convolutional code with parity-check and generator matrices given by D D D D 1 4 H (D ) = D 9 D 19 D 8 D 17 D D 24 D 18 D 6 D 13 D 27 G(D ) =
a1 (D ) (D ) b1 (D ) (D ) a2 (D ) (D ) b2 (D ) (D ) a3 (D ) (D ) b3 (D (D )

15

(3.3)
(35)

where

1 0 0 1

(25)

(3.4)

a1 (D ) = D 4 (1 + D 7 + D 10 + D 14 + D 18 + D 29 ), a2 (D ) = D 3 (1 + D 3 + D 6 + D 18 + D 21 + D 36 ), a3 (D ) = D 7 (1 + D 4 + D 8 + D 11 + D 15 + D 22 ), b1 (D ) = D 13 (1 + D 6 + D 14 + D 15 + D 23 + D 28 ), b2 (D ) = D 12 (1 + D 2 + D 11 + D 21 + D 23 + D 35 ), b3 (D ) = D 21 (1 + D 3 + D 4 + D 5 + D 10 + D 16 ), (D ) = 1 + D 4 + D 14 + D 25 + D 26 + D 33 . The generator matrix G(D ) was obtained from the parity-check matrix H (D ) using Gaussian- elimination. The syndrome former memory of H (D ) is ms = 28. We conjecture that the above convolutional code has a free distance dfree of 24. By choosing one of the information sequences equal to the denominator polynomial, i.e., 1 + D 4 + D 14 + D 25 + D 26 + D 33 , and the other information sequence as the all zero sequence, we obtain a code sequence of weight 24. This only provides an upper bound on the dfree of this convolutional code, but we have been unable to

38

nd any lower weight code sequences. Interestingly, as we shall see, this is the same as the upper bound obtained by Mackay and Davey [30] for LDPC matrices constructed by using non-overlapping permutation matrices that commute with each other, as is the case here. Further, the results of [29] guarantee that the minimum distance of the QC block code, 20 in this case, provides a lower bound on the free distance of the associated convolutional code. The matrix G(D ) in (3.4) is not in minimal form. The minimal-basic generator matrix [7] has the minimum total memory among all equivalent rational and polynomial generator matrices and is thus of interest. The minimal-basic form of G(D ), which has total memory 23 + 22 = 45, is given by a1 (D ) a2 (D ) a3 (D ) a4 (D ) a5 (D ) Gmin (D ) = b1 (D ) b2 (D ) b3 (D ) b4 (D ) b5 (D ) where a1 (D ) = D 6 + D 10 + D 14 + D 17 + D 18 , a2 (D ) = D 5 + D 8 + D 9 + D 11 + D 13 + D 14 + D 15 + D 19 + D 21 + D 22 + D 23 , a3 (D ) = D 9 , a4 (D ) = D 2 + D 10 + D 13 + D 15 + D 16 + D 18 + D 19 + D 20 + D 21 + D 23 , a5 (D ) = 1 + D + D 3 + D 9 + D 10 + D 11 + D 12 + D 13 + D 15 , b1 (D ) = D 4 + D 5 + D 6 + D 8 + D 10 + D 11 + D 12 + D 13 + D 14 + D 16 + D 17 , b2 (D ) = D 3 +D 4 +D 5 +D 6 +D 8 +D 11 +D 12 +D 13 +D 14 +D 16 +D 17 +D 18 +D 19 +D 20 , b3 (D ) = D 7 + D 8 + D 9 + D 12 , b4 (D ) = 1+ D + D 2 + D 5 + D 8 + D 9 + D 10 + D 11 + D 13 + D 14 + D 15 + D 16 + D 21 + D 22 , b5 (D ) = 1 + D 2 + D 4 + D 7 + D 8 + D 13 + D 14 . 2
(25)

(3.5)

39

By pulling out common factors from the matrix H (D ) in (3.3) we obtain the matrix D D D D 1 H equiv (D ) = D 5 D 15 D 4 D 3 1 D 18 D 12 1 D 7 D 21
3 7 15

.
(35)

(3.6)

It is easy to see that the matrices H (D ) and H equiv (D ) describe the same code. However, H equiv (D ) has syndrome former memory, ms = 21 as against ms = 28 for H (D ). Several dierent LDPC convolutional codes can be obtained from the same QC LDPC block code. For example, reordering the rows of the parity-check matrix of the quasi-cyclic code within each block of circulant sub-matrices will leave it unchanged but can lead to a completely dierent LDPC convolutional code3 . Consider once again the [155, 64, 20] QC code. Cyclically shift the rst block of m = 31 rows above by one position, the middle block of 31 rows by ve positions, and the last block of 31 rows by 25 positions, so that now the rst row in each block has a one in the rst column. The resulting LDPC matrix is given by I 1 I 2 I 4 I 8 I 16 H= I 1 I 6 I 16 I 5 I 14 I 1 I 26 I 14 I 21 I 4

,
(93155)

where again I x is a 31 31 identity matrix with rows cyclically shifted to the left by x 1 positions. Clearly, the QC block code and its associated Tanner graph
Here we are only interested in those convolutional codes that have the same graph connectivity (i.e., nodes of the same degrees) as the base QC block code. For example, equivalent representations for the QC block code can be obtained by suitable linear combinations of rows of H . However, in such a case, the new representation(s) of the block code and that of the corresponding convolutional code will not in general have the same node degrees as the original representation of the QC code.
3

40

are unaected by these row shifts. However, the convolutional code obtained by following the above procedure has the parity-check matrix D D 1 D D 5 15 H 1 (D ) = D 4 D 13 1 D D 1 D 25 D 13 D 20 D 3
3 7 15

.
(35)

Looking at the third constraint equation of H (D ) and H 1 (D ), we see that that the two codes are in fact dierent. (Note that the rst and second constraint equations of H (D ) and H 1 (D ) are equivalent, since the rst and second rows of H (D ) are just delayed versions of the rst and second rows of H 1 (D ) respectively.) Example 5: A rate 1/3 LDPC convolutional code. From the QC code of Example 2, a rate 1/3 convolutional code with the following parity-check and generator matrices is obtained 1 D D H (D ) = D5 D4 D2 G(D ) =
D +D 3 1+D 2 +D 4

(23)

1 D D D3 D2 1
D2 1+D 2 +D 4

(23)

,
(13)

In this case, the syndrome former memory4 is ms = 3 and the minimal basic generator matrix has total memory four. The convolutional code has a free distance dfree = 6, equal to the minimum distance of the original QC code. In this manner, it is possible to construct numerous convolutional codes with a sparse Tanner graph representation. For example, for every entry in Table 1,
4

The parity-check matrix with common factors removed from the second row.

41

it is possible to construct a convolutional code with an actual rate equal to the design rate indicated in the table.

3.2 Properties of constructed codes This section describes the properties of the codes constructed in the previous section. Specically, the structure of the Tanner graphs, the minimum distance of the codes, and encoding techniques are described.

3.2.1 Relation between the block and convolutional Tanner graphs The Tanner graphs for the rate 1/3 convolutional code in Example 5 and the corresponding QC code (Example 2) are shown in Figure 3.3. We observe that the Tanner graph of the convolutional code is strikingly similar to that of the QC code. The convolutional code Tanner graph can be viewed as being obtained by unwrapping that of the QC code. The construction technique described in the previous section thus unwraps the Tanner graph of the QC code to obtain the convolutional code.

Quasi-cyclic code

Convolutional code

Figure 3.3. Tanner graph for the rate 1/3 convolutional code of Example 5.

42

3.2.2 Quasi-cyclic block codes viewed as tail-biting convolutional codes Tail biting is a technique by which a convolutional code can be used to construct a block code without any loss of rate [31][32]. An encoder for the tail-biting convolutional code is obtained by reducing each polynomial entry in the generator matrix of the convolutional code modulo D m + 1 for some positive integer m and replacing each of the entries so obtained with circulant matrices of size m m. The block length of the block code5 so derived depends on m. Since the generator matrix consists of circulant sub-matrices, the tail-biting convolutional code obtained is a QC block code. However, this tail-biting code has a rate equal to that of the convolutional code which is less than or equal to the rate of the original QC code. As expected, the two QC codes, i.e., the original QC code and the tail-biting convolutional code, are closely related, as the following theorem shows. Theorem 3.2.1 Let C be a length Km QC code with the Jm Km parity check matrix H , where H is composed of m m circulants (i.e., its period is K ). Let C be a convolutional code obtained by unwrapping H . Then the QC block code (tail-biting convolutional code) C of length Km constructed from C is a sub-code of C . Proof: Since the tail-biting code C is quasi-cyclic with period K , any codeword in C can be described by a set of polynomials in the ring F2 [D ]/ D m + 1 . Therefore any codeword in C is of the form a(D )G(D ) mod D m + 1 , where G(D ) is the generator matrix of the convolutional code C and a(D ) is an information polynomial. Now, we know that the polynomial generator matrix of the convolutional
With a feedforward encoder for the convolutional code, a tail-biting code of any block length may be obtained [6].
5

43

code C satises the parity constraints imposed by its parity-check matrix H (D ), i.e., G(D )H T (D ) = 0. Therefore, G(D )H T (D ) 0 mod D m + 1 . Since, by construction, H (D ) is also the parity-check matrix of the original QC block code C , with the entries in H (D ) now interpreted as being in the ring F2 [D ]/ D m + 1 , any codeword in C satises the constraints of the original QC code C , i.e., the tail-biting code C is a sub-code of C . 2

If there were no rank reduction in the parity-check matrix of the original QC block code, it would have a rate equal to that of the convolutional code. In such a case, it is easy to see that the tail-biting convolutional code would be exactly the same as the original QC block code. We have derived a rate 2/5 convolutional code (Example 4) from the [155,64,20] QC block code (Example 1); from the rate 2/5 convolutional code we can in turn derive a tail-biting convolutional code, i.e., another QC block code of block length 155 and 62 information bits (rate 2/5), that is a sub-code of the original [155,64,20] QC block code and has dmin 20. 3.2.3 Girth The Tanner graph representing the parity-check matrix H in equation (3.2) cannot have girth less than six6 . This is seen by observing that the relative dierence between the shifts across any two columns is dierent along dierent rows. However, the Tanner graph cannot have girth larger than twelve. Figure 3.4 is a graphical illustration of why this is so (see also [33]). The dark diagonal lines in the gure represent the non-zero entries of the parity-check matrix and the empty regions represent the zeros. The dotted lines indicate a cycle of length twelve in the Tanner graph involving bit nodes b1 through b6 and constraint
6

This holds for prime circulant sizes m but is not necessarily true for non-prime m.

44

b1 b2

b3

b4

b5 b6

p1 p2 p3 +X X p4 p5 p6 Y +Y +Z Z

Figure 3.4. A 12-cycle.

nodes p1 through p6. The structure of the circulant sub-matrices in H gives rise to numerous 12-cycles. The Tanner graphs of the LDPC convolutional codes constructed herein have their girth lower bounded by the girth of the corresponding QC LDPC Tanner graph. For any cycle in the convolutional code Tanner graph we can nd an equivalent cycle in the QC Tanner graph. Say a particular set of bit and constraint nodes form a cycle in the convolutional code Tanner graph. Then the relative shifts between the bit nodes sum to zero. The corresponding sequence of bit and constraint nodes in the QC code (obtained by reducing indices modulo m, where m is the circulant size in the QC code) have exactly the same relative shifts (now read modulo m) between the bit nodes, and hence sum to zero i.e., in the QC code Tanner graph, we nd a corresponding cycle of the same length. However, a cycle in the QC Tanner graph does not always lead to a cycle in the convolutional Tanner graph. While the relative shifts between bit nodes may

45

sum to zero modulo m, the shift may be a non-zero multiple of m, meaning the corresponding bit nodes in the convolutional code do not form a cycle. Hence, it is possible that the convolutional code Tanner graph may have a larger girth than the QC code Tanner graph. Observe, however, that in Figure 3.4 the relative shifts between bit nodes sum to zero, so that the corresponding bit nodes in the convolutional code also form a 12 cycle. Hence, the girth of the convolutional codes is also upper bounded by 12.

3.2.4 Minimum distance At high SNRs, the maximum likelihood decoding performance of an errorcorrecting code is dominated by the codes minimum distance. Mackay and Davey obtained an upper bound on the minimum distance of certain LDPC codes whose parity-check matrices are composed of non-overlapping blocks of permutation submatrices that all commute with each other [30]. The regular LDPC codes constructed here satisfy this condition, so the upper bound in [30] on the minimum distance is applicable. The result in [30] says that if a parity-check matrix H M N is composed of an M (J + 1)M/J array of J (J + 1) non-overlapping permutation sub-matrices that all commute with each other, then the minimum distance of the corresponding code is at most (J + 1)!. For small column weight J , the bound imposes a stringent limitation on the codes minimum distance, especially for codes with large block lengths. In particular, the regular LDPC codes described here are not asymptotically good in contrast to the ensemble of randomly constructed regular LDPC codes of Gallager. Based on the results in [29] it follows that the LDPC convolutional codes obtained by unwrapping the Tanner graph of the QC codes have their free distance

46

dfree lower bounded by the minimum distance of the corresponding QC code. Essentially, this is because wrapping back any codeword of the convolutional code produces a valid codeword in the QC code. More succinctly, a codeword in the convolutional code reduced modulo D m + 1 is a codeword in the QC code, as shown for the tail-biting codes in Theorem 3.2.1. However, as will be seen in the next chapter, using arguments similar to that used by Mackay and Davey, an upper bound on the free distance independent of the constraint length of the LDPC convolutional codes can be obtained.

3.2.5 Encoding The QC codes can be encoded by rst determining the generator matrix from its parity-check matrix by means of Gaussian elimination. Other possible encoding strategies which exploit the structure of the LDPC QC codes are described in [27]. Similarly, for the convolutional codes a generator matrix G(D ) can be obtained by Gaussian elimination of H (D ) (since H (D ) corresponds to a c b = J and c = K convolutional code this is quite simple). However, it is also possible to directly use the parity-check matrix to encode the convolutional codes [24]. As described in Chapter 2 a convolutional code dened by a syndrome former with memory ms must satisfy
T T vtH T 0 (t) + v t1 H 1 (t) + + v tms H ms (t) = 0, t Z.

(3.7)

Equation (3.7) can be used to obtain an encoder for the LDPC convolutional code. Consider the rate R = 1/3, J = 2, K = 3, convolutional code of Example 5 with the second and third rows of the syndrome former matrix interchanged for

47

PSfrag replacements

vt , vt , vt ut = v t
(2)

(2)

(1)

(0)

vt1 vt1 vt1

(0)

(1)

(2)

vt2 vt2 vt2

(0)

(1)

(2)

vt3 vt3 vt3

(0)

(1)

(2)

Figure 3.5. A systematic syndrome former based encodes for the R = 1/3 code of Example 5

convenience. The modied syndrome former matrix H T (D ) can be rewritten as 1 D 1 0 0 1 0 0 0 0 = 0 1 + D 0 0 + D2 0 0 + D3 1 0 . 3 H T (D ) = D 1 0 0 0 0 0 1 1 0 D D2 Let v t = (vt , vt , vt ) represent the three codebits at time t in a code sequence. Then it follows that 0 1 0 0 0 0 1 0 vt 0 1 + v t1 0 0 + v t2 0 0 + v t3 1 0 = 032 . 0 0 0 1 1 0 0 0 Now, if we let v t
(2) (0) (1) (2)

(3.8)

correspond to an information bit ut then the corresponding

parity-bits can be determined by solving (3.8). The 2 2 identity sub-matrix corresponding to v t in (3.8) implies that the encoding can be easily implemented as Figure 3.5 shows. The encoder of Figure 3.5 is systematic and the number of taps for each parity-bit equals K 1 = 2.

48

For a (J, K ) regular LDPC convolutional code the syndrome former based encoder has complexity O (K 1) independent of the syndrome former memory ms . In general, the generator matrix of a LDPC convolutional code would not be sparse and the encoding complexity would be of the order of the syndrome former memory. Further, as long as the matrix H t (0) has full rank a systematic encoder based on the syndrome former matrix can be obtained, irrespective of whether the code is time-invariant or not.

3.3 Performance results The LDPC convolutional codes of Section 2 are iteratively decoded using the BP decoder. The implementation of the decoder is described in Chapter 5. In this section the performance obtained with a few of the constructed convolutional codes with continuous transmission on the additive white Gaussian noise (AWGN) channel are presented. Figure 3.6 shows the performance of the LDPC convolutional codes and corresponding QC block codes with J = 3 and K = 5 for 50 BP iterations. We see that the LDPC convolutional codes7 signicantly outperform their block code counterparts in the waterfall region [34]. At higher SNRs the convolutional codes exhibit an error oor and the improvement obtained with respect to the block code is reduced. (Occasionally, with the (3, 5) convolutional LDPCs, the BP algorithm needed 200-300 iterations to converge. We found that using an ad hoc limiting factor on the reliabilities passed in BP decoding insured faster convergence, i.e., within 50 iterations).
The syndrome former memory is that of the parity-check matrix with common factors removed from each row.
7

49

10

[1055,424] QC code R = 2/5 conv code, ms=187 10

1

[2105,844] QC code. R = 2/5 conv code, ms=389 (50 BP iterations)

10
bit error rate

10

10

10

Threshold limit for (3,5) LDPCs

10

0.5

0.965

1.5 Eb/No (dB)

2.5

Figure 3.6. The performance of convolutional LDPC codes versus the corresponding QC block LDPC codes for (3,5) connectivity.

50

Figure 3.7 shows the performance of the LDPC convolutional codes and corresponding QC block codes with J = 5 and K = 7 . Again, the convolutional codes outperform the block codes. In this case, though, there is no apparent error oor exhibited by the convolutional codes. Observe that the waterfall for the ms = 966 convolutional code occurs at a lower SNR than the threshold for (5, 7) regular LDPC block codes. This, as shall be seen in Chapter 7, may be because the initial constraint nodes of the convolutional code have degree less than K . Figure 3.8 shows the performance of a rate 0.8239, block length 5219, (3, 17) regular QC LDPC code (Example 4) and that of the corresponding rate 14/17 LDPC convolutional code derived from this QC code. Once again the convolutional code performance surpasses that of the original QC code. The convolutional code can be viewed as being unwrapped by extending the size of each circulant block in the parity-check matrix of the QC block code to innity. This unwrapping is the principal reason for the performance gain of the convolutional code over the QC block code. To illustrate this, we form a class of block codes with parity-check matrices of increasing circulant sizes, while retaining the same structure within each circulant. As an example, consider the block length N = 1055 (3, 5) regular QC code. The circulant sub-matrices are of size m = 211 in this case. Increasing the circulant size m to 422 and 844 gives QC codes of block lengths N = 2110 and N = 4220, respectively. The performance of these block codes with BP decoding is shown in Figure 3.9. The gure also shows the performance of the corresponding convolutional code. The number of BP iterations is xed at 50 in all cases8 . The gradual improvement in performance
Since we are limiting the number of iterations for the convolutional code to 50 and m s in this case is 187, this implies that BP decoding for the convolutional code occurs across nodes up to 50 (187 + 1) time units away. So its behavior is expected to be equivalent to that of a block code with a circulant size of m = 9400.
8

51

10

[1477,426] QC code R = 2/7 conv code, m =198 s 10

1

[7357,2106] QC code R = 2/7 conv code, ms=966 (50 BP iterations)

10

bit error rate

10

10

10

10

Threshold limit for (5,7) LDPCs

2.2

2.4

2.6 2.65 2.8

3.2 Eb/No (dB)

3.4

3.6

3.8

Figure 3.7. The performance of convolutional LDPC codes versus the corresponding QC block LDPC codes for (5,7) connectivity.

52

10

[5219,4300] QC code =303 R=14/17 conv. code, m s 10

2

(100 BP iterations)

10

bit error rate

10

10

10

Threshold limit for (3,17) LDPCs

10

2.5

2.813 Eb/No (dB)

3.5

Figure 3.8. The performance of convolutional LDPC codes versus the corresponding QC block LDPC codes for (3,17) connectivity.

of the block codes illustrates that the convolutional code represents the limiting case of the block codes as the circulant size increases.

53

10

10

m=211, R=0.4019 m=422, R=0.4010 m=844, R=0.4005 m=9400, R=0.4000 (50 BP iterations)

10

bit error rate

10

10

10

10

0.5

1.5 Eb/No (dB)

2.5

Figure 3.9. Performance of regular (3, 5) QC block codes with increasing circulant size m.

54

CHAPTER 4 A CONSTRUCTION FOR TIME INVARIANT LDPC CONVOLUTIONAL CODES

In the previous chapter a technique for obtaining time invariant regular LDPC convolutional codes was described. We now present a technique to construct a broader class of time invariant LDPC convolutional codes. The construction method can be used to obtain both regular and irregular LDPC convolutional codes. A similar approach to construct QC LDPC codes has been proposed in [35]. The construction of LDPC codes from protographs [36] is also closely related to the design technique presented in the chapter. The two step construction technique is described in the next section. We then indicate how LDPC QC codes can be constructed from the LDPC convolutional codes. This is exactly the reverse of the construction technique described in the previous chapter. In Section 4.2, design of the constructed LDPC convolutional codes so as to make them suited for decoding with message passing algorithms is discussed. In particular, we determine the graph structure of the codes so as to optimize thresholds with BP decoding over the AWGN channel. Finally, an upper bound on the free distance of time invariant LDPC convolutional codes is presented. This bound shows that time invariant LDPC convolutional codes are not asymptotically good.

55

4.1 Code construction technique The construction technique involves the polynomial extension of a base matrix to obtain LDPC convolutional codes. In the rst step, a (c b) c matrix B of non-negative integers is constructed. The parity-check matrix H (D ) (also a (c b) c matrix) of the LDPC convolutional code is obtained by replacing each entry of B with a polynomial in D having weight equal to the corresponding entry in B . A zero in B is replaced by the zero polynomial in H (D ). The degrees of the symbol and constraint nodes in the Tanner graph corresponding to H (D ) are equal to the column and row weights, respectively, of B . B completely determines the possible edge connections of the symbol and constraint nodes of the graph, and is hence referred to as the base matrix. Thus, the base matrix denes an ensemble of codes with a particular degree distribution and graph connectivity. Example 4.1.1 A (3, 6) LDPC convolutional code An ensemble of (3, 6) LDPC convolutional codes with b = 3 and c = 6 can be obtained by choosing B 36 such that each entry in B is a 1. In this case, the ensemble of time invariant LDPC convolutional codes is dened by paritycheck matrices H (D )36 consisting of monomials. A specic code is obtained by choosing the monomials. The monomials are chosen randomly while still ensuring that the corresponding Tanner graph does not have small cycles. For example the (3, 6) LDPC convolutional code with parity-check matrix D D 1 H (D ) = D 48 D 71 1 D 23 D 151 D 58
37 109

43

89

12

D 203 D 120 D 71 1 D 92 D8

(36)

56

has monomial entries chosen such that the resulting Tanner graph has girth greater than six. This code has rate R = 3/6 and syndrome former memory ms = 203. 2 In general, by choosing the entries of the base matrix B so that: 1. the sum of entries in every row is J 2. the sum of entries of every column is K a (J, K ) regular LDPC convolutional code ensemble can be obtained. Once again, in order to obtain a specic LDPC convolutional code, the polynomials which replace the non-zero entries of the base matrix must be specied. These polynomials are chosen randomly while still avoiding small cycles. Example 4.1.2 A (3, 6) LDPC convolutional code with parity-check matrix consisting of binomials and monomials Suppose B is

It is easy to check that the sum of the entries in every row of B is six and the sum of the entries in every column of B is three. Hence, an ensemble of (3, 6) regular LDPC convolutional codes can be obtained with B as base matrix. A rate R = 3/6 LDPC convolutional code with syndrome former memory ms = 203 and parity-check matrix 1+D 97 H (D ) = D 0
194

2 1 1 1 0 1 B= 1 1 1 1 1 1 0 1 1 1 2 1

.
36

158

166

144

0 D 65

65

D 49 D 106

1 D 83 57

D 203

D 37 1

D 138 D 48 + D 132

(36)

can be obtained from B . Note that there are two binomials in H (D ) corresponding to the 2s in B . The polynomials in H (D ) were chosen so that the girth of the corresponding Tanner graph is greater than six. 2

The construction technique can also be used to obtain irregular LDPC convolutional codes. If the the sum of the entries in every row (or column) of B is not the same then the the Tanner graphs of the codes constructed with B as the base matrix are irregular. Example 4.1.3 An irregular LDPC convolutional code Let B be chosen as 1 1 1 1 0 1 0 0 .

1 1 1 1 1 1 0 1 B= 1 1 1 1 1 0 1 1 1 1 1 0 1 0 1 0

48

B can be used to construct an ensemble of irregular LDPC convolutional codes with b = 4 and c = 8. The Tanner graphs of codes constructed with B as base matrix have degree distribution

4 = 3/8, 2 = 3/8, 3 = 2/8,

5 = 2/4, 7 = 2/4

where i is the fraction of variable nodes with degree i and i is the fraction of constraint nodes with degree i. A rate R = 4/8 time invariant convolutional code

58

with ms = 199 obtained from B is 1 D D D 0 D D 7 D 51 D 160 D 30 D 121 1 H (D ) = 167 D 64 D 56 D 128 D 31 0 D D 96 D 118 D 35 0 D 27 0

180 62 199 40

0
112

0 D 1 1

D 93 0

48

In this case, the polynomials were chosen so that the Tanner graph has girth 8. 2 The time invariant LDPC convolutional codes can be used to obtain LDPC QC codes by reversing the process of the last chapter wherein LDPC convolutional codes were obtained by unwrapping algebraically constructed LDPC QC codes. To obtain a QC code the polynomial entries in H (D ) are interpreted as corresponding to m m circulant sub-matrices, i.e., as elements of the ring F2 [D ]/ D m +1 rather than as elements of the rational eld F2 (D ). The zero polynomial corresponds to the m m zero matrix in the parity-check matrix of the QC code. If H (D ) consists only of monomials then as in Chapter 3 the parity-check matrix of the QC code consists of circularly shifted identity matrices. Observe that by varying the circulant size m QC codes of varying block lengths can be obtained from a single convolutional code. The LDPC convolutional codes constructed here can be encoded using their parity-check matrices as described in the previous chapter and decoding can be accomplished by means of message passing algorithms. In Fig 4.1, the performance obtained with the convolutional codes in Examples 4.1.1-4.1.3 on the AWGN channel with 100 iterations of BP decoding is presented. For the code of Example 4.1.3 we also show the performance of a QC code with block length N = 1600. The convolutional code outperforms the associated QC code. Not surprisingly, the ir-

59

10

10 bit error rate

(4,8) blk Ex.3 (3,6) conv Ex.1 (3,6) conv Ex.2 (4,8) conv Ex.3

10

PSfrag replacements ms = 2048 N = 204 901 N = 4098 b = 1, c = 2 I = 50 J = 3, K = 6

Eb /N0 (dB)

10

10

0.5

1 Eb/No (dB)

1.5

Figure 4.1. Performance of the codes in Examples 4.1.1-4.1.3 on the AWGN channel. The performance of a QC code with N = 1600 obtained from the convolutional code of Example 4.1.3 is also shown.

regular LDPC convolutional code of Example 4.1.3 outperforms the regular LDPC convolutional codes.

4.2 Choosing the base matrix The base matrix B determines the Tanner graph structure of the LDPC convolutional code ensemble. Hence by appropriately choosing B , the Tanner graph of the codes can be made to have properties suited for message passing based decoding. 60

c
a

c
Figure 4.2. Existence of six cycle with polynomials of weight 3 or more.

4.2.1 Girth properties In this section we make some simple observations on the relationship between the choice of the base matrix B and the girth of the corresponding LDPC convolutional codes. Theorem 4.2.1 If any entry of B is greater than two then the Tanner graph of LDPC convolutional codes constructed from B has girth less than or equal to six. Proof: For a base matrix satisfying the conditions of the theorem, there is an entry in the parity-check matrix H (D ) of the form D a + D b + D c + . As shown in Figure 4.2 in such a case there is always a six cycle. Similarly, the following theorem can be proved. Theorem 4.2.2 If any row of B has more than one entry greater than 1 then the corresponding LDPC convolutional code always has girth less than or equal to eight. Finally, as already noted in the previous chapter, if B has a 2 3 matrix of non-zero entries then the corresponding Tanner graph always has 12 cycles. 2

61

4.2.2 Optimizing thresholds The graph structure of the convolutional codes constructed as described in Section 4.1 is determined by the base matrix. Thus, a good choice for the base matrix is one that leads to a Tanner graph structure with the best possible threshold. In this section, the design of the base matrix so as to optimize thresholds is described. The base matrix not only determines the degree distribution but also imposes a further structure on the connectivity of nodes in the resulting Tanner graph. Consider the base matrix B in Example 4.1.3. An ensemble of convolutional codes CB with b = 4 and c = 8 can be obtained from B . At each time instant the Tanner graph of codes in CB consists of c = 8 symbol nodes corresponding to the eight columns of B and b = 4 constraint nodes corresponding to the four rows of B . The degree of the symbol and constraint nodes are determined by the row and column weights, respectively, of B . Further the connectivity of nodes in the Tanner graph of codes in CB is determined by B , e.g., any of the degree ve constraint nodes corresponding to the rst row of B must connect to symbol nodes corresponding to the rst, second, third, fourth, and sixth columns of B . Similarly, e.g., any of the degree four symbol nodes corresponding to the rst column of B are restricted to be connected to four constraint nodes, one corresponding to each of the four rows of B . Thus, the base matrix structures the decoding neighborhood of a symbol in the code. Randomly constructed ensembles as in [16] do not have this feature, i.e., decoding neighborhoods are random. Hence to evaluate the thresholds of codes in CB , we perform density evolution on the Tanner graph corresponding to B rather than use the degree distribution.

62

It should be emphasized that density evolution assumes that arbitrary number of independent decoding iterations is possible. For ensembles of convolutional codes constructed as described in this chapter we have not been successful in proving the existence of such codes in the ensemble. Nevertheless we use density evolution as a rule of thumb to optimize the choice of the base matrix. Density evolution is carried out as usual by tracking the probability density function of the messages in both directions, i.e., from symbol to constraint node and from constraint to symbol node, along each edge of B . On the AWGN channel, following [37], messages are approximated as having a Gaussian distribution and it suces to track the mean of the messages. For a particular noise variance, if the mean of the messages evolve so that the probability of error approaches zero as the number of iterations is increased then decoding is successful. The threshold for the channel is the largest noise variance for which decoding is successful. In Table 4.1, the best thresholds (in terms of noise variance) we obtained with base matrices of dierent sizes for rate R = 1/2 are presented. In each case nonnegative integer matrices were randomly generated while still avoiding matrices of the type mentioned in Section 4.2.1. For each base matrix, the associated threshold was calculated and the best threshold so obtained is listed in the table.

4.3 Distance bound As already observed time invariant LDPC convolutional codes are convenient for implementation. However, as we shall see in this section, their free distance is upper bounded by a constant independent of constraint length. Further, from the

63

TABLE 4.1 BEST THRESHOLDS OBTAINED FOR BASE MATRICES OF DIFFERENT SIZES Base Matrix Size 48 5 10 8 16 Threshold 0.88 0.89 0.90 Gap from capacity 0.10 0.09 0.08

results of [29] it follows that this is also an upper bound on the minimum distance of QC codes constructed from the convolutional codes as described in Section 4.1. Lemma 4.3.1 Let H sys (D ) = [I (cb)(cb) , P (cb)b ], where I (cb)(cb) is the (cb)(cb) identity matrix, be the systematic form of the (cb)c matrix H (D ), the parity-check matrix of a rate R = b/c convolutional code. If the maximum weight of any column of H (D ) is J then any entry in P is the ratio of two polynomials each having weight at most J (cb) . Proof: We assume without loss of generality that the (c b) (c b) sub-matrix, H (cb)(cb) (D ), formed by the rst (c b) columns of H (D ) has full rank. Let
1 H (cb)(cb) (D ) be the inverse of H (cb)(cb) (D ). Then it follows that

1 H sys (D ) = [I (cb)(cb) , P (cb)b ] = H (cb)(cb) (D )H (D ).

(4.1)

1 The inverse matrix H (cb)(cb) (D ) can be obtained as

1 H (cb)(cb) (D ) =

1 Adj(H (cb)(cb) (D )), (D )

(4.2)

64

where (D ) and Adj(H (cb)(cb) (D )) are the determinant and adjoint of the matrix H (cb)(cb) (D ), respectively. We rst obtain a bound on the weight of (D ). The determinant (D ) can be calculated by expanding along the rst column of H (cb)(cb) (D ). Since the maximum column weight of H (D ) is J , it follows that (D ) is the sum of at most J (c b 1) (c b 1) determinants. Each of the (c b 1) (c b 1) determinants in turn is the sum of at most J (c b 2) (c b 2) determinants. Proceeding similarly and noting that a 1 1 determinant, i.e., an entry of H (cb)(cb) (D ) has weight at most J it follows by induction that (D ) has weight at most J (cb) . The (i, j )th entry of Adj(H (cb)(cb) (D )) is the determinant of the (c b 1) (c b 1) sub-matrix of H (cb)(cb) (D ) obtained by deleting the ith column and j th row of H (cb)(cb) (D ). From the analysis above it follows that any entry of Adj(H (cb)(cb) (D )) has weight at most J (cb1) . Now it follows from (4.1) and (4.2) that the (i, j )th entry of P is of the form
p(i,j ) (D ) . (D )

The polynomial p(i,j ) (D ) is the product of the ith row of Adj(H (cb)(cb) (D )) and the (c b + j )th column of H (D ). Since, the weight of any column of H (D ) is at most J and the weight of any entry in Adj(H (cb)(cb) (D )) is J (cb1) it follows that the weight of p(i,j ) (D ) is at most J (cb) . 2 Using Lemma 4.3.1, a code sequence with weight independent of the constraint length can be determined. Theorem 4.3.1 If the parity-check matrix H (D ) of size (c b) c of a rate R = b/c convolutional code is such that the maximum weight of any column is J then the free distance of the code is upper bounded by (c b + 1)J (cb) . Proof: From the structure of the systematic form of the parity-check matrix H (D ) in Lemma 4.3.1, it follows that there is a code sequence of the form ((D ), . . . , 0, p1,1 (D ), . . . , pcb,1 (D ). From Lemma 4.3.1 it follows that each of these polynomials 65

has weight at most J (cb) . Hence, it follows that the free distance of the code is upper bounded by (c b + 1)J (cb) . 2

For a xed b and c it follows from Theorem 4.3.1 that the free distance to constraint length ratio of time invariant LDPC convolutional codes is upper bounded by (c b + 1)J (cb) /c(ms + 1). As ms the upper bound on the free distance to constraint length ratio tends to zero, i.e., time invariant LDPC convolutional codes are not asymptotically good. However as we shall see in Chapter 6 the larger class of time varying codes contains codes with free distance increasing linearly with constraint length. Since the upper bound of Theorem 4.3.1 is also valid for the minimum distance of QC codes, it follows that the minimum distance to block length ratio for the QC codes is upper bounded by (c b +1)J (cb) /cm. The upper bound once again tends to zero as the block length N = cm is increased by increasing the circulant size m. This is in contrast to Gallagers result [12] where the existence of LDPC codes with minimum distance increasing linearly with block length was demonstrated.

66

CHAPTER 5 IMPLEMENTATION OF LDPC CONVOLUTIONAL CODE DECODERS

In this chapter, we turn our attention from code design to decoder implementation issues. Analogous to LDPC block codes, LDPC convolutional codes have sparse Tanner graphs and hence they can be decoded using message passing algorithms. The LDPC convolutional decoder continuously slides across the received sequence. At each time instant, it processes a nite window of symbols and after an initial delay also outputs decoding results at each time instant. A pipeline implementation of such a sliding window decoder was described in [24]. Section 5.1 reviews the implementation of the sliding window decoder presented in [24]. We use the Tanner graph representation of LDPC convolutional codes rather than their syndrome former matrices as was done in [24]. The features of a convolutional code Tanner graph introduced in Chapter 2; the notion of time, causality, and the restrictions on connectivity of nodes imposed by the syndrome former memory, result in a highly parallelized decoder. Section 5.2 compares the performance of LDPC convolutional and block codes for xed decoding parameters, e.g., computational complexity, delay, etc. Recent work has shown that analog transistor circuits can be used to decode LDPC block and convolutional codes. Analog decoders oer improvement with respect to decoding speed, power consumption, and chip area in comparison to digital

67

decoders. The concepts of analog decoding and a decoding architecture to decode LDPC convolutional codes using analog transistor circuits are the subject of Section 5.3.

5.1 Decoder description Suppose that we wish to perform I message passing iterations for each symbol to decode an LDPC convolutional code. The sliding window decoder performs the I message passing iterations on symbols sequentially. For a convolutional code with syndrome former memory ms , two symbols connected by a single constraint node can be at most ms time units away. Hence, in one iteration of message passing messages are exchanged across nodes at most ms time units away. Thus, to perform I iterations of message passing on a node at time t symbols no more than I (ms + 1) time instants in the past and the future are required, i.e., symbols in the interval [t I (ms + 1) + 1, t + I (ms + 1) 1] are sucient. Consider a symbol node s at time t. To perform the I th iteration and complete decoding of s, we need updated messages from the constraint nodes to which it connects. Recall from Chapter 2 that for a convolutional code with syndrome former memory ms , by virtue of causality, a symbol node at time t can only be connected to constraint nodes in the interval [t, t + ms ] and equivalently a constraint node at time t can only connect to symbol nodes in the interval [t ms , t]. Hence, the constraint nodes neighboring s can send updated messages to s as soon as symbol nodes in the interval [tms , t+ms ] have completed their (I 1)th updates. Since message updates are performed sequentially, the (I 1)th updates of symbol nodes at times prior to t are readily available. Hence, the I th update at s can be completed once the (I 1)th update of symbol nodes in the interval [t, t+ms ]

68

is completed. However, symbol nodes in the interval [t, t + ms ] require the (I 2)th update from symbol nodes in the interval [t + ms , t +2ms ] to perform their (I 1)th update. Proceeding similarly, it follows that we need the ith, i = 1, . . . , I 1, update of symbols in the interval [t + ms (I i), t + (ms + 1)(I i) 1] to complete the I th iteration for s. Thus, the decoder must be able to process nodes in the interval [t, t + (ms + 1)I 1], the decoding window at time t, and have access to the (I 1)th updates of symbol nodes in the interval [t ms , t 1] to perform I iterations of message passing for a symbol node at time t. At the I th iteration for a symbol node, its APP value is calculated and used to make a hard decision on it. Thus, for a maximum of I message passing iterations, the decoding window has size (ms + 1)I time units and slides across the Tanner graph of the convolutional code. Decoding results are output continuously after an initial delay equal to the window size. This is analogous to the operation of a Viterbi decoder with a nite backsearch limit. Figure 5.1 shows a part of the Tanner graph of the b = 1, c = 3, ms = 3 convolutional code in Example 5 of Chapter 3 1 . The decoder position corresponding to the time just before the three symbol nodes at time t are decoded is shown in dashed lines in Figure 5.1. The sliding decoding window decoder has size 4I and slides from left to right in the gure. At each time instant three symbol nodes and the two corresponding constraint nodes enter the decoding window from the right and the same number exit the decoding window from the left. At rst glance, it might appear that each time the sliding window decoder shifts to a new part of the Tanner graph all messages must be recalculated. However, the sliding window decoder can be implemented so that by storing intermediate values recomputation of messages is avoided. Further, the decoder can be implemented
1

The Tanner graph corresponds to H equiv (D).

69

t PSfrag replacements Channel values Decoding results Constraint nodes Symbol nodes

t + I (ms + 1) 1

decoding window (size= (ms + 1)I = 4I ) Figure 5.1. The sliding window decoder for I iterations of the code of Example 5.

in a highly parallelized fashion. The decoder consists of I processors operating independently and in parallel on ms + 1 time units ((ms + 1)c symbols) as shown in Figure 5.2. The ith processor performs the ith iteration of message passing, i = 1, 2, . . . , I . Suppose that each edge of the Tanner graph has associated with it a memory element to store the message passed along that edge during iterations. Channel messages are stored in memory elements associated with the symbol nodes. A processor can access the memory elements along edges connected to constraint nodes under its inuence. Each processor rst activates the c b = 2 constraint nodes and then the c = 3 variable nodes referred to as active nodes in Figure 5.2. The constraint nodes which are activated correspond to those that have just entered into the inuence of the ith processor and the activated symbol nodes are those that will exit the inuence of the ith processor once updated. When a constraint node is activated, it reads the memory locations corresponding to the edges which it connects to

70

PSfrag replacements
t

Proc. I

Proc. I 1

Proc. 2

Proc. 1

t + 4I 1

Decoding results

       

  

Active constraint nodes decoding window (size= I (ms + 1))

Active symbol nodes

Figure 5.2. Continuous decoding of the code of Example 5 with I iterations.

and updates these locations with new messages. Activating a symbol node leads to exactly the same set of operations. In this case access to the memory locations containing the channel values is also needed to update messages. The message updates are calculated according to the particular message passing algorithm being used, for example in the case of the BP algorithm the update equations are as in Chapter 2. Consider the part of the Tanner graph under the inuence of the rst processor in Figure 5.2, i.e., the interval [t + 4(I 1) 1, t + 4I 1]. The bottom of the two active constraint nodes at time t + 4I 1 is connected to three symbol nodes, the rst at time t + 4I 1, the second at time t + 4I 3, and the third at time t + 4(I 1). The corresponding edges are shown using dashed lines in Figure 5.2. Consider the situation just before the constraint nodes at time t + 4I 1 are activated by processor 1. The edges corresponding to symbol nodes in the past (shaded arrows) already contain the channel values (in general the values from the previous iteration). The constraint node only needs the channel value along the 71

Channel values

   

  

edge (in general the value from previous iteration) corresponding to the symbol node at t + 4I 1 to be able to update messages along its connecting edges. The last of the three active symbol nodes at t + 4(I 1) connects to two constraint nodes, one at time t + 4(I 1), and the other at time t + 4I 1. The corresponding edges are shown using dashed lines. Once more let us look at the situation just before the constraint nodes at t + 4I 1 are activated by processor 1. Since the constraint node at time t + 4(I 1) was activated earlier (in fact 3 time units back) this edge (shaded arrow) contains an updated message. The symbol node only lacks the message along the other edge (empty arrow), i.e., from the constraint node three time units away. Once processor 1 activates the two constraint nodes at t + 4I 1 each of them can use the values along the edges to which it connects2 and calculate the new message for each edge. Thus, all edges connected to the two constraint nodes now contain updated messages. When the symbol nodes are then activated all its edges contain updated messages from the associated constraint nodes. The symbol nodes can now calculate the new messages to be output along each of the connecting edges and the rst iteration is complete for symbol nodes at time t + 4(I 1). Observe that each processor can operate independent of the other processors. The I th processor calculates APP values and outputs the nal decoded values for the three active symbols under its inuence. This process is now repeated as the decoder slides along the Tanner graph. The memory locations are overwritten only when either constraint nodes or symbol nodes dening the edge are activated. For any edge, the associated constraint node is activated rst, followed ms time units later by the activation of
The memory locations along each edge are initialized with the channel messages of the associated symbol nodes.
2

72

the associated symbol node. This ensures that a memory location has exactly the values required for the nodes to complete their ith update when accessed by them. It is not necessary to have memory elements for each edge of the Tanner graph. As shown in [24], (ms + 1)cI (J + 1) memory elements are sucient. It is also possible to use one single processor rather than I processors. Such a processor would sequentially hop from block to block of ms + 1 time units to perform the I decoding iterations. A similar implementation of the sliding window decoder can also be used for decoding LDPC convolutional codes with periodically timevarying syndrome former matrices. In the case of time-invariant convolutional codes, the graph connections are similar at each time instant; this may be used to further simplify the processors.

5.2 Comparison of message passing decoding of block and convolutional codes 1. Computational Complexity Let Cconstraint (Csym ) denote the number of operations required for a constraint (symbol) node of degree K (J ) to calculate its K (J ) updated messages. Clearly, Cconstraint and Csym depend only on J and K 3 . Consider a rate R = b/c, (J, K ) regular LDPC convolutional code with syndrome former memory ms , decoded using the sliding window decoder described in the previous section. Each processor activates c symbol nodes and (c b) constraint nodes. Hence, the decoding
conv for I iterations of message passing is complexity per symbol Cbit

The exact dependence varies according to the particular message passing algorithm being considered.

conv Cbit = ((c b)Cconstraint + c Csym ) I/c.

73

Similarly, for a (N, J, K ) LDPC block code the decoding complexity per symbol
blk Cbit is blk Cbit = (LCconstraint + N Csym ) I/N. conv blk Thus, Cbit = Cbit and in fact, the complexity of decoding is independent of the

constraint length (ms + 1)c in case of LDPC convolutional codes and the block length N in case of LDPC block codes. 2. Processor (Hardware) Complexity The sliding window decoder of LDPC convolutional codes operates on (ms + 1)cI symbols. However, as described in the previous section decoding can be carried out by using I parallel processors, each capable of handling (ms + 1)c symbols. Hence, it is sucient to design the processor hardware for (ms + 1)c symbols. For a block length N code the processor must be capable of handling N symbols. Thus, for equal processor complexity, N = (ms + 1)c. 3. Storage For the sliding window decoder, we need a memory element for each edge and also memory elements are needed for storing the channel values of symbols in the decoding window. Thus (ms + 1)c I (J + 1) memory elements are required for decoding convolutional codes. Similarly for the block codes we need N (J + 1) memory elements4 . Thus for the same storage requirements we must have N = (ms + 1)c I . 4. Delay Let Ts denote the time between arrival of successive symbols, i.e., the symbol rate is 1/Ts . Then for the sliding window decoder the maximum time from the
In both cases we ignore the buering usually needed before symbols can be processed by the decoder.
4

74

arrival of a symbol until it is decoded, i.e., the delay conv is io conv = ((c 1) + (ms + 1)I )Ts . io The rst component of the delay is the time between the arrival of the rst and last of the c encoded symbols output by the rate R = b/c convolutional encoder at each encoding. The second component, which is the dominant term, is the time each symbol spends in the decoding window, i.e., the window size. Since the c symbols output at each encoding are loaded into the decoder simultaneously the sliding window decoder needs a buer to hold the rst (c 1) symbols. With block codes data is typically transmitted over several block lengths. Thus, depending on the data rate and processor speed several scenarios are possible. We consider the best case for the block codes: each block is decoded by the time the rst bit of the next block arrives, i.e., blk io = N Ts . In this case there are no buers needed. Let Tdec (N, I ) denote the time required to perform I iterations of message passing on a block of N symbols. Thus, we require Tdec (N, I ) Ts , i.e., we need very fast decoders or equivalently low data rates for this to be possible. Note that in contrast Tdec (c, I = 1) cTs for the convolutional codes. Thus, for equal decoding delays N = (c 1) + (ms + 1)c I , assuming the least possible delay for block codes. In Figure 5.3 we plot the performance of a b = 1, c = 2, ms = 2048, (3, 6) regular LDPC convolutional code5 with I = 50 iterations on the AWGN channel. Also shown is the performance of two (3, 6) regular LDPC block codes. The block lengths were chosen so that in one case the decoders have the same processor
5

The LDPC convolutional was constructed as described in [24].

75

10

10 bit error rate

I = 50 J = 3, K = 6 b = 1, c = 2

N = 4098

10

PSfrag replacements

10

N = 204 901

10

ms = 2048

1.2

1.4
Eb /N0 (dB)

1.6

1.8

Figure 5.3. Performance comparison of LDPC block and convolutional codes on the AWGN channel for the same processor complexity and decoding delays. The block code is decoded with a very fast decoder.

complexity, i.e., N = (ms +1)c and in the other the same decoding delay assuming the least possible delay for the block codes, i.e., N = (c 1) + (ms + 1)c I . For the same processor complexity the convolutional code outperforms the block code by about 0.6 dB at a bit error rate of 105 . For the same decoding delay the convolutional and block code performance is nearly identical. Note that in the case of equal decoding delay the memory requirements for the block code is almost the same as that of the convolutional code.

76

PSfrag replacements

N symbols Buer Decoder 1

Tdec (N, I = 1) = N Ts Decoder 2 Decoder I

io = (I + 1)N 1 Ts Figure 5.4. Serial decoding of LDPC block codes using I decoders.

Typically, the data rates and decoding speeds are such that some buering is needed to decode the block codes. There are several dierent decoding architectures possible in this case. However, the delay in all of these scenarios is at least as much as in the scenario considered earlier so that the convolutional code would outperform the block code in these cases for the same decoding delay. The block code can also be decoded by means of an architecture similar to the convolutional code by treating it as a convolutional code with memory zero. Such a decoder would consist of a buer to store N symbols and I processors each performing one iteration of message passing. This decoding architecture is shown in Figure 5.4. In this case blk io = (N 1 + IN )Ts and Tdec (N, I = 1) N Ts . The block length N is analogous to c in this case. This decoder structure for block codes is appropriate when transmitting a large amount of data (>> N ) at high data rates.

5.3 Analog decoding Recently, Hagenauer [38] and Loeliger et al. [39] independently proposed that the BP algorithm be carried out using analog circuits. An analog decoder operates asynchronously while in a digital (more generally discrete time) decoder all computations are synchronized and carried out in accordance with a system clock. 77

V 3

CONNECTIVITY

V 2

V 1

CONNECTIVITY:
SYM:

CHK:

Figure 5.5. Transistor circuits for implementing the symbol (SYM) and constraint (CHK) node updates.

Remarkably, a simple transistor circuit can be used to implement the message update equations at the constraint and symbol nodes in the BP algorithm. The circuits are quite similar to the Gilbert multiplier which is commonly used for real number multiplication. Figure 5.5 shows the transistor circuitry and the connectivity for implementing the updates at the symbol (SYM) and constraint (CHK) nodes. In either case the circuit in Figure 5.5 calculates one output from the two inputs. Thus, three such circuits are needed to calculate all three updated messages of a degree three constraint node. For a degree three symbol node, two circuits with the symbol node connectivity are required to obtain one updated message since three LLRs are to be summed (the channel message and messages from the other two constraint nodes). Thus, calculating all three updated messages requires three units, each unit consisting of two of the circuits with the

78

symbol node connectivity of Figure 5.5. Similarly, for higher degree nodes the nal updated message can be calculated by using several such two input blocks. To decode a code with the BP algorithm a collection of the circuits for symbol and constraint nodes are simply wired together according to the Tanner graph corresponding to the code. Decoding begins by loading channel values into the circuit and after a small delay (due to parasitic resistance and capacitances) the circuit settles into a steady state (usually) and decoding is complete. Similar circuits can also be used for implementing trellis based decoding algorithms like Viterbi algorithms. Practical implementation of analog decoders have been reported in [40][41][42]. Loeliger and others [43] have observed that analog decoding networks are quite robust against the nonidealities of real VLSI and outperform digital implementations by two orders of magnitude in terms of speed for the same power consumption and vice versa.

5.3.1 Analog decoding of LDPC convolutional codes A straightforward way for continuous decoding of LDPC convolutional codes in analog is to implement the sliding window decoder introduced in Section 4.2 using the circuits of Figure 5.5. One problem with this approach arises in the storage of messages. In a digital decoder the required storage is obtained by using shift registers, which however are dicult to implement in analog. This problem can be overcome by means of a rotating ring decoder [44]. A similar architecture for trellis based decoding of convolutional codes has been suggested in [45] and [46]. The analog rotating ring decoder is shown in Figure 5.6. The decoder is once again based on a window of received values, the decoding of bits at time t requires the received values (LLRs) from t W to t + W . The position where the decoded

79

READ OUT t+1 t t1

PSfrag replacements

t+W 1 t+W WRITE IN

tW +1 tW

Figure 5.6. The rotating ring decoder

results are read out and the position where new received values are written into the ring rotate at the transmission rate. The design of the rotating ring decoder involves the following trade-os: 1. Window size: Using a larger W means more information is collected in decoding any symbol. However, from a circuit design point of view a smaller W is desirable. 2. Erroneous message exchange: In the ring decoder, symbol nodes at times t W and t + W are connected together which is not the case in the Tanner graph of the convolutional code. Thus, there is potentially an erroneous

80

exchange of messages at this point in the decoding window. Once again a large W mitigates the eect of this erroneous exchange of messages on the decoded symbols at time t. 3. Reloading messages: If there is no storage then all channel messages must be reloaded into the decoding window and all messages recomputed at each time instant to decode. It is much more convenient to only load in the newly received values into the rotating ring and also avoid recomputation by just using the previously calculated messages. In order to determine the inuence of these factors we compare the performance of dierent decoding architectures on the code presented in Example 6
6

of Chapter 3. We consider a) the sliding window decoder, b) the rotating ring decoder with channel values reloaded each time, and c) the rotating ring decoder with no reloading. All results unless otherwise mentioned are obtained using discrete time simulations. Figure 5.7 shows the performance obtained by decoding the test code for two dierent window sizes. The reference curve corresponds to the performance obtained with a large window size W = 502. The performance with W = 15 is almost identical to the reference curve. The decoder with W = 9 has an acceptable performance loss with respect to the reference curve. The total memory of this code is small enough to permit trellis based decoding and the maximumlikelihood (ML), i.e., Viterbi decoding performance7 is also shown in Figure 5.7. As can be seen in Figure 5.7, BP decoding is not optimal and using a small W leads to further loss.
6 7

In this case we use H (D). The maximum a-posteriori decoding performance is identical.

81

10

Viterbi decoder reference curve; W=15, Decoder c)

10 bit error rate

W=9, Decoder c)

10

10

10

5 Eb/N0 (dB)

Figure 5.7. Bit error rate curves on the AWGN channel with rotating ring decoders (Type c)) of dierent window sizes. The ML performance is also shown.

Figure 5.8 compares decoders of dierent types for W = 9 to break down the eects described earlier. A comparison of decoder type c) and decoder type b) indicates that not reloading channel values has negligible eect on performance. In this case an analog rotating ring decoder was also simulated. Interestingly, these results are slightly better than that of the corresponding digital decoder. Decoder type a) was also simulated as an analog decoder thus separating the inuence on performance of using a small window size (compare reference curve to W = 9 decoder a) analog) and of erroneous message exchange and not reloading channel

82

10

reference curve W=9 Decoder b) and c)

10 bit error rate

W=9 Decoder c) analog W=9 Decoder a) analog

10

10

10

5 Eb/N0 (dB)

Figure 5.8. Bit error rate curves for dierent decoder types with W=9. Type a) is the usual sliding window decoder. Type b) is a rotating ring decoder with channel values reloaded at each time. Type c) is a standard rotating window decoder (no reloading)

values (compare W = 9 decoder a) analog to W = 9 decoder c) analog). Since not reloading channel values has negligible eect on performance, the dierence is only due to erroneous message exchange. In order to understand the dierence between analog and digital decoding a few test cases were considered. In each case, both decoders were fed with the same channel inputs. Figure 5.9 shows results for four such cases. In each case the decoders found the same xed point and the output LLRs converge to almost the

83

same values. We see also that the two decoders display approximately the same dynamic behavior (small oscillations) with the oscillations of the analog decoder being slightly more damped. Finally, Figure 5.10 shows probability distribution functions (generated in [47]) for the output LLRs of analog and digital decoders. This shows that the average behavior of the analog and digital decoders is nearly the same. The results on bit error rates in Figure 5.8 are nearly identical, hence it follows that the tails of the pdfs, which determine the bit error rates, are also similar for the analog and digital decoders.

84

20

Final LLR

20 0

20

Final LLR
0 100 200

20 0

LLR

20

50

100

20

LLR

50

20

20

50

20

Final LLR

Iteration

20 0

Bit position

20

Time

Final LLR
200

20 0

Bit position

LLR

LLR

20

50

100

20

50

20

100

20

50

20

Final LLR

Iteration

20 0

Bit position

20

Time

Final LLR
200

20 0

Bit position

LLR

LLR

20

50

100

20

50

20

100

20

50

20

Final LLR

Iteration

20 0

Bit position

20

Time

Final LLR
200

20 0

Bit position

LLR

LLR

20

50

100

20

50

20

100

20

50

Iteration

Bit position

Time

Bit position

Figure 5.9. Decoding results for the time discrete decoder (left two columns) and the time continuous decoder (right two columns) for one received block of length 55 bits. Each row represents a dierent test case.

85

0.08 0.07 0.06

1dB

digital analog

p(L(z))

0.05 0.04 0.03 0.02 0.01 0 10 0 10

5dB 3dB 1dB

LLR L(z)

20

30

40

50

Figure 5.10. Probability distributions of output LLRs for analog and digital decoders. L(z ) is the decoder output and p(L(z )) is the probability distribution function of the random variable L(z ).

86

CHAPTER 6 DISTANCE BOUNDS FOR LDPC CONVOLUTIONAL CODES

In Chapter 4, we proved that the class of time invariant LDPC convolutional codes with xed b and c is not asymptotically good. However, this is not the case with time varying LDPC convolutional codes. In [48], the distance spectrum of (J, K ) LDPC convolutional codes with K = 2J was analyzed. A technique to numerically calculate the distance spectrum of the codes as a function of the constraint length was described in this work. The results obtained in [48] suggest the existence of LDPC convolutional codes with free distance increasing linearly with constraint length. In this chapter we prove the existence of (J, K ) regular LDPC convolutional codes with free distance satisfying, dfree LDPCC (J, K ) . This parallels the result of Gallager for LDPC block codes stated in Chapter 2. As already noted, Gallagers coecient is smaller than the corresponding GV parameter. The analogue of the GV bound for convolutional codes is the Costello bound. Costello proved that there exist convolutional codes with dfree C (R) for suciently large constraint lengths . The coecient C (R) is several times larger than GV (R). We shall similarly nd that LDPCC (J, K ) > G (J, K ), and that LDPCC (J, K ) is smaller than the corresponding Costello coecient. We begin by describing the ensemble of (J, K ) regular LDPC convolutional codes of interest in Section 6.1. The lower bound on free distance is obtained by lower bounding the row distances of the code. We describe the technique to 87

obtain lower bounds on the Lth order row distances of the ensemble of codes in Section 6.2. In Section 6.3 we obtain another bound on the row distance by weakening the bound of the previous section. The weakened bound permits analytic evaluation and is used to provide a rigorous proof that the free distance increases linearly with the constraint length of the code. Finally, we provide numerical estimates of the parameter LDPCC (J, K ) and conclude with a discussion of the results.

6.1 An LDPC convolutional code ensemble We begin by introducing a new class of (J, K ) LDPC convolutional codes. The syndrome formers of these codes are composed of blocks of M M permutation matrices. (A permutation matrix P is a matrix consisting of zeros and ones such that each row and column has exactly a single one.) These codes are the convolutional counterpart of the LDPC block codes introduced in [49]. Let a = gcd(J, K ) denote the greatest common divisor of J and K . Then there exist positive integers J and K such that J = aJ , K = aK , and gcd(J , K ) = 1. We construct LDPC convolutional codes dened by syndrome formers H T with syndrome former memory a 1. For i = 0, 1, . . . , a 1, the sub-matrices H T i (t + i) of the syndrome former are + i) + i) (1,1) P (1,0) (t + i) P i (t + i) i T H i (t + i) = ... ... (K 1,0) (K 1,1) Pi (t + i) P i (t + i)
(0,0) P i (t (0,1) P i (t

+ i) ... , ... ... (K 1,J 1) ... Pi (t + i) (6.1)

(1,J 1) (t Pi

...

(0,J 1) (t Pi

+ i)

88

M M permutation matrices

HT =

PSfrag replacements

J =3

K=6

Figure 6.1. syndrome former for a code in CP (3, 6, M ). where each P i

(k,j )

(t + i), k = 0, 1, . . . , K 1, j = 0, 1, . . . , J 1, is an M M

permutation matrix. All other entries of the syndrome former are zero matrices. Equivalently, each H T i (t + i), i = 0, 1, . . . , a 1, is a c (c b) binary matrix where c = K M and b = (K J )M . By construction, it follows that each row of the syndrome former H T has J ones and each column K ones. Let CP (J, K, M ) denote this ensemble of (J, K ) LDPC convolutional codes. (Note that the ensemble CP (J, K, M ) is time-varying.) Figure 6.1 shows the syndrome former of a (3, 6) LDPC convolutional code in CP (3, 6, M ). The syndrome formers in the ensemble CP (J, K, M ) have ms = a 1 independent of M while b and c depend on M . This is dierent from the LDPC convolutional codes considered so far where b and c were small xed numbers

89

while ms was large. For the ensemble CP (J, K, M ) as M increases, i.e., as b and c increase, the syndrome formers become increasingly sparse. For large M , the convolutional codes in the ensemble CP (J, K, M ) have characteristics which maybe useful for applications where the information has a natural block characteristic. For encoding, blocks of symbols (the number of symbols in a block depends on M ) are input to a systematic LDPC convolutional encoder. Decoding is carried out using the sliding window decoder (Chapter 5). At each time instant the channel messages of a block of symbols (the number of symbols in a block equals c = K M ) is input to the sliding window decoder and the same number of symbols are decoded and output from the decoder. For the ensemble CP (3, 6, M ), the matrices H T 0 (t) consist of two M M permutation matrices and hence have rank equal to M , i.e., the code rate is R = M/2M . In this case, by permuting rows of the syndrome former of any code in CP (3, 6, M ) an equivalent b = 1, c = 2, rate R = 1/2 code with syndrome former memory at most 3M 1 can be obtained (see Figure 6.2). Since distance properties are unaected by row permutations, the distance bounds obtained for codes in the ensemble CP (3, 6, M ) are also valid for the equivalent b = 1, c = 2 codes. In general, there are at least J 1 dependent columns in H T 0 (t) for any code
(J 1) in CP (J, K, M ). Hence, H T denes a rate R 1 J MK code. The constraint M

length of codes in CP (J, K, M ) is dened as = (ms + 1) c = a K M = KM . Thus the codes in the ensemble CP (3, 6, M ) have constraint length 6M . The syndrome formers of the LDPC convolutional code ensemble described here have a structure similar to that of the permutation matrix based LDPC block code ensemble described in [49]. The parity-check matrices of the codes in the block code ensemble considered in [49] are composed of J K permutation

90

P 0 (t) P 1

(0)

(0)

(t + 1) P2

(0)

11
(t + 2)

1 1

(0) (t + 1) (0) (t + 2) P P 1 2

1 1 .

P 0 (t) P 1

(1)

(1)

P2 (t + 1)

(1)

11
(t + 2)

1
(1) (t + 2) (1) (t + 1) P P 1 2

1 1 1 1 .

P1

(0)

(t + 1) P2

(0)

(t + 2)

(0) (0) (0) P 1 (t + 2) P 2 (t + 3) P 0 (t + 1)

11

1 1

(0) (t + 2) (0) (t + 3) P P 1 2

1 1 1 1 1 .
. .

(1)

(t + 1) P2

(1)

(t + 2)

P0

(1)

(t + 1) P1

(1)

(t + 2) P2

(1)

11
(t + 3)

1
(1) (t + 3) (1) (t + 2) P P 2 1

1 1

P 1 (t + 2) P 2 (t + 3)

(0)

(0)

1 1 .

. P1

P0

(0)

P1 (t + 2)

(0)

(t + 3) P2

(0)

11

1 1

(1)

P2 (t + 2)

(1)

(t + 3)

(t + 4)

1 1

(0) (t + 4) (0) (t + 3) P P 2 1

1 1 .

(1) (1) (1) P 2 (t + 4) P 0 (t + 2) P 1 (t + 3)

11

1
(1) (t + 4) (1) (t + 3) P P 2 1

1 1

P1

(0)

P2 (t + 3)

(0)

(t + 4)

1 1 .

. P1

1 1

(1)

(t + 3)P 2

(1)

(t + 4)

Figure 6.2. Permuting rows of a syndrome former H T in CP (3, 6, M ). matrices, where each permutation matrix is of size M M . Thus the paritycheck matrices are of size JM KM and have exactly K ones in each row and J ones in each column. This ensemble is a vanishingly small sub-ensemble of Gallagers ensemble [12]. In fact, if a = gcd(J, K ) = 1, i.e., J and K are relatively prime, then the ensemble of LDPC convolutional codes CP (J, K, M ) has syndrome former memory ms = a 1 = 0. This ensemble of memory zero (J, K ) regular LDPC convolutional codes is equivalent to the block code ensemble considered in [49]. In [49], we show that asymptotically, i.e., as N = KM , almost all codes in the ensemble have minimum distance satisfying Gallagers bound1 , i.e., dmin G (J, K )N . A probability distribution is dened on the ensemble CP (J, K, M ) as follows. Assume that all of the permutation matrices comprising the syndrome former of
1

This holds for small values of J and K .

91

a code in CP (J, K, M ) are chosen independently and such that each of the M ! possible permutation matrices is equally likely.

6.2 A lower bound on row distance Our aim is a lower bound on the free distance, i.e., the minimum Hamming weight over all non-zero code sequences of the convolutional code. In general, obtaining a lower bound on the free distance of convolutional codes is more complicated than lower bounding the minimum distance of block codes since for convolutional codes code sequences with non-zero segments of varying lengths must be considered. For the class of periodically time-varying codes, it is sucient to consider code sequences with starting positions within one period (see [7]) but having arbitrary length2 . However, in the most general case of non-periodically time-varying codes, such as codes from the ensemble CP (J, K, M ), all possible starting positions must be considered. Our approach is to rst obtain a lower bound on the Hamming weight of code sequences with non-zero segments of length at most L for all possible starting positions, i.e., a lower bound on the Lth order row distance [7]. The minimum of the lower bounds on row distance provides a lower bound on the free distance. To obtain a lower bound on the Lth order row distance, we calculate the expected number of code sequences with non-zero segments of length at most L and Hamming weight less than d in a randomly chosen code from the ensemble CP (J, K, M ). If the expected number is less than one, then it follows that there exists at least one code in the ensemble with Lth order row distance lower bounded by d. To avoid cumbersome notation, we henceforth focus on the (3, 6) case,
The Costello bound was derived for the class of periodically time-varying codes with large periods.
2

92

i.e., the ensemble CP (3, 6, M ), though the same technique can also be used more generally.

6.2.1 Row distance Consider sequences v [i+1,i+L] = (. . . , 0, v i+1 , v i+1 , . . . , v i+L , v i+L , 0, . . . ), v j FM 2 , for j = i + 1, . . . , i + L, h = 0, 1, with v i+1 = 0, i.e., sequences starting at t = i + 1 with a non-zero segment of length at most L. The sequence v [i+1,i+L] consists of at most 2LM non-zero binary symbols. If v [i+1,i+L] is a code sequence, then at each time instant t, i < t < (i + L + 1), v [i+1,i+L] contains an information block v t
(0) (h) (0) (0) (1) (0) (1)

of length M and the corresponding length M parity block v t .

(1)

The row distance d(L) of a code in CP (3, 6, M ) is dened as the minimum weight of code sequences of the form v [i+1,i+L] , v i+1 = 0, over all possible starting positions i + 1. Hence, the free distance dfree of the code satises
(0)

dfree = min d(L), L 1.

L

To obtain a lower bound on the row distance d(L) we rst calculate the expected number of code sequences of the form v [i+1,i+L] that have weight less than d in a randomly chosen code from the ensemble CP (3, 6, M ). Without loss of generality, we investigate code sequences starting at time t = 1, i.e., of the form v [1,L] , and obtain a lower bound on the weight of such sequences. For a sequence v [1,L] , let dt = (dt , dt ), t = 1, . . . , L, where dt
(h) (0) (1) (h)

is the

Hamming weight of v t , t = 1, . . . , L, h = 0, 1. Dene the 2LM (L + 2)M

93

v1 v1 v2

(0)

d1 d1

(0)

(1)

(1)

(0)

d2 d2 d3 d3 1 2 3 4 5

(0)

v2 v3

(1)

(1)

(0) (1)

(0)

v3

(1)

Figure 6.3. Illustration of length L = 3 segment of H T [1,L] .

matrix H T [1,L] as HT 0 (1) HT 1 (2) HT 0 (2) HT 2 (3) HT 1 (3) .. .

HT [1,L]

.. .. . . . T T T H 0 (L 1) H 1 (L) H 2 (L + 1) T T T H 0 (L) H 1 (L + 1) H 2 (L + 2) HT 2 (4)

If v [1,L] is a code sequence, then it must satisfy the (L + 2)M constraints imposed by the matrix H T [1,L] . Figure 6.3 illustrates the non-zero part of the sequence v [1,3] and the segment H T [1,3] of the syndrome former for a randomly chosen code in CP (3, 6, M ). For any t, 1 t L + 2, let t be the probability in the ensemble CP (3, 6, M ) that v [1,L] satises the constraints (M equations) imposed by the sub-matrices
T T HT i (t) of H [1,L] , i.e., the tth column of H [1,L] . Let [1,L] be the probability that

94

v [1,L] is a valid code sequence. Then

L+2

[1,L] =
t=1

t .

(6.2)

The terms t in (6.2) can be calculated by following the technique introduced in [49] as described below.

6.2.2 Probability for a single block of constraints We rst calculate the probability that an IM dimensional vector, I 2, w[1,I ] = (w1 , w2 , . . . , w I ), w i FM 2 , i = 1, . . . , I, satises the condition w [1,I ] = 0, (6.3)

where the IM M matrix is given by = [P 1 , P 2 , . . . , P I ]T and the P i , i = 1, . . . , I , are M M permutation matrices. For the block code ensemble in [49] only the case I = K was needed. However, for the (J, K ) LDPC convolutional code ensemble considered here, the cases I = 2, 4, . . . , K , are required. In particular, for (3, 6) LDPC convolutional codes, i.e., the ensemble CP (3, 6, M ), the cases I = 2, 4, 6, must be considered. Since the permutation matrices comprising the syndrome formers in CP (3, 6, M ) are chosen independently and are equally likely, we assume the same for the matrices P i , i.e., they are chosen independently and take on one of the M ! possible values with equal probability. Let di , i = 1, . . . , I , be the Hamming weight of the M dimensional vector w i . We say that w[1,I ] has weight composition d = (d1 , d2 , . . . , dI ). The Hamming weight d of the vector w[1,I ] with weight composition d is d = d1 + d2 + + dI .

95

If w[1,I ] satises (6.3), then each of the M constraint equations dened by the columns of must include an even number of ones, i.e., 0, 2, . . . , 2 I/2 , from w [1,I ] . Since is composed of blocks of permutation matrices, a constraint equation involves at most a single one from each w i , i = 1, 2, . . . , I . For the mth constraint equation of , m = 1, . . . , M , we associate an I dimensional binary vector pm . The ith component of pm , i = 1, . . . , I , is one if a one from w i is involved in the mth constraint equation and is zero otherwise. Since any constraint equation involves an even number of ones, pm can take on one of 2I 1 values. Let 0 denote the number of constraint equations involving only zeros of w [1,I ] . Then, let 2 (i1 , i2 ) be the number of constraints involving two ones, one from each of the vectors w i1 and w i2 , and zeros from the other I 2 components of w[1,I ] . In general, let 2q (i1 , . . . , i2q ), q = 1, . . . , I/2 , denote the number of constraints involving 2q ones, one from each of the vectors wi1 , . . . , wi2q , and zeros from the remaining (I 2q ) components of w [1,I ] . Observe that the arguments of 2q (i1 , . . . , i2q ) are distinct, i.e., i1 = i2 = = i2q . Further, 2q (i1 , . . . , i2q ) is invariant to permutations of the arguments, for example, 2q (i1 , i2 , . . . , i2q ) = 2q (i2 , i1 , . . . , i2q ). In other words, 2q (i1 , . . . , i2q ) is a function of the set {i1 , i2 , . . . , i2q }. To emphasize this fact, we henceforth write 2q ({i1 , . . . , i2q }) for 2q (i1 , . . . , i2q ). There exist
I 2q

dierent sets {i1 , i2 , . . . , i2q }. Therefore the function

I 2q

2q ({i1 , i2 , . . . , i2q }) can take on

values, not necessarily distinct.

Now assume that I = 6. Then the number of ones in each constraint is at most six. In this case, since the total number of constraints is M , we have3
We omit the arguments and write 6 since, in this case, we have a one from each of the six components of w[1,6] .
3

96

0 +
{i1 ,i2 }

2 ({i1 , i2 }) +

{i1 ,i2 ,i3 ,i4 }

4 ({i1 , i2 , i3 , i4 }) + 6 = M.

(6.4)

Further, it follows from the denition of 2q ({i1 , i2 , . . . , i2q }) that for any i1 {1, 2, , I }, 2 ({i1 , i2 }) + 4 ({i1 , i2 , i3 , i4 }) + 6 = di1 . (6.5)

{i2 }

{i2 ,i3 ,i4 }

For any vector w [1,6] with weight composition d, (0 , {2 ({i1 , i2 })}, {4 ({i1 , i2 , i3 , i4 })}, 6 ) is called a constraint composition of w [1,6] if 0 , {2 ({i1 , i2 })}, {4 ({i1 , i2 , i3 , i4 })}, 6 satisfy (6.4) and the six equations implied in (6.5). To clarify some of the above notation we present a simple example. Example 6.2.1 Let I = 6 and M = 5. In this case each of the vectors w i , i = 1, . . . , 6, is a ve dimensional vector. Let the vectors w i be

w 1 = (0, 1, 1, 1, 0) w 2 = (1, 0, 0, 1, 0) w 3 = (0, 0, 1, 1, 0) w 4 = (0, 1, 0, 0, 1) w 5 = (1, 0, 0, 1, 1) w 6 = (0, 0, 1, 0, 1).

97

Hence, w [1,6] has weight composition d = (3, 2, 2, 2, 3, 2). Suppose now that = [P 1 , P 2 , . . . , P 6 ]T , with
0 0 = 0 0 1 0 0 1 0 0 1 0 0 0 0 0 1 0 0 0 1 0 0 T 0 , P 2 = 0 1 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 1 0 0 0 0 T 1 , P 3 = 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 , 0 1 0

PT 1

PT 4

0 1 = 0 0 0

0 0 1 0 0

1 0 0 0 0

0 0 0 0 1

0 0 0 T 0 , P 5 = 1 1 0 0 0 0

0 0 0 0 1

0 1 0 0 0

1 0 0 0 0

0 0 0 T 0 , P 6 = 1 1 0 0 0 0

0 1 0 0 0

1 0 0 0 0

0 0 0 0 1

0 0 . 1 0

Since I = 6, the vector pm associated with the mth constraint equation, m = 1, . . . , 5, is a six dimensional vector. The rst constraint equation involves a one from each of the vectors w 2 , w3 , w4 , and w 6 . Hence, p1 = (0, 1, 1, 1, 0, 1). Similarly, we obtain

p2 = (1, 0, 0, 0, 1, 0) p3 = (0, 0, 0, 0, 0, 0) p4 = (1, 1, 1, 1, 1, 1) p5 = (1, 0, 0, 0, 1, 0). The third constraint equation involves no ones from w [1,6] , and hence 0 = 1. The second and fth constraint equations involve ones from w 1 and w 5 , so 2 ({1, 5}) = 2. In fact, this is the only non-zero term in the set {2 ({i1 , i2 })}. The rst constraint equation involves ones from w 2 , w3 , w4 , and w6 . This is the only non-zero term in the set {4 ({i1 , i2 , i3 , i4 })}, and we have 4 ({2, 3, 4, 6}) = 1. The 98

fourth constraint equation involves a one from each of the vectors w i , i = 1, . . . , 6. Hence 6 = 1. Finally, it is easy to check that (6.4) and the six equations implied in (6.5) are satised. 2

Let c = (p1 , . . . , pM ). If w[1,6] has constraint composition (0 , {2 ({i1 , i2 })}, {4 ({i1 , i2 , i3 , i4 })}, 6 ) then it follows that 0 of the vectors pm , m = 1, . . . , M , are the all-zero vector, 2 ({i1 , i2 }) of the vectors pm have ones only in positions i1 and i2 for all {i1 , i2 }, 4 ({i1 , i2 , i3 , i4 }) of the vectors pm have ones only in positions i1 , i2 , i3 , and i4 for all {i1 , i2 , i3 , i4 }, and 6 of the vectors pm are the all-one vector. For any constraint composition (0 , {2 ({i1 , i2 })}, {4 ({i1 , i2 , i3 , i4 })}, 6 ) there are in total 0 ! possible vectors c. Let M! 2 ({i1 , i2 })! 4 ({i1 , i2 , i3 , i4 })! 6 !
{i1 ,i2 ,i3 ,i4 }

{i1 ,i2 }

= {(0 , {2 ({i1 , i2 })}, {4 ({i1 , i2 , i3 , i4 })}, 6 )} represent the set of possible constraint compositions of w [1,6] . Note that is only a function of the weight composition d of w [1,6] , i.e., = (d). For a given weight composition d there are in total M! (d) 0 !
{i1 ,i2 }

6 (d) =

2 ({i1 , i2 })!

{i1 ,i2 ,i3 ,i4 }

4 ({i1 , i2 , i3 , i4 })!6 !

(6.6)

possible vectors c, where the summation in (6.6) is over all constraint compositions included in (d). The formulas (6.4)-(6.6) can be easily generalized to an arbitrary I . In the general case the probability that w [1,I ] satises (6.3) is determined by the following lemma.

99

Lemma 6.2.1 The probability I (d) that a vector w[1,I ] = (w 1 , w 2 , . . . , wI ) with weight composition d = {d1 , d2 , . . . , dI } satises (6.3) is I (d) = I (d)
I M i=1 di

(6.7)

Proof: Consider an arbitrary vector w [1,I ] with weight composition d. For a xed w [1,I ] and c there are di !(M di )! ways of choosing the permutation matrix P i , i = 1, . . . , I . Since the number of possible c is equal to I (d) and each of the permutation matrices can be chosen in M ! ways, we obtain I (d)
I i=1

I (d) =

di !(M di )! , (M !)I 2

which results in (6.7). For the case I = 2, we must have d1 = d2 . In this case (6.6) simplies to M! . d1 !(M d1 )!
M d1 M d1 M d2

2 (d) =

(6.8)

Hence we have 2 (d) = 2 (d)

2 i=1 M di

1
M d1

(6.9)

In general, the function I (d) has a complex structure. Therefore we obtain an upper bound on I (d) by rst upper bounding I (d). Lemma 6.2.2 The function I (d) is upper bounded by the inequality
I

I (d) exp M

gI (1 , 2 , . . . , I )

i i
i=1

(6.10)

100

where gI (1 , 2 , . . . , I ) = ln
I i=1 (1

+ e i ) + 2

I i=1 (1

e i )

(6.11)

i = di /M , i 0, is the normalized Hamming weight of the vector w i , and the i , i = 1, . . . , I, are arbitrary constants. Proof: We present the proof for the I = 6 case, but generalization to an arbitrary I is straightforward. Multiply each of the summands in (6.6) by exp
6

i1 =1

i 1

{i2 }

2 ({i1 , i2 }) +

{i2 ,i3 ,i4 }

where each i1 is an arbitrary constant. Observe that this does not change the sum in (6.6) by virtue of the constraints of (6.5). We obtain an upper bound on 6 (d) by summing over all sets {0 , {2 ({i1 , i2 })}, {4 ({i1 , i2 , i3 , i4 })}, 6 } satisfying (6.4) but not necessarily (6.5). The multinomial theorem, (6.6), and (6.12) together imply that
6 i=1 (1

4 ({i1 , i2 , i3 , i4 }) + 6 di1

(6.12)

6 (d)

+ e i ) + 2

6 i=1 (1

e i )

6 i=1

i di

(6.13)

Inequality (6.10) follows from (6.11) and (6.13).

From Lemmas 6.2.1 and 6.2.2 we obtain the following upper bound on the probability I (d) I (d) exp M GI ([1,I ] , [1,I ] )
I i=1 M M i

(6.14)

101

where
I

GI ([1,I ] , [1,I ] ) = gI (1 , 2 , . . . , i ) [1,I ] = (1 , 2 , . . . , I ), [1,I ] = (1 , 2 , . . . , I ).

i i ,
i=1

(6.15) (6.16) (6.17)

6.2.3 Lower bound on row distance For any code in CP (3, 6, M ), the entries H T i (t) of the syndrome former are 2M M matrices composed of two M M permutation matrices. Hence the calculation of each of the terms t , t = 1, 2, . . . , L + 2, is identical to that of I (d) above . Let t = (t , t ), t = 1, . . . , L, where t
(h) (0) (1) (h)

= dt /M is the normalized
(0) (1)

(h)

Hamming weight of v t , t = 1, . . . , L, h = 0, 1, and let i (t) = (i (t), i (t)), t = 2, . . . , L + 1, i = 0, 1, 2, where i (t) and i (t) are arbitrary constants. Now from (6.9) and (6.14) it follows that t satises 1
M (0) M 1 (0) (1)

1 = L+2 = 2 L+1 t

(6.18) (6.19) (6.20) (6.21) , (6.22)

1
M (0) M L

exp [G4 ([0 (2), 1 (2)], [2 , 1 ])]

2 t=1 1 M h) h=0 M ( t L t=L1 1 M h) h=0 M ( t

exp G4 ([1 (L + 1), 2 (L + 1)], [L , L1 ]) exp G6 ([0 (t), 1 (t), 2 (t)], [t , t1 , t2 ])

t i=t2 1 M h) h=0 M ( i

t = 1, 2, L + 1, L + 2.

102

v1 v1 v2 v2

(0)

1 (2)

(0)

2 (3)

(0)

1 1
(0)

(0)

(1)

1 (2)

(1)

2 (3)

(1)

(1)

(0)
0 (2)
(0)

1 (3)

(0)

2 (4)

2 2 3 3

(0)

(1)

0 (2)

(1)

1 (3)

(1)

2 (4)

(1)

(1)

v (2j-2,1) 3 (2j-2,2) (2j,1) v (1) 3 (2j,2)

(0)

0 (3)

(0)

(0)

1 (4)

(0)

0 (3)

(1)

1 (4)

(1)

(1)

Figure 6.4. Set of parameters 3 and 3 for calculating the function F3 ([1,3] , [1,3] )

Note that any code sequence v [1,L] must have 1

(0)

= 1

(1)

and L = L . Let

(0)

(1)

[1,L] = (0 (2), 1 (2), 0 (3), . . . , 2 (L), 1 (L + 1), 2 (L + 1)) and [1,L] = (1 , . . . , L ). Now from (6.2) and equations (6.18)-(6.22) it follows that [1,L] can be upper bounded as [1,L] where exp F ([1,L] , [1,L] )
L t=1 1 M h) h=0 M ( t 3

M (1) M 1

M (1) , M L

(6.23)

F ([1,L] , [1,L] ) =
L

G4 ([0 (2), 1 (2)], [2 , 1 ]) +

t=3

G6 ([0 (t), 1 (t), 2 (t)], [t , t1 , t2 ]) (6.24)

+ G4 ([1 (L + 1), 2 (L + 1)], [L , L1 ]).

103

Figure 6.4 shows the parameters needed to calculate F ([1,3] , [1,3] ), where it can be seen that the parameters i (t) and i (t) are associated with the permutation matrices P i (t) and P i (t) that comprise the matrix H i (t)4 . The expected number E ([1,L] ) of code sequences v [1,L] in a code of the ensemble C (M, 2M, 3) having normalized weight composition [1,L] is given by
L 1 (0) (1) (0) (1)

E ([1,L] ) = [1,L]
t=1 h=0

M (h) . M t

(6.25)

Using (6.23) we can obtain an upper bound on E ([1,L] ). Lemma 6.2.3 The function E ([1,L] ) can be upper bounded using the inequality E ([1,L] ) 2e2 M where
(0) 3H (1 ) L 1 1 (h) 2L+1

exp M F ([1,L] , [1,L] ) ,

(6.26)

F ([1,L] , [1,L] ) = F ([1,L] , [1,L] )

(0) 3H (L )

H (t ) (6.27) (6.28)

t=2 h=0

H () = ln (1 ) ln(1 ). Proof: From (6.25) and (6.23) we have

L 1

E ([1,L] ) = [1,L]
t=1 h=0

M (h) M t M (1) M 1 M (1) . M L

(0)

4 (1) P i (t).

exp F ([1,L] , [1,L] )

L t=1 1 M h) h=0 M ( i 2

(6.29)

Since J = 1, we omit the second index in the superscript and simply write P i (t) and

104

We then use the inequality (see [12]) M (h) M t 1 2M t (1 t ) exp

(h) (h) 1 (h) (h) 12M t (1t ) (h) (h)

>

exp[M H (t )], if t

= 0, 1, (6.30)

and the simple equality M (h) M t

(h)

= 1, if t
(h)

= 0, 1,

(6.31)
(h)

to bound the combinatorial terms. Now dt dt /M . Hence, if t in this case we have

(h) (h) (h) (h)

{0, 1, . . . , M 1, M } and t
(h)

= 0, 1, it follows that t

{1/M, . . . , (M 1)/M }, and

2M t (1 t ) exp Hence M (h) M t

(h)

1 12M t (1 t ) 1 2e2 M
(h) (h)

<

2M e.

(6.32)

>

exp[M H (t )]
(0)

(h)

(6.33) = 1
(1)

for all t . Combining (6.27), (6.29), (6.33) and the equalities 1 L = L we obtain (6.26).
(0) (1)

and 2

In the following analysis, we use the fact that if, for a normalized weight composition [1,L] of v [1,L] , there exists a vector [1,L] such that F ([1,L] , [1,L] ) < 0, then the expected number of v [1,L] with normalized weight composition [1,L] tends to zero as M tends to innity. The tightest bound on E ([1,L] ) is obtained by minimizing F ([1,L] , [1,L] ) in (6.26) with respect to [1,L] . Letting

U ([1,L] ) = min F ([1,L] , [1,L] ), [1,L]

(6.34)

105

we dene ([1,L] ) = where [1,L] denotes

L t=1

max U ([1,L] ), [1,L] : [1,L] =[1,L]

1 h=0 (h)

(6.35)

t , the normalized Hamming weight corre-

sponding to [1,L] . We now calculate an upper bound on the average number of code sequences v [1,L] with total normalized Hamming weight A([1,L] ) = {[1,L] : [1,L] = [1,L] . Let

[1,L] = [1,L] } be the set of [1,L] with normalized Hamming

weight [1,L] . From Lemma 6.2.3 we have the inequality 2e2 M

A([1,L] ) 2L+1

A([1,L] )

E ([1,L] )

exp M U ([1,L] ) exp M ([1,L] ) ln M 2e2 + (2L + 1) M M

(6.36) (6.37) , (6.38)

(h)

2e2 M
A([1,L] )

2L+1

exp M

([1,L] ) + (4L + 1)

where we have used (6.35) in obtaining (6.37) from (6.36) . For any t, h, dt M t
(h)

can take on at most M values. Therefore, the total number of normalized

weight compositions [1,L] is upper bounded by M 2L , and thus we obtain (6.38) from (6.37). Hence, if ([1,L] ) < 0, the expected number of code sequences v [1,L] with total Hamming weight [1,L] tends to zero as M tends to innity. Let [1,L] be such that ([1,L] ) < 0 for all [1,L] [1,L] . Then it follows from (6.36)(6.38) that the expected number of code sequences v [1,L] with normalized Hamming weight less than [1,L] can be upper bounded by the inequality
M [1,L] M [1,L]

E ([1,L] ) <
d=1
d A( M )

2e2 M
d=1
d A( M )

2L+1

exp M

d M

(6.39)

106

Since the total number of normalized weight compositions [1,L] is upper bounded by M 2L , the outer summation involves at most M terms, and
d M

< 0 for

all terms in the summation, it follows that as M the expected number of code sequences with normalized Hamming weight less than [1,L] in the ensemble CP (3, 6, M ) tends to zero. Since the above arguments are independent of t, they are valid for any code sequence v [t+1,t+L] , irrespective of t. Hence, if there are S possible starting positions for code sequences of length L, it follows that the expected number of sequences with weight less than [1,L] can be upper bounded by
M [1,L] M [1,L]

S
d=1 A(
d M

E ([1,L] ) < )
d=1 A(
d M

2e2 M )

2L+1

exp M

d M

ln S M

(6.40)

Typically the number of starting positions S . Hence, if M faster than ln S then it follows that [1,L] M is a lower bound on the row distance d(L). Finally, the best lower bound L M on the row distance is determined by calculating the largest [1,L] for which the above holds, i.e., L = max {[1,L] : ([1,L] ) < 0, if [1,L] [1,L] } = max {[1,L] : = max {[1,L] : max U ([1,L] ) < 0, if [1,L] [1,L] } [1,L] : [1,L] =[1,L]

(6.41) (6.42)

min F ([1,L] , [1,L] ) < 0 if [1,L] [1,L] }. max [1,L] : [1,L] =[1,L] [1,L] (6.43)

6.3 A lower bound on free distance As noted previously, the free distance dfree satises dfree = minL d(L), L 1. Hence, a lower bound on the free distance is M minL L . Thus it would be 107

necessary to obtain a lower bound on L for arbitrary L in order to use the row distance bound to provide a lower bound on dfree . However, F ([1,L] , [1,L] ) is a complicated function of [1,L] and [1,L] and the max-min problem in (6.43) is dicult to solve analytically. This section describes an analytical technique to lower bound the row distance for any L. From this lower bound it follows that the free distance increases linearly with constraint length. The function F ([1,L] , [1,L] ) is upper bounded by a function F ([1,L] , [1,L] ) and then the max-min of F ([1,L] , [1,L] ) is calculated to obtain the desired lower bound. The function F ([1,L] , [1,L] ) is chosen so that determining max[1,L] : [1,L] =[1,L] min[1,L] F ([1,L] , [1,L] ) is analytically tractable. Thus, we weaken the bound in order to solve it for any L We make use of the inequalities: 1 2 (a + a2 2 ), 2 1 1 ai aj , ai < 1, i = 1, . . . , 4, 6 i<j 1 15 ai aj , ai < 1, i = 1, . . . , 6.
i<j

a1 a2 a1 a2 a3 a4

(6.44) (6.45) (6.46)

a1 a2 a3 a4 a5 a6

Expanding (6.11) with I = 4, for i < 0 we obtain 7 4

4

g4 (1 , 2 , 3 , 4 ) = ln(1 +
i<j

ei ej + e1 e2 e3 e4 ) < ln(1 +

e2i )
i=1

= g4 (1 , 2 , 3 , 4 ),

(6.47)

and similarly 31 g6 (1 , 2 , 3 , 4 , 5 , 6 ) < ln(1 + 6

6

e2i ) = g6 (1 , 2 , 3 , 4 , 5 , 6 ).
i=1

(6.48)

108

Analogous to the denition in (6.15), we dene

I

GI ([1,I ] , [1,I ] ) = gI (1 , 2 , . . . , I )

i i .
i=1

(6.49)

Note that GI ([1,I ] , [1,I ] ) > GI ([1,I ] , [1,I ] ) and hence it follows that F ([1,L] , [1,L] ) =
L

G4 ([0 (2), 1 (2)], [2 , 1 ]) +

t=3

G6 ([0 (t), 1 (t), 2 (t)], [t , t1 , t2 ])

(0) 3H (1 )

+ G4 ([1 (L + 1), 2 (L + 1)], [L , L1 ])

(0) 3H (L )

L1

H (t )

(h)

t=2 h=0

> F ([1,L] , [1,L] ).

(h)

(6.50)

In the region where F ([1,L] , [1,L] ) is dierentiable with respect to i (t), the minimum of F ([1,L] , [1,L] ) with respect to [1,L] is attained when F ([1,L] , [1,L] )
(h) i (2)

= =

g4 (0 (2), 1 (2))
(h) i (2)

2i = 0, i, h = 0, 1,
(h)

(h)

(6.51)

F ([1,L] , [1,L] )
(h) i (L

g4 (1 (L + 1), 2 (L + 1))
(h) i (L

+ 1)

+ 1)

L+1i = 0, i = 1, 2, h = 0, 1, (6.52)

F ([1,L] , [1,L] )
(h) i (t)

g6 (0 (t), 1 (t), 2 (t))

(h) i (t)

ti = 0, t = 3, . . . , L, i = 0, 1, 2, h = 0, 1. (6.53)

(h)

Let U ([1,L] ) = min F ([1,L] , [1,L] ), [1,L] (6.54)

109

i.e., U ([1,L] ) = F ([1,L] , [1,L] ), where [1,L] is chosen to satisfy (6.51)(6.53). Once again we seek max min F ([1,L] , [1,L] ) = max U ([1,L] ). [1,L] : [1,L] =[1,L] [1,L] [1,L] : [1,L] =[1,L] To nd the conditional maximum of U ([1,L] ), we use the Lagrange method of undetermined multipliers. First we introduce a function (6.55)

U ([1,L] ) = U ([1,L] ) +

[1,L] [1,L] ,

(6.56)

where is the Lagrange multiplier. Then the stationary points satisfy U ([1,L] ) 1
(0) (0) 1 (2) (0) 2 (3)

= 2 = 2 =

+ +

+ 3 ln

(0) (0)

U ([1,L] ) L t
(0)

1 1

+ = 0,
(0) (0)

(6.57) (6.58)

(0) 0 (L)

(0) 1 (L

+ 1) + 3 ln
(h) 2 (t

U ([1,L] )
(h)

1 L

+ = 0, t
(h)

(h) 0 (t)

(h) 1 (t

+ 1) +

+ 2) + 2 ln

+ = 0, (h) 1 t t = 2, . . . , L 1, h = 0, 1.(6.59)

Note that [1,L] is chosen to satisfy (6.51)(6.53). Using (6.53) and (6.59), following some algebraic manipulations it can be shown that t 2, . . . , L 1. The main advantage of using F ([1,L] , [1,L] ) is that we can solve (6.51)(6.53) to obtain [1,L] explicitly. Using the values of [1,L] so obtained, it can be shown (see Appendix for details) that the conditional maximum of U ([1,L] ) is attained
(0)

= t

(1)

for t =

110

at the point [1,L] = (0, 0, 3 , 0, . . . , 0, L2 , 0, 0), with 3

(0)

= L2 for L > 6.

(0)

Evaluating the expression for U ([1,L] ) (see Appendix) at this point of maximum , it follows that the lower bound on the Lth (L > 6) order row distance is obtained at ana satisfying the equation 3 ln 2 ana 3 31 + H( ana ) = 0. 2 (6.60)

Solving the above equation numerically, we obtain ana = 0.005, which implies that, for L > 6, d(L) 0.005M . For L < 6, it can be similarly shown that d(L)
0.005M . Thus we have d(L) ana M for all L. Hence dfree LDPCC (3, 6) , where LDPCC (3, 6) = ana /6 = 0.0008, and the free distance increases linearly

with constraint length.

6.4 Results and discussion For small values of L it is possible to use numerical methods to solve the maxmin problem of (6.43) and obtain L . Following exactly the same analysis as in the previous section the symmetry condition t = t , for t = 2, . . . , L1, can also be shown to hold in the calculation of max[1,L] : [1,L] =[1,L] min F ([1,L] , [1,L] ). [1,L] This reduces the number of variables by half and simplies the numerical evaluation of F ([1,L] , [1,L] ). In Figure 6.5 the ratio L /6, a lower bound on the row distance to constraint length ratio d(L)/6M , is plotted as a function of L for L 16. Observe that, for L = 1, the code sequence has weight 1 /6 = 1/3, i.e., d1 = 2M . The ensemble CP (3, 6, M ) is composed of permutation matrices, and hence in any code there always exists a code sequence of the form (. . . , 0, v t , v t , 0, . . . ) with v t
(1) (0) (1) (0) (0) (1)

v t = 1, where 1 is the M dimensional all-one vector. In fact, for any J and K , 111

0.4 0.35 0.3 0.25 0.2 0.15 0.1

PSfrag replacements

L /6

0.05 0

10

15

Figure 6.5. Lower bound on row distance to constraint length ratio as a function of L.

d(L) 6M

such a code sequence exists for any code in the (J, K ) regular LDPC convolutional code ensemble CP (J, K, M ). Figure 6.5 shows that the lower bound on row distance to constraint length ratio L /6 is virtually unchanged from L = 9 to L = 16. In Fig. 6.6 we plot the normalized weight composition [1,L] satisfying [1,L] = L that maximizes

U ([1,L] ) for L = 9, 11, 13, 15 (Fig. 6.6(a)) and L = 10, 12, 14, 16 (Fig. 6.6(b)). We nd that the maximizing weight composition satises t
(0)

= Lt . i.e.,

(0)

the normalized Hamming weights are symmetrically distributed about the middle block. To emphasize this symmetry we have used the distance from the middle block in Fig 6.6(a) and (b) and have only plotted the rst component of t , i.e., t , t = 1, . . . , L.
(0)

112

0.012

Distance ratio t

(0)

0.01 0.008 0.006 0.004 0.002 0 8 6 4 2 0 2 4

L=9 L=11 L=13 L=15

Distance from middle block (a)

0.012 0.01 0.008 0.006 0.004 0.002 0 8 6 4 2 0 2 4 6 8 L=10 L=12 L=14 L=16

PSfrag replacements

Distance ratio t

(0)

Distance from middle block (b)

Figure 6.6. Maximizing weight vector [1,L] for dierent L.

113

The maximizing weight composition depends on L, though the overall weight is the same. In each case the maximizing weight composition [1,L] is largest in the middle and both ends have almost zero weight. It is reasonable to conclude that
a similar behavior holds for larger L, i.e., L = 16 = 0.5166, for all L > 16, and

that the most likely normalized weight composition has weight concentrated in the middle with small non-zero weights at the ends. This conclusion leads us to the result that the free distance satises dfree LDPCC (3, 6) , where LDPCC (3, 6) = (0.5166/6) = 0.0861. This bound is weaker than the Costello coecient C (0.5) = 0.39 for rate R = 1/2 codes. However, LDPCC (3, 6) is almost four times larger than Gallagers coecient G (3, 6) = 0.023 for (3, 6) block codes. Interestingly, for the general class of rate R = 1/2 codes, C (0.5) = 0.39 is also almost four times larger than the corresponding GV (0.5) = 0.11. The lower bound on free distance obtained here is much stronger than the analytically calculated value in Section 6.3. However, the analytical result provides a rigorous proof that the free distance of the ensemble C (M, 2M, 3) increases linearly with constraint length. Observe that in this case the most likely normalized weight composition has weight concentrated in the middle, which was not the case with the weaker bound of Section 6.3. This suggests that an improved analytical bound may be obtained by nding tighter approximations in (6.47) and (6.48) that still permit evaluation of the max-min problem for any L. In Table 6.1 we present the numerically calculated value of the parameter LDPCC (J, K ) for the (3, 6) and (4, 8) cases. In both cases the asymptotic distance bound ratio is several times larger than for the corresponding LDPC block codes.

114

TABLE 6.1 DISTANCE BOUND FOR LDPC BLOCK AND CONVOLUTIONAL CODES FOR DIFFERENT (J, K ). (J, K ) (3,6) (4,8) LDPCC (J, K ) 0.0861 0.1908 G (J, K ) 0.023 0.0627

As has already been observed in the case when J and K are relatively prime, the convolutional code ensemble considered in the paper is identical to the block code ensemble of [49]. Hence the free distance in this case satises the same lower bound, i.e., dfree G (J, K ) . The main obstacle in evaluating LDPCC (J, K ) for other values of (J, K ) is the numerical solution of the max-min optimization problem of (6.43). For larger values of J and K , both the number of parameters and the length at which the bound saturates increase. Further, for rates R = 1/2, the symmetry conditions do not necessarily hold, so that the number of parameters is larger. In these cases, typically, the numerical optimization algorithms become stuck in local maxima in evaluating the max part of the problem. As has already been noted, the convolutional codes as dened always have code sequences with the symbols in exactly two of the blocks being the all-ones vector. This limits the asymptotic free distance ratio to LDPCC (J, K )
2 . K

This is a

severe restriction for codes with large K . However, we can expurgate such code sequences from the ensemble at the expense of a small loss in rate. For example, in the (J, 2J ) case we can x the rst information symbol in each block to be a zero, so that the rate is reduced to R =
M 1 . 2M

In general, the problematic code

sequences can be avoided by xing in each block with information symbols, one 115

information symbol to zero. The block code ensemble of [49] also suers from the same problem and this can be overcome as described here. With this restriction the block code ensemble of [49] satises Gallagers bound for arbitrary (J, K ).

116

CHAPTER 7 CONVERGENCE ANALYSIS ON THE ERASURE CHANNEL

In [50], the existence of a sequence of (J, K ) regular LDPC convolutional codes for which an arbitrary number of independent iterations is possible was demonstrated. Based on this result, it follows that the threshold of (J, K ) LDPC block codes is a lower bound on the threshold of (J, K ) LDPC convolutional codes for any message passing algorithm and channel. However, simulation results on the AWGN channel (see Chapter 3 and [51]) indicate the possibility that LDPC convolutional codes may have better thresholds than corresponding LDPC block codes. In this chapter, we analyze the iterative belief propagation decoding of terminated convolutional codes in the ensemble CP (J, K, M ), the ensemble for which distance properties were analyzed in the previous chapter, on the erasure channel. Termination leads to a structured irregularity in the Tanner graph and this in turn results in signicantly better thresholds for the terminated convolutional codes compared to the corresponding randomly constructed regular and irregular LDPC block codes. We focus on the ensemble CP (J, 2J, M ), i.e., (J, 2J ) LDPC convolutional codes. The Tanner graph structure of codes in CP (J, 2J, M ) is described in Section 7.1. In section 7.2, we prove that LDPC convolutional codes in CP (J, 2J, M ) can be terminated. The structured irregularity in the Tanner graph of the terminated convolutional codes is then described. Section 7.3 describes the analysis of BP 117

PSfrag replacements

t+1

t+2

0 (t)

(0)

0 (t)

(1)

1 (t + 1)

(0)

1 (t + 1)

(1)

2 (t + 2)

(0)

2 (t + 2)

(1)

vt

(0)

vt

(1)

Figure 7.1. Tanner graph connections of the 2M symbol nodes at time t.

decoding of codes in CP (J, 2J, M ) on the BEC. The chapter concludes with a summary of the results.

7.1 Tanner graph description Consider for example the (3, 6) case. At each time instant the Tanner graph of a code in the ensemble CP (3, 6, M ) (see Chapter 5 for the ensemble description) is comprised of c = 2M symbol nodes and c b = M constraint nodes. A symbol node in the Tanner graph has degree 3 and a constraint node degree 6. Further, each node can be connected at most ms = 2 time units away. Figure 7.1 illustrates how the 2M symbol nodes at time t, i.e., the M information symbols v t and the M parity symbols v t , are connected through dierent permutation matrices to constraint nodes at time t, t + 1, and t + 2. Analogously, Figure 7.2 shows how constraint nodes at time t are connected through dierent permutation matrices to symbol nodes at times t 2, t 1, and t.
(1) (0)

118

PSfrag replacements v t2
(0)

v t2

(1)

v t1

(0)

v t1

(1)

vt

(0)

vt

(1)

2 (t)

(0)

2 (t)

(1)

1 (t)

(0)

1 (t)

(1)

0 (t)

(0)

0 (t)

(1)

Figure 7.2. Tanner graph connections of the M parity-check equations at time t.

For practical applications, a convolutional encoder starts from a known state (usually the all-zero state) and after the data to be transmitted has been encoded, the encoder is terminated back to the all-zero state using tail bits. Trellis based decoding algorithms for convolutional codes can exploit this known starting and ending state of the encoder while decoding. As we shall see, the message passing decoder can also exploit the fact that the encoder starts and ends in the all-zero state. For LDPC convolutional codes the syndrome former matrix can be used to determine the tail bits needed for termination. In the next section we describe this procedure and show that it can be used for termination of almost all codes in the ensemble CP (J, 2J, M ). 7.2 Termination of LDPC convolutional codes Let the vector v L = (v 1 , v 1 , . . . , v L , v L ), v j
(0) (0) (1) (0) (1) (h)

FM 2 , for j = 1, . . . L,

h = 0, 1, with v 1 = 0 denote the systematically encoded codeword corresponding to an information block of length L starting at t = 1. For each time instant

119

t, 1 t L, v L comprises an information block v t

(1)

(0)

of length M and the

corresponding length M parity block v t . For a code C in CP (J, 2J, M ) dene the partial syndrome vector associated with v L at time t as st = (s(t,1) , . . . , s(t,ms ) ) where s(t,j ) = [v t , v t ]H T j (t + j ) + s(t1,j +1) , j = 1, . . . , ms ,
(0) (1)

(7.1)

s(0,j ) = 0 for all j , and s(t,j ) = 0, 1 t L, j > ms . Each s(t,j ) , j = 1, . . . , ms , is an M dimensional vector. Encoding can be carried out by solving 0 = [v t , v t ]H T 0 (t) + s(t1,1) , to determine the parity-block v t
(1) (0) (1)

(7.2)

for each t. Thus, knowledge of the partial

(0)

syndrome vector st1 at time t 1 and the information block v t

makes the

future independent of the past. Hence a trellis, the syndrome former trellis, can be dened with the partial syndrome vectors as states. Equations (7.1) and (7.2) dene the transitions, i.e., the next state and branch labels, of this trellis. The initial syndrome former state s0 is the zero vector, i.e., we encode from the all-zero state. We then seek tail bits v [L+1,L+ ] = (v L+1 , v L+1 , . . . , v L+ , v L+ ), v L+t FM 2 , for t = 1, . . . , , h = 0, 1, so that the nal syndrome former state sL+ is equal to the zero vector, i.e., we terminate back to the all-zero state. This is equivalent to solving the linear equation v [L+1,L+ ] H T [L+1,L+ ] = (sL , 0, . . . , 0)1( +ms )M ,
(0) (1) (0) (1) (h)

(7.3)

120

where the 2 M ( + ms )M matrix H T [L+1,L+ ] is

T H 0 (L

HT [L+1,L+ ]

+ 1) . . . .. .

HT ms (L

+ ms + 1) .. .

HT 0 (L + )

. . . HT ms (L + + ms ) (7.4)

LDPC convolutional codes are typically encoded using the technique described in Chapter 3. In this encoding technique the partial syndrome state vectors are not explicitly calculated. However to determine the tail bits we only need the nal partial syndrome state vector sL . This is determined as the last (c b)ms bits
T of the product v L H T [1,L] , where the 2LM (L + ms )M matrix H [1,L] is dened

similar to H T [L+1,L+ ] in (7.4). We now prove the following property of the partial syndrome state vector for codes in CP (J, 2J, M ). Lemma 7.2.1 For any code C in CP (J, 2J, M ) the partial syndrome state vector st associated with v L at time t is such that each of its components s(t,j ) has even weight. Proof: We use induction on t to prove the lemma. Encoding starts from the all-zero state, i.e., the initial syndrome former state s0 = 0. Since the matrix HT 0 (1) is composed of two permutation matrices each constraint equation involves a single symbol from each of v 1 and v 1 . Therefore for (7.2) to be satised, it follows that for each one in v 1 v1
(0) (0) (0) (1)

there is a one in v 1

(1)

and vice-versa. Thus,

and v 1
(0)

(1)

have the same weight. For any t 1, the ith component of the
(1) (0)

T vector [v t , v t ]H T j (t + j ) is zero i the ith column of H j (t + j ) involves an even

number of ones from v t

and v t

(1)

and is one otherwise. Let r be the number

(0)

of columns of H T j (t + j ) involving a one from both v t

and v t . Then, since

(1)

HT j (t + j ) is composed of two permutation matrices it follows that the weight of 121

[v t , v t ]H T j (t + j ) equals the sum of the weights of v t

(0) (1)

(0)

(1)

(0)

and v t

(1)

less 2r . Hence,

the vector [v t , v t ]H T j (t + j ) has even weight i the total number of ones in vt

(0)

and v t

(1)

is even. Therefore, it follows from (7.1) that s(1,j ) has even weight.

Suppose now that each component of st has even weight. Then from (7.2) and the fact that the vector [v t+1 , v t+1 ]H T 0 (t + 1) has even weight i the total number of ones in v t+1 and v t+1 is even it follows that v t+1 and v t+1 must both have odd or even weight. Hence, the vector [v t+1 , v t+1 ]H T j (t + 1 + j ) has even weight. Now by the induction assumption s(t,j +1) has even weight and it follows from (7.1) that s(t+1,j ) also has even weight. 2
(0) (1) (0) (1) (0) (1) (0) (1)

We claim that for almost all codes in CP (J, 2J, M ) if = ms + 1, then (7.3) has a solution for any valid partial syndrome state vector sL , i.e., any sequence v L can be terminated with a tail of 2M = 2M (ms + 1) bits. For simplicity we consider the ensemble CP (3, 6, M ), i.e., the (3, 6) case, in detail. In this case with = ms + 1 = 3 the tail-bits are determined by solving the equation v [L+1,L+3] H T [L+1,L+3] = (sL , 0, 0, 0)15M , where the 6M 5M matrix H T [L+1,L+3] is
T H 0 (L

(7.5)

+ 1)

HT [L+1,L+3]

HT 1 (L

T T . ( L + 4) ( L + 3) H ( L + 2) H HT 2 1 0 T T T H 0 (L + 3) H 1 (L + 4) H 2 (L + 5) (7.6)

+ 2)

HT 2 (L

+ 3)

The sub-matrices H T i (t) is composed of two M M permutation matrices. Figure 7.3 shows the permutation matrices comprising H T [L+1,L+3] . The existence or

122

P 0 (t)

(0)

P 1 (t + 1) P 2 (t + 2)

(0)

(0)

P 0 (t)

(1)

P 1 (t + 1)

(1)

P 2 (t + 2)

(1)

P 0 (t + 1) P 1 (t + 2) (0) P 1 (t + 1) (0) P 2 (t + 2) (1) P 1 (t + 1) (1) P 2 (t + 2) (0) P 1 (t + 2) (0) P 2 (t + 3)

(0)

(0)

P 2 (t + 3)

(0)

P 0 (t + 1) P 1 (t + 2)

(1)

(1)

P 2 (t + 3)

(1)

(1) P 1 (t + 2) (1) P 2 (t + 3) (0) P 1 (t + 3) (0) P 2 (t + 4) (1) P 1 (t + 3) (1) P 2 (t + 4)

P 0 (t + 2)

(0)

P 1 (t + 3) P 2 (t + 4)

(0)

(0)

P 0 (t + 2) P 1 (t + 3) P 2 (t + 4)

(1)

(1)

(1)

Figure 7.3. The matrix H T [L+1,L+3] .

lack of solutions of (7.5) depends on the rank of H T [L+1,L+3] . We now proceed to show that H T [L+1,L+3] has almost full rank. For convenience let A denote the matrix H T [L+1,L+3] . With A viewed as a 6 5 block matrix, let A(i,j ) , i = 1, . . . , 6, j = 1, . . . , 5, denote the M M block matrix in the ith row and j th column of A. Note that the matrices A(i,j ) are either permutation matrices or the all-zero matrix. Consider the ensemble A consisting of matrices of the form A and with probability distribution inherited from the ensemble CP (3, 6, M ), so that each of the permutation sub-matrices comprising A is chosen independently and all M ! values are equally likely. We perform row operations on A to reduce it to diagonal form. We will use the following property

123

of permutation matrices: For any two permutation matrices P 1 , P 2 , there exists a permutation matrix P 3 such that P 1P 3 = P 2. (7.7)

This follows from the fact that a permutation matrix is invertible and the fact that the product of any two permutation matrices is another permutation matrix.
1 In fact P 3 = P 1 P 2.

Fig. 7.4(a) shows the matrix A. The row operations on A are carried out by viewing it as a 6 5 block matrix. Step 1: Let C (2,1) be such that A(1,1) C (2,1) = A(2,1) . Multiply each of the matrices A(1,j ) , j = 1, . . . , 6, in the rst row by C (2,1) and add the products to the matrices A(2,j ) , j = 1, . . . , 6, of the second row. Then the matrix in the rst column of the second row, i.e., in position (2, 1), of the resulting matrix is the all-zero matrix. The two non-zero entries in the second row of the resulting matrix, i.e., (2, 2), and (2, 3), are the sum of two permutation matrices. Further, each of these permutation matrices can be assumed to be chosen independently and equally likely. We now repeat the process with the third and fourth and fth and sixth rows of A so that the sub-matrices in positions (4, 2) and (6, 3) of the are the all-zero matrix. The matrix A obtained at the end of resulting matrix A these operations is shown in Fig. 7.4(b). is the sum of two permutation Step 2: The sub-matrix in position (2, 2) of A (1) (1) (2) (1) (2) matrices, say A (2,2) and A(2,2) . Let C (2,2) and C (2,2) be such that A(3,2) C (2,2) = (1) and A (3,2) C (2) = A (2) . Multiply each of the matrices A (3,j ) , j = 1, . . . , 6, A (2,2) (2,2) (2,2) (1) and add the products to the second row. Then the matrix in (1) + C by C (2,2) (2,2) position (2, 2) of the resulting matrix is the all-zero matrix. The resultant matrix is

124

A(6,5)

A(1,1) 0 0 0 0 A(1,1) 0 A(1,2) A(1,3)

A(1,2)

A(1,3)

(1,1) A (1,2) A (1,3) A

A(2,1) 0 0 0 0 0 0 0 A(2,3)

A(2,2) A(2,3)

(2,2) A (2,3) A

A(2,4)

0 0 Step 1 0 0 0 0 (4,3) A 0 (4,4) A 0 Step 2 0

A(3,2) 0 0 A(3,2)

A(3,3)

A(3,4)

(3,2) A

(3,3) A

(3,4) A

A(3,3)

A(3,4)

A(4,2)

A(4,3)

A(4,4)

A(4,4)

A(4,5)

A(2,3)

0 0 0

A(5,3) 0

A(5,4)

A(5,5)

(5,3) A (5,5) A

(5,4) A

A(5,3)

A(5,4)

A(5,5)

A(6,3) A(6,4) 0 0 0 A(6,5)

(6,4) A

(6,5) A

A(6,4)

A(6,5)

(a)

(b)

(c)

Figure 7.4. The matrices resulting from row operations on A.

125
A(1,1) 0 0 A(1,2) A(1,3) 0 0 0 A(2,4) A(2,5) 0 Step 3 0 0 0 A(3,2) A(3,3) A(3,4) 0 Step 4 A(4,4) A(4,5) 0 0 A(5,3) A(5,4) A(5,5) 0 0 0 A(6,4) A(6,5)

(d)

(e)

such that the sub-matrix in position (2, 3) is the sum of four permutation matrices and the one in position (2, 4) is the sum of two permutation matrices. We perform a similar operation with the fourth and fth rows so that position (4, 3) of the so obtained is shown in Fig. resulting matrix is the all-zero matrix. The matrix A 7.4(c). is the sum of four permuStep 3: The sub-matrix in position (2, 3) of A
(i) (2 tation matrices, say A ,3) , i = 1, . . . , 4. Once again we determine matrices i) (i) (i) ( C (2,3) , i = 1, . . . , 4, such that A(5,3) C (2,3) = A(2,2) , i = 1, . . . , 4. Then multi-

(5,j ) , j = 1, . . . , 6, by plying the matrices A

4 i=1

i) ( C (2,3) and adding the products

to the second row makes the sub-matrix in position (2, 3) of the resulting matrix equal to the all-zero matrix. The sub-matrices in position (2, 4) and (2, 5) of A are the sum of six and four permutation matrices respectively. The matrix A A is shown in Fig. 7.4(d). . The rst and sixth row of matrices are Step 4: Now reorder the rows of A left unchanged. The second row is moved to the fourth row, the third row to the second, the fourth to the fth and the fth to the third. The resulting matrix A is shown in Figure 7.4(e). Here
0

and

refer to the fact that the corresponding

2

entries are the all-zero and permutation matrices respectively,

, and

refer to the fact that the corresponding entries are the sum of two, four, and six, permutation matrices respectively. Observe that the 3 3 sub-matrix in the upper left corner of A in Figure 7.4(e) has full rank, i.e., 3M . Let B T denote the 3 2 sub-matrix in the bottom right corner of A , i.e., the matrix formed by the last three rows and two columns of A . In terms of symbols B = (b(i,j ) ), i = 1, . . . , 2M, j = 1, . . . , 3M, is a 2M 3M matrix. The sum of the rst M columns (in terms of symbols1 ) and the sum of the last M columns
1

The matrix B will be viewed in terms of symbols unless stated otherwise

126

of B T is the zero vector. Hence, the maximal rank of B T is 2M 2. The next lemma shows that the rank of B T is in fact 2M 2. Lemma 7.2.2 The probability that the sub-matrix B T of the matrix A has rank less than 2M 2 tends to zero as M . Proof: For convenience we operate with the matrix B . The rst step is to prove that the probability that the j th row of B ,

bj = b(j,1) , b(j,2) , . . . , b(j,3M ) , j = 1, 2, . . . , 2M,

(7.8)

is the zero vector tends to zero as M . We consider two cases: j [1, M ] and j [M + 1, 2M ]. Say j [1, M ]. The structure of B implies that the vector comprising the last M symbols of bj ,
(3)

bj = b(j,2M +1) , b(j,2M +2) , . . . , b(j,3M ) , j = 1, 2, . . . , M,

(7.9)

is the sum of two M dimensional vectors, each vector a row of a M M permutation matrix. Any row of a permutation matrix contains a single one. Hence, bj
(3)

is the zero vector i the single one in each of the two M dimensional vectors occur in the same position. Since the permutation matrices are chosen independently and uniformly a one is equally likely to occur in any of the M positions. Hence the probability that bj
(3)

is the zero vector is 1/M . The vector comprising the

middle M symbols of bj ,
(2)

bj = b(j,M +1) , b(j,2M +2) , . . . , , b(j,2M ) , j = 1, 2, . . . , M,

(7.10)

127

is the sum of four M dimensional vectors, each vector a row of a permutation matrix. By using a union bounding argument the probability that bj is the zero vector can be upper bounded by 3/M 2 . Finally, the vector comprising the rst M symbols of bj ,
(1) (2)

bj = b(j,1) , b(j,2) , . . . , b(j,M ) , j = 1, 2, . . . , M,

(7.11)

is the sum of six M dimensional vectors, each vector a row of a permutation matrix. The probability that bj
(1)

is the zero vector can be upper bounded by

(i)

15/M 3 . The vector bj is the zero vector i all three of its sub-vectors bj , i = 1, 2, 3, are zero vectors. Hence, the probability that bj is the zero vector is upper bounded by 45/M 6 . Then the probability that at least one of the row vectors bj , j = 1, 2, . . . , M, is the zero vector is upper bounded by 45/M 5 . Analogously the probability that at least one of the row vectors bj , j = M + 1, M + 2, . . . , 2M, is the zero vector is upper bounded by 3/M 3 . Consequently, the probability that at least one row of B is the zero vector is upper bounded by (45/M 5 ) + (3/M 3 ), which tends to zero as M . Now we estimate the probability that a linear combination a of d1 , d1 M/2
2

rows bj , j = 1, 2, . . . , M, and d2 , d2 M/2 rows bj , j = M + 1, M +

2, . . . , 2M, is the zero vector. Say

a = [y , z ]B ,

(7.12)

where the M dimensional vectors y and z have weights d1 and d2 respectively. Once again we split the vector a into three parts, a(i) , i = 1, 2, 3, where the M
2

x is the greatest integer less than or equal to x.

128

dimensional vector a(i) consists of the (i 1)M + 1th to the iM th symbols of a. The vector a(3) is a linear combination of d1 rows of the sub-matrix

B (1,3) = (b(i,j ) ), i = 1, 2, . . . , M, j = 2M + 1, 2M + 2, . . . , 3M,

(7.13)

and d2 rows of the sub-matrix

B (2,3) = (b(i,j ) ), i = M + 1, M + 2, . . . , 2M, j = 2M + 1, 2M + 2, . . . , 3M. (7.14)

(1)

The sub-matrix B (1,3) is the sum of two permutation matrices, say P (1,3) and P (1,3) and B (2,3) is the sum of two permutation matrices, say P (2,3) and P (2,3) . Now form the matrix C 3 by stacking the four permutation matrices as shown in Fig. 7.5(a). Then it follows that a(3) satises the equation a(3) = (y , y , z , z )C (3) .
(2) (1) (2)

(7.15)

Equation (7.15) is similar to (6.3) of the previous chapter and the technique of Section 6.2.2 can be used to upper bound the probability P (d1 , d2 )(3) that a(3) is the zero vector. From (6.14) it follows that exp M G4 ([1,4] , [1 , 1 , 2 , 2 ])
M d1 2 M d2 2 (3)

P (d1 , d2 )

(3)

<

(7.16)

where the function G4 ([1,4] , [1 , 1 , 2 , 2 ]) is as dened in (6.15), [1,4] is a four dimensional vector of constants and 1 =
d1 M

(3)

(3)

and 2 =

d2 . M

129

P (1,2) PSfrag replacements P (1,3)

(2) (1) (2)

(1)

P (1,2)
(3)

P (1,3)
(1)

P (1,2)
(4)

P (2,3)
(2)

P (1,4)
(1)

P (2,3)

P (2,2)
(2)

P (2,2)

(a)

(b)

Figure 7.5. The matrices C 3 and C 2 .

Similarly the vector a(2) can be expressed as a(2) = (y , y , y , y , z , z )C (2) ,

(7.17)

where the matrix C (2) (Fig. 7.5(b)) is now comprised of six permutation matrices. Hence, the probability P (d1 , d2 )(2) that a(2) is the zero vector is upper bounded by P (d1 , d2 )
(2)

<

exp M G6 ([1,6] , [1 , 1 , 1 , 1 , 2 , 2 ])
M d1 4 M d2 2

(2)

(7.18)

Similarly, the probability P (d1 , d2 )(1) that a(1) is the zero vector is upper bounded by exp M G10 ([1,10] , [1 , 1 , 1 , 1 , 1 , 1 , 2 , 2 , 2 , 2 ])
M d1 6 M d2 4 (1)

P (d1 , d2 )(1) <

(7.19)

130

Hence, the probability P (d1 , d2 ) that a is the zero vector is P (d1 , d2 ) = There are in total
M d1 M d2

3 i=1

P (d1 , d2 )(i) .

possible combinations of (y , z ). Hence, the probability

P (d1 , d2 ) that there exists a linear combination of d1 M/2 rows bj , j = 1, . . . , M, and d2 M/2 rows bj , j = M + 1, . . . , 2M, equal to the zero vector is upper bounded by the inequality exp M F (, 1 , 2 )
M d1 11 M d2 7

P (d1 , d2 ) < <

(7.20) ,(7.21)

2e2 M

exp M F (, 1 , 2 ) 11H (1 ) 7H (2 )

where
(1)

F (, 1 , 2 ) = G10 ([1,10] , [1 , 1 , 1 , 1 , 1 , 1 , 2 , 2 , 2 , 2 ])
(2) (3)

(7.22)

+ G6 ([1,6] , [1 , 1 , 1 , 1 , 2 , 2 ]) + G4 ([1,4] , [1 , 1 , 2 , 2 ]), = ([1,10] , [1,6] , [1,4] ),

(1) (2) (3)

(7.23)

and the function H () is dened in (6.28). The tightest bound is obtained by minimizing F (, 1 , 2 ) = F (, 1 , 2 ) 11H (1 ) 7H (2 ) with respect to for a xed 1 , 2 and if min F (, 1 , 2 ) < 0 for all 1 , 2 , then it follows that as M , P (d1 , d2 ) 0. In fact, it is sucient to check if max1 ,2 min F (, 1 , 2 ) < 0. This is easily veried numerically and hence P (d1 , d2 ) 0 as M . Consider now linear combinations a = (y , z )B with weight of y = d1 > M/2 and weight of z = d2 < M/2. Since the sum of the rst M rows of B is the zero vector it follows that we can replace the d1 rows of B in the linear combination corresponding to the d1 ones in y by the M d1 rows of B corresponding to the M d1 zeros in y without changing a. Equivalently, a = (1 + y , z )B , where 1 131

is the M dimensional all-ones vector. Now 1 + y has weight less than M/2 and as already observed such linear combinations do not exist. The other cases, i.e., d1 < M/2, d2 > M/2, and d1 > M/2, d2 > M/2, can be similarly dealt with. This completes the proof. We can now prove the following lemma. Lemma 7.2.3 The probability that the rank of any matrix A in the ensemble A is less than 5M 2 tends to zero as M . Proof: From the row operations described earlier it follows that A has rank 3M +rank(B T ). Lemma 7.2.2 implies that the probability that rank(B T ) < 2 2

3M 2 tends to zero as M . This proves the Lemma. be stated.

Using lemma 7.2.3 a theorem to guarantee the existence of solutions to (7.5) can

Theorem 7.2.1 The equation (7.5) has a solution for almost all codes in C P (3, 6, M ). Proof: Consider the matrix H T [L+1,L+3] . Since the sum of two permutation matrices is the zero vector it follows that the sum of the rst, third, and fourth block columns of H T [L+1,L+3] is the zero vector. Similarly, the sum of the second, third and fth block columns is the zero vector. This implies that there are at least two dependent columns in H T [L+1,L+3] or equivalently two dependent equations in (7.5). From lemma 7.2.3 it follows that for almost all codes these are the only two dependent equations in (7.5). Now lemma 7.2.1 implies that both components of the partial syndrome state sL have even weight, i.e., the sum of its components is zero. It is easy to check that the components of the 5M dimensional vector (s(L,1) , s(L,2) , 0, 0, 0) corresponding to the columns of H T [L+1,L+3] that sum to zero (i.e., the columns corresponding to the blocks of permutation matrices that sum

132

L+3

2M PSfrag replacements

t=1 time

t=2

t=L+2

t=L+3

Figure 7.6. Tanner graph of a terminated convolutional code obtained from C (M, 2M, 3). to zero) also sum to zero, i.e., the two dependent equations are consistent. Hence, it follows that (7.5) has a solution. Theorem 7.2.1 implies the main result of this section: Corollary 7.2.2 Almost all codes in CP (J, 2J, M ) can be terminated with a tail of duration ms + 1, i.e., 2M (ms + 1) bits. Further, the tail bits are determined by solving (7.3) Suppose that we wish to transmit LM information bits using a code from CP (J, 2J, M ). It follows from Corollary 7.2.2 that the terminated code has rate
J R = L/2(L + ms + 1) = 0.5/(1 + L ). Note that for L >> J , the rate loss is

negligible. In Figure 7.6 we show the Tanner graph of a terminated code obtained from a convolutional code in the ensemble CP (3, 6, M ). Observe that symbols are zero both before encoding begins, i.e., t = 1, and after termination, i.e., t = L + 3. 133

If a symbol node is known to be zero then it can be omitted (along with its associated edges) without aecting the Tanner graph. Thus in the Tanner graph of terminated codes from CP (3, 6, M ) symbol nodes at times t < 1 and t > L + 3 can be omitted, since these are known to be zero. It follows that, while all symbol nodes in Figure 7.6 have degree three, the constraint nodes can have degree either two, four, or six. Thus, even though the convolutional code is regular encoding and terminating to the all-zero state leads to a slight irregularity in the Tanner graph.

7.3 Decoding analysis on the erasure channel Figure 1.2 shows the binary erasure channel. The probability of an erasure is p and with probability 1 p we receive the transmitted symbol correctly. In each iteration a message passing decoder exchanges messages between the symbol nodes and the constraint nodes. On the erasure channel, a symbol node can be recovered correctly if its channel value or any of the messages from the constraint nodes to which it connects is not an erasure. Thus convergence of belief propagation decoding on the erasure channel can be analyzed by tracking the probability of erasure of the messages. This is straightforward to do as long as the messages exchanged during iterations are independent. The next theorem guarantees that the number of independent iterations possible on the Tanner graph of the block code produced by terminating convolutional codes from the ensemble CP (J, 2J, M ) can be made arbitrarily large. Theorem 7.3.1 Let
0

log M + log log a 2 log[(J 1)(2J 1)]

(7.24)

134

where a= (2J 1)J 2 [(J 1)(2J 1) 1]2 (7.25)

is a constant independent of M and 0 < < 1/2. Then there exists an M0 such that for all M > M0 the probability that the number of independent iterations for any symbol in the block code with L J information blocks obtained by terminating a randomly chosen code from CP (J, 2J, M ) satises < . Proof: Consider an arbitrary symbol s at time t in a randomly chosen code from the ensemble CP (J, 2J, M ). We form the decoding neighborhood of s one check node (along with the other symbol nodes to which the check node connects) at a time and obtain an upper bound on the probability that this leads to a dependency. The rst check node is uniformly chosen from the M check nodes at time t. The degree of the check node is less than or equal to K = 2J depending on t. For simplicity we assume that the check nodes added to the decoding tree at any step have degree equal to K , the worst possible. Any check node at time t is connected to (see Section 7.1) one symbol node at time t and two symbol nodes each from time t J to t 1. From the Tanner graph structure described in Section 7.1 it follows that these symbol nodes are distinct so that there is no dependency in this step. In the next step a check node uniformly chosen from the M check nodes at time t + 1 is added to the tree. This check node connects to one symbol node from time t, two symbol nodes each from time t (J + 1) to t 1 and two symbol nodes from time t +1. In this step dependencies can occur, for example if the new symbol node at time t coincides with the already existing symbol node at time t. The probability of this happening is 1/M . Similarly, the probability of dependencies 135
0

is less than

being caused by any of the newly added symbol nodes at time t (J + 1) to t 1 is 1/M . Note that the newly added symbols at times t + 1 cannot cause a dependency. Thus, the probability of a dependency being caused by the addition of the second check node is (K 3)/M . At the nth step of this process a check node and the associated 2J 1 symbol nodes are added to the tree. Let di (n), i = 1, . . . , 2J 1, be the number of dangerous symbols (i.e., those leading to dependencies) and ei (n), i = 1, . . . , 2J 1, the number of prohibited symbols before the nth step of tree construction. Then the probability that the ith symbol of the nth step leads to a dependency is di (n)/(M ei (n)). Observe that di (n) (n 1), i = 1, . . . , 2J 1, and ei (n) (n 1), i = 1, . . . , 2J 1. Hence, it follows that the probability that the nth step leads to a dependency is upper bounded by (K 1)n/(M n). Let F be the number of check nodes in the decoding neighborhood of s for the probability that there are less than upper bounded by
F

iterations. Hence,

independent decoding iterations Pdep is

Pdep =

i=1 F 1 i=0 F

(K 1)

i1 M i+1 i M i

(7.26) (7.27) (7.28) (7.29)

(K 1)

x dx M x 0 F F = M (K 1) 0 log(1 0 ) . M M For any 0 < x < 1/2 we have x log (1 x) < x2 .

136

Hence, if F0 < 1/2, M then (7.29) implies that F0 M

2

(7.30)

Pdep < M (K 1)

(7.31)
2

K 1 (J 1)(K 1) 0 1 = J M (J 1)(K 1) 1 (J 1)(2J 1)2 0 a = M If

0

(7.32) (7.33)

satises (7.24) then (7.30) is satised and (7.33) implies that Pdep < .

Theorem 7.3.1 implies that the fraction of symbols for which the number of independent iterations is less than
0

is at most . We x these symbols, for

example by setting them to bit value 0, and do not transmit them. Fixing these symbols leads to a decrease in rate by a factor (1 ), but all remaining symbols are then guaranteed to have at least
0

independent iterations. By making M

suciently large can be made arbitrarily small, and the rate loss is therefore negligible. The asymptotic growth in
0

indicated by (7.24) is exactly the same

as obtained for LDPC block codes [12] except that the block length is replaced by the constraint length = 2JM . Consider now the th iteration of the decoding procedure, where 1 <
0.

The message sent from a constraint node at time t to a connected symbol node is an erasure if at least one of the symbols represented by the other neighboring nodes has been erased. As shown in Figure 7.7 (a), these symbol nodes belong to dierent time instants. It follows that the probability qt,tk that the message from a constraint node at time t to a symbol node at time t k is an erasure is
( )

137

PSfrag replacements

channel

t2 t2 t1 t1 (a)

t+1 t+2 (b)

Figure 7.7. Illustration of the messages (a) to a symbol node and (b) to a constraint node for the case J = 3.

equal to
( ) ( 1) k =k ( 1) 2

qt,tk = 1
( 1)

1 ptk,t

1 ptk ,t

k, k {0, ms } .

(7.34)

Here ptk,t denotes the probability that the message sent in the previous iteration, i.e., 1, from a symbol node at time t k to a constraint node at time t = 0 these values are initialized as pt ,t = 0 for
( ) (0)

corresponds to an erasure. For

all t and t. For terminated convolutional codes we also have pt ,t = 0 if t < 1 or t > L + ms + 1. This condition takes into account the lower constraint node degrees at the beginning and the end of the Tanner graph. The message from a symbol node to a constraint node is an erasure if all incoming messages from the neighboring constraint nodes and that from the channel are erasures. The ow of these messages is shown in Figure 7.7 (b). Thus we have
( ) ( )

pt,t+k = p
k =k

qt+k ,t ,

k, k {0, ms } .

(7.35)

138

PSfrag replacements

t=1

t=2

t=1

t=2

t=3

t=2

t=3

t=4

PSfrag replacements t=2 t t t t t t t t t =1 =2 =3 =4 =1 =2 =2 =2 =3 t=3

t=3

t=4

t=5

Figure 7.8. The rst level of computation trees for t = 1, 2, 3 with J = 3.

For regular LDPC block codes, the probability distribution of the messages exchanged in iteration are the same for all nodes regardless of their position

within the graph. Likewise, for the random irregular code ensembles considered in [52], the message distributions are averaged over all codes and only a single mixture density has to be considered for all constraint nodes and for all symbol nodes. In our case, while nodes at the same time instant behave identically, the messages from nodes at dierent times behave dierently and must all be tracked separately. Figure 7.8 shows the rst level of the decoding computation trees for the rst three symbol levels in the case J = 3. Although only the rst and last ms levels of constraint nodes have lower degrees, their eect evolves through the complete Tanner graph. To take this structure into account, in each iteration = 1, . . . ,
0 1,

rst (7.34) and then (7.35) is

139

TABLE 7.1 THRESHOLDS FOR THE ENSEMBLE CP (3, 6, M ) WITH DIFFERENT L AND FOR THE CORRESPONDING IRREGULAR BLOCK CODES. L + ms + 1 10 25 50 100 150 Rconv 0.350 0.440 0.470 0.485 0.490 p conv 0.504 0.488 0.488 0.488 0.488 p irr-blk 0.501 0.456 0.442 0.436 0.433 Iter. 161 3423 15912 40899 65884

applied for all t = 1, . . . , L + ms . Finally, for = over all k without exclusion of k .

the product in (7.35) is taken

7.4 Results To obtain convergence thresholds for terminated codes from the ensemble CP (J, 2J, M ) the recursive equations (7.34) and (7.35) can be evaluated numerically for all time instants and verifying if the erasure probabilities of all symbols converge to zero. The convergence threshold for an ensemble of codes can be found by testing this condition for dierent channel values p. Note that in addition to the node degrees J and K the value L is another parameter that inuences the result. In Table 7.1 we present the thresholds obtained for dierent L for the (3, 6) case. The rst column shows L + ms + 1 = L + 3 (the number of information bits per block is LM ), the second column shows the rate of the terminated convolutional code, and the third column gives the threshold 140

3 p conv . For L + ms + 1 = 10, the threshold is quite high, in fact larger than the

capacity of rate R = 1/2 codes. However, in this case there is a signicant rate loss and the terminated code has rate only R = 0.35. For larger L the threshold remains constant at p conv = 0.488. Tracking messages becomes increasingly dicult as we increase L but the behavior in Table 7.1 suggests that the rate of the terminated code can be made arbitrarily close to 0.5 without aecting the threshold. The fourth column of Table 7.1 shows the thresholds for random irregular LDPC block codes having the same degree distributions as the terminated convolutional codes (see [52] or [17] for a denition of degree distributions). Note that the degree distribution of the irregular code is determined by L and that they have the same rate as the terminated convolutional codes. The thresholds of the terminated convolutional codes are better than those of the irregular LDPC block codes. With increasing L the degree distribution of the terminated convolutional code tends to that of a (3, 6) regular LDPC block code. Therefore it is not surprising that the thresholds of the corresponding irregular LDPC block codes tend to the threshold of (3, 6) regular LDPC block codes. However, the thresholds of the terminated LDPC convolutional codes remain unchanged. Hence, the improvement in threshold can be attributed to the structure imposed on the Tanner graph by the convolutional nature of the code. The fth column in Table 7.1 shows for dierent L the number of iterations of density evolution required for the erasure probability of all messages to reach the breakout value. This reects the fact that for larger L the messages from the stronger nodes at the ends need more time to aect the symbols in the middle. Since for large L the required number of iterations seems to grow linearly with
3

Strictly speaking what we obtain is only a lower bound on the threshold.

141

L, according to Theorem 7.3.1, L cannot be increased more than logarithmically with M to guarantee a certain target bit erasure probability with density evolution. While the threshold itself is independent of the length L (if large enough), we cannot prove that for a xed M the eect from the ends of the Tanner graph carries through for arbitrary lengths. On the other hand, the logarithmic relationship between
0

and M is due to the independence assumption required for the

particular method of analysis and may be too restrictive in practice. For LDPC block codes, e.g., excellent performance can be observed with iteration numbers that exceed those given in Theorem 7.3.1 by far. The eect of the structured irregularity can also be visualized graphically. Consider a (J, K ) regular LDPC block code operating on the erasure channel. As already noted the probability distribution of the messages exchanged in iteration are independent of the position of the nodes in the graph. Let xi be the probability of erasure of a symbol node in the ith iteration. Then it follows that xi+1 = f (xi ) = p 1 (1 xi )K 1
J 1

(7.36)

In Fig. 7.9 we plot the function y = f (x) of (7.36) for J = 3, K = 6, at channel erasure probability p = 0.45. Observe that at this channel erasure probability (greater than the threshold for (3, 6) LDPC block codes), irrespective of the number of iterations, the probability of erasure cannot be made less than 0.356 (the xed point A in Fig. 7.9). In Fig. 7.9 we also plot the probability of erasure (trajectory C) of a symbol node in the middle for a terminated code (L = 50) in CP (3, 6, M ). In this case the erasure probability eventually reaches zero. Initially, the trajectory C is identical to that of (3, 6) LDPC block codes. However, the structured irregularity leads to a tunneling eect so that the point A is not a

142

0.5 0.45 0.4 0.35 0.3 0.25 0.2

PSfrag replacements C y y = f (x) (x) y=f

y=x

0.15 0.1 0.05 0 0 0.1 0.2

x Bbr

0.3

0.4

0.5

Figure 7.9. Illustration of the tunneling eect and the breakout value

143

xed point for the terminated convolutional code. Observe that the points of C are dense around the point A, i.e., a large number of iterations are required before the probability of erasure drops below 0.356, after which convergence is relatively fast. (x) is an upper bound on the function f (x) obtained Finally, the function y = f by just considering the lowest order term of f (x), as described in [53]. Clearly, if the probability of erasure for the (3, 6) regular block code can be made smaller than the breakout value, Bbr [53], it follows that all symbols of the block code can eventually be recovered. Since, the symbol nodes at all levels in the terminated convolutional code are at least as good as those in the corresponding regular block code the same is true for the terminated codes in CP (3, 6, M ). This implies that it is sucient to numerically track the probability of erasure for symbols at dierent levels till they reach Bbr to guarantee convergence. In Table 7.2 thresholds are presented for dierent J . In each case L is chosen so that there is a rate loss of 0.2%. The rst column in Table 7.2 shows the values J and K of the underlying convolutional code, the second column the rate of the terminated code, the third column the threshold obtained, and the fourth column the threshold for randomly chosen regular LDPC block codes with the same J and K as the convolutional code. The thresholds of the terminated convolutional codes are much better than for the corresponding block codes. This is reasonable since the constraint nodes at either end of the terminated convolutional codes have lower degrees than in the block code. Our observation has been that if the probability of erasure of symbol nodes at either ends tends to zero then after a sucient number of iterations the probability of erasure of symbol nodes at all time instants tends to zero.

144

TABLE 7.2 THRESHOLDS FOR THE ENSEMBLES CP (J, 2J, M ) WITH DIFFERENT J . (J, K ) (3,6) (4,8) (5,10) Rconv 0.499 0.499 0.499 p conv 0.488 0.497 0.499 p blk 0.429 0.383 0.341

Interestingly, the terminated convolutional codes with higher J have thresholds better than those with lower J . This behavior is dierent from that of randomly constructed (J, K ) regular LDPC block codes, where for a xed rate increasing J usually worsens the threshold. This is the case since for maintaining the same rate it is necessary to increase K accordingly. Note that symbol nodes with higher degrees are stronger than those with lower degrees, however lower degree constraint nodes are stronger than higher degree constraint nodes. For randomly constructed LDPC block codes larger constraint node degrees counteract the gain resulting from higher symbol node degrees, adversely aecting performance. However, in our case the codes with higher J still have strong constraint nodes of low degrees at either ends. Thus, the symbols at the ends are better protected for codes with larger symbol degrees and, hence, result in better thresholds. Oswald and Shokrollahi have constructed LDPC block code ensembles from sequences of right regular degree distributions4 for which the convergence thresholds of the bit erasure probability approach capacity as the maximum symbol node degree tends to innity [54]. According to Table 7.2, with increasing J the
4

all constraint nodes have the same degree

145

convergence thresholds of the terminated LDPC convolutional codes also seem to tend to the capacity limit of rate R = 1/2 codes. However, in this case the degree distributions are not right regular but left regular, i.e., all symbol nodes have the same degree. Further, following the analysis in [53], for the terminated convolutional codes it can be shown that at the calculated thresholds not only the bit but also the block erasure probability with iterative belief propagation decoding goes to zero as M goes to innity.

146

CHAPTER 8 CONCLUDING REMARKS

The dissertation deals with the design and analysis of LDPC convolutional codes. In the initial half of the dissertation construction techniques for LDPC convolutional codes were described. The latter half dealt with the asymptotics of an ensemble of LDPC convolutional codes. The results obtained in the dissertation indicate that LDPC convolutional codes provide a good alternative to LDPC block codes. We began by constructing LDPC convolutional codes by unwrapping the Tanner graphs of algebraically constructed QC codes. A performance improvement is obtained with the convolutional codes compared to the QC codes from which they are constructed. The same construction procedure can also be used to obtain LDPC convolutional codes from other classes of LDPC QC codes, for example the codes in [55]. In Chapter 4, a construction technique for obtaining a large class of time invariant LDPC convolutional codes by the polynomial extension of a base matrix was described. It is also possible to obtain QC codes from the constructed convolutional codes. The Tanner graph of the convolutional code is wrapped into the Tanner graph of the QC code. Once again the convolutional code performs better than the corresponding QC code.

147

The choice of the base matrix so as to optimize thresholds on the AWGN channel was explored in Chapter 4. Among the unanswered questions is, if it is possible to construct LDPC convolutional codes with the proposed technique for which an arbitrary number of independent iterations is possible. This is needed to guarantee the accuracy of the calculated thresholds. The best choice for the base matrix was obtained by randomly searching over several hundred matrices of a xed size. A more systematic procedure, as in [36], would probably lead to better results and also make the search feasible for base matrices of larger size. The chapter concluded with a proof of the fact that time invariant LDPC convolutional codes are not asymptotically good. The next chapter described the sliding window decoder and compared the decoders for LDPC block and convolutional codes. It was shown on the basis of simulations that LDPC convolutional codes compare favorably to LDPC block codes for the same processor complexity and delay. Chapter 5 also described an analog implementation of an LDPC convolutional decoder. An interesting observation here is that the performance of the analog decoder was slightly better than that of the digital decoder. A natural question is if for any LDPC code, analog decoders perform at least as well as a digital implementation. In Chapter 6, we turned our attention to the free distance of regular LDPC convolutional codes and proved a linear increase in free distance with constraint length for J > 3. The numerically estimated asymptotic distance ratio for the convolutional codes is several times larger than the corresponding ratio for LDPC block codes. The ensemble of Chapter 6 had a xed syndrome former memory

148

with large b and c. It would be interesting to prove that a linear increase in free distance also holds for ensembles with xed b and c but large ms . The penultimate chapter analyzed the iterative decoding of the ensemble of codes introduced in Chapter 6 on the erasure channel. We proved that terminated convolutional codes in the ensemble have thresholds much better than that of corresponding block codes. The analysis of these codes on more complicated channels like the AWGN and BSC is being carried out. Preliminary results indicate that a similar improvement is possible. The improvement in performance is due to the convolutional structure of the Tanner graphs. It would be of great interest to determine if there are other structures for which such dramatic improvements in threshold are possible. As mentioned in Chapter 7, thresholds of the codes improve with increasing J . It would be interesting to determine if by increasing J the thresholds can be made equal to the capacity of the erasure channel. If this were to be the case, these codes would be the rst class of capacity achieving codes in the sense that not only the bit but also block error probability could be made arbitrarily small at a rate equal to the capacity of the channel. The one unsatisfying result in this chapter is the requirement of a tail of length ms + 1 time units to terminate a convolutional code in the ensemble. Typically, a tail of ms time units is sucient to terminate a convolutional code. It would be nice to nail down the reason for the extra bits needed for termination. The dissertation has focused mostly on the AWGN channel with binary input. The results obtained however suggest that LDPC convolutional codes would also be good choices for other channel models, e.g., AWGN channels with higher order modulation schemes, wireless fading channels etc.

149

APPENDIX A MAXIMUM OF THE FUNCTION U ([1,L] )

Solving (6.51)(6.53) and using t = t

(0)

(1)

for t = 1, . . . , L, we obtain

U ([1,L] ) = ln 3 2 (0) (0) (0) (0) [1 + 2 + L1 + L ] ln 7

(0) (0) (0) (0)

3 31
(0)

L2 t=1 (0)

t + t+1 + t+2 21 ln 1
(0) (0)

(0)

(0)

(0)

(0)

(0)

L1

t=2 L2 t=1

t ln t 2L ln L 1 (1 + 2 ) ln 1 (1 + 2 ) 1 (t + t+1 + t+2 ) ln 1 (t + t+1 + t+2 )

(0) (0) (0) (0) (0) (0)

(A.1)

Rather than use (6.57)(6.59) to determine max[1,L] : [1,L] =[1,L] U ([1,L] ), we use the following approach. Let t , t > 2, be xed and let 1 + 2 = 12 be xed. We then show that U ([1,L] ) is maximized by setting 1 = 0 and 2 = 12 . We have U ([1,L] ) 1 ln
(0)
(0) (0) 2 =12 1

(0)

(0)

(0)

(0)

(0)

= 1
(0) (0) 1 (0) (0) (0)

3 31

+ ln
(0)

12

ln 1 + 1 (12 + 3 + 4 )
(0)

3 ln 1 1

+ 4 ln 1 + 1 12

(A.2)

150

and 2 U ([1,L] )
(0) 2 1 2 =12 1
(0) (0)

1
(0) 1

1 12
(0) 1

3 1
(0) (0) 1

3 1+ .
(0) 1

12 (A.3)

1+

(0) 1

(12 + 3 + 4 )

(0)

It is straightforward to show that

2 U ([1,L] ) 2 1 (0)
(0)

> 0. Hence, any stationary

2 =12 1
(0) (0)

point of U ([1,L] ) with respect to 1 corresponds to a local minimum. Further, it can be shown that

U ([1,L] )|(0) =0,(0) =12 U ([1,L] )|(0) =12 ,(0) =0 > 0.

1 2 1 2

(A.4)

Hence U ([1,L] ) is maximized by setting 1

(0)

= 0 and 2
(0)

(0)

= 12 . Similarly, it
(0)

follows that U ([1,L] ) is also maximized by setting L

= 0 and L1 = 12 .
(0) (0)

Proceeding in the same manner it can also be shown that 2 = L1 = 0. Thus it is sucient to look at normalized weight compositions of the form [1,L] = (0, 0, 3 , 4 , . . . , L3 , L2 , 0, 0). Now let t , 4 t L 2, be xed, and let 3 + 4
(0) (0) (0) (0) (0)

= 34 be xed.

Repeating the analysis above, in this case we nd that U ([1,L] ) is maximized by setting 4 = 0 and 3 = 34 . Continuing in this fashion, we nd that U ([1,L] ) is maximized at [1,L] = (0, 0, 3 , 0, . . . , 0, L2 , 0, 0) with 3 = L2 .
(0) (0)

151

BIBLIOGRAPHY

1. C. E. Shannon, A mathematical theory of communication, Bell System Technical Journal, no. Vol. 27, pp. 379423 (Part I) and 623656 (Part II), 1948. 2. W. W. Peterson and E. J. Weldon, Jr., Error Correcting Codes. Cambridge: MIT Press, 2nd ed., 1972. 3. F. J. MacWilliams and N. J. A. Sloane, The Theory of Error Correcting Codes. North Holland, Amsterdam, 1977. 4. R. E. Blahut, Algebraic Codes for Data Transmission. Cambridge, UK: Cambridge University Press, 2003. 5. S. B. Wicker, Error Control Systems for Digital Communication and Storage. Englewood Clis, NJ: Prentice-Hall, 1995. 6. S. Lin and D. J. Costello, Jr., Error Control Coding: Fundamentals and Applications. Englewood Clis, NJ: Prentice-Hall, 2nd ed., 2004. 7. R. Johannesson and K. S. Zigangirov, Fundamentals of Convolutional Coding. IEEE Press, 1999. 8. P. Elias, Coding for noisy channels, IRE Transacations on Information Theory, pp. 3746, 1955. Also in Key Papers in Development of Coding Theory, IEEE Press, New York, NY, 1974. 9. A. J. Viterbi, Error bounds for convolutional codes and an asymptotically optimum decoding algorithm, IEEE Transactions on Information Theory, vol. 13, pp. 260269, Apr. 1967. 10. G. D. Forney, The viterbi algorithm, Proceedings of the IEEE, vol. 61, pp. 268278, Mar. 1973. 11. C. Berrou, G. Glavieux, and P. Thitimajshima, Near Shannon limit coding and decoding: Turbo codes, in Proccedings of IEEE International Conference on Communications, (Geneva, Switzerland), pp. 10641070, May 1993. 152

12. R. G. Gallager, Low-density parity-check codes. M.I.T. Press, Cambridge, Massachusetts, 1963. 13. V. Zyablov and M. Pinsker, Estimation of the error-correction complexity of gallager low-density codes, Problemy Peredachi Informatsii (Problems of Information Transmission), vol. 11, pp. 2336, Jan.-Mar. 1975. 14. R. M. Tanner, A recursive approach to low complexity codes, IEEE Transactions on Information Theory, vol. IT-27, no. 5, pp. 533547, Sept. 1981. 15. D. J. C. Mackay and R. M. Neal, Good codes based on very sparse matrices, in Cryptography and Coding. 5th IMA Conference (C. Boyd, ed.), no. 1025 in Lecuture Notes in Computer Science, (Berlin, Germany), pp. 100 111, Springer, May 1995. 16. T. Richardson and R. Urbanke, The capacity of low-density parity check codes under message-passing decoding, IEEE Transactions on Information Theory, vol. IT-47, no. 2, pp. 599618, Feb. 2001. 17. T. Richardson, A. Shokrollahi, and R. Urbanke, Design of capacityapproaching irregular low-density parity-check codes, IEEE Transactions on Information Theory, vol. 47, no. 2, pp. 619637, Feb 2001. 18. S.-Y. Chung, G. D. Forney, Jr., T. J. Richardson, and R. Urbanke, On the design of low-density parity-check codes within 0.0045 dB of the Shannon limit, IEEE Communication Letters, vol. 5, pp. 5860, Feb. 2001. 19. E. N. Gilbert, A comparison of signalling alphabets, Bell System Technical Journal, vol. 31, pp. 504522, 1952. 20. R. R. Varsharmov, Estimates of the number of signals in error correcting codes, Doklady A.N.S.S.R, vol. 117, no. 5, pp. 739741, 1957. 21. T. Richardson and R. Urbanke, Ecient encoding of low density parity check codes, IEEE Transactions on Information Theory, vol. IT-47, no. 2, pp. 638 656, Feb. 2001. 22. M. Luby, M. Mitzenmacher, M. A. Shokrollahi, and D. A. Spielman, Ecient erasure correcting codes, IEEE Transactions on Information Theory, vol. IT47, pp. 569584, Feb. 2001. 23. M. Luby, M. Mitzenmacher, M. A. Shokrollahi, and D. A. Spielman, Improved low-density parity-check codes using irregular graphs, IEEE Transactions on Information Theory, vol. IT-47, pp. 585598, Feb. 2001.

153

24. A. Jim enez Feltstr om and K. S. Zigangirov, Time-varying periodic convolutional codes with low-density parity-check matrix, IEEE Transactions on Information Theory, vol. IT-45, no. 6, pp. 21812191, Sept. 1999. 25. R. M. Tanner, A [155,64,20] sparse graph (LDPC) code,. presented at the Recent Results Session, IEEE Intl. Symp. on Information Theory, (Sorrento, Italy), June 2000. 26. S. L. Hakimi and J. Bredeson, Graph theoretic error-correcting codes, IEEE Trans. Inform. Theory, vol. IT-14, pp. 584591, July 1968. 27. R. M. Tanner, D. Sridhara, A. Sridharan, T. E. Fuja, and D. J. Costello, Jr., LDPC block and convolutional codes based on circulant matrices, IEEE Transactions on Information Theory, Dec. 2004. to appear. 28. R. M. Tanner, D. Sridhara, and T. E. Fuja, A class of group-structured LDPC codes, in Proceedings of ISTA, (Ambleside, England), 2001. 29. R. M. Tanner, Convolutional codes from quasi-cyclic codes: A link between the theories of block and convolutional codes,. Technical Report, Computer Science Research Laboratory, UC Santa Cruz, Nov. 1987. 30. D. J. C. MacKay and M. C. Davey, Evaluation of Gallager codes for short block length and high rate applications, vol. 123, ch. 5, pp. 113130. the IMA Volumes in Mathematics and its Applications, 2001. 31. H. H. Ma and J. K. Wolf, On tail-biting convolutional codes, IEEE Transactions on Communications, vol. COM-34, pp. 104111, Feb. 1986. 32. G. Solomon and H. C. A. van Tilborg, A connection between block and convolutional codes, SIAM Journal on Applied Mathematics, vol. 37(2), pp. 358 369, Oct. 1979. 33. M. P. C. Fossorier, Quasi-cyclic low density parity check codes from circulant permutation matrices, IEEE Transactions on Information Theory, To appear. 34. A. Sridharan, D. J. Costello, Jr., D. Sridhara, T. E. Fuja, and R. M. Tanner, A construction for low density parity check convolutional codes based on quasi-cyclic block codes, in Proccedings of IEEE Intl. Symp. on Information Theory, (Lausanne, Switzerland), p. 481, June 2002. 35. S. Lin, J. Xu, I. Djurdjevic, and H. Tang, Hybrid construction of LDPC codes, in Proccedings of the 40th Allerton Conference on Communication, Control, and Computing, (Monticello, IL, USA), Oct. 2002.

154

36. J. Thorpe, Low-density parity-check codes constructed from protographs. IPN Progress Report 42-154, JPL, Aug. 2003. 37. S.-Y. Chung, T. Richardson, and R. Urbanke, Analysis of sum-product decoding of low density parity-check codes using a Gaussian approximation, IEEE Transactions on Information Theory, vol. IT-47, no. 2, pp. 657670, Feb. 2001. 38. J. Hagenauer, Decoding of binary codes with analog networks, in Proc. Information Theory Workshop, (San Diego, USA), pp. 1314, 1998. 39. H. A. Loeliger, M. Helfenstein, F. Lustenberger, and F. Tark oy, Probability propagation and decoding in analog VLSI, in Proc. 1998 IEEE Intl. Symp. Information Theory, (Cambridge, MA), p. 146, Aug. 1998. 40. F. Lustenberger, M. Helfenstein, H. A. Loeliger, F. Tark oy, and G. S. Moschytz, All analog decoder for a binary (18,9,5) tail-biting trellis code, in Proc. 25th European Solid-State Circuits Conference, (Duisburg, Germany), pp. 362365, Sept. 1999. 41. M. Moerz, T. Gabara, R. Yan, and J. Hagenauer, An analog 0.25 m BiC-MOS tail-biting MAP decoder, in Proc. IEEE International Solid-State Circuits Conference, (San Francisco, USA), pp. 356357, Feb. 2000. 42. C. Winstead, J. Dai, W. J. Kim, S. Little, Y. B. Kim, C. Myers, and C. Schlegel, Analog MAP decoder for (8, 4) Hamming code in subthreshold CMOS, in Proc. Advanced Research in VLSI, (Salt Lake City, USA), pp. 132147, Mar. 2001. 43. H. A. Loeliger, F. Lustenberger, M. Helfenstein, and F. Tark oy, Probability propagation and decoding in analog VLSI, IEEE Transactions on Information Theory, vol. IT-47, pp. 837843, Feb. 2001. 44. A. Schaefer, A. Sridharan, M. Moerz, J. Hagenauer, and D. J. Costello, Jr., Analog rotating ring decoder for an LDPC convolutional code, in Proc. IEEE Information Theory Workshop, (Paris, France), pp. 226229, Apr. 2003. 45. A. Veinblat, Analog convolutional decoder, Masters thesis, Technion - Israel Institute of Technology, May 2002. 46. M. Moerz, Analog sliding window decoder core for mixed signal turbo decoder, in Proc. 5th ITG Conference on Source and Channel Coding, (Erlangen, Germany), pp. 6370, Jan. 2004.

155

47. B. K onig, Concatentating LDPC convolutional codes for analog decoding, Masters thesis, Institute for Communications Engineering, Munich University of Technology, Mar. 2004. 48. K. Engdahl, M. Lentmaier, and K. S. Zigangirov, On the theory of lowdensity convolutional codes, in Applied Algebra, Algebraic Algorithms and Error-Correcting Codes, 13th International Symposium, AAECC-13 (M. P. C. Fossorier, H. Imai, S. Lin, and A. Poli, eds.), vol. 1719 of Lecture Notes in Computer Science, pp. 7786, Springer, Nov. 1999. 49. A. Sridharan, M. Lentmaier, D. V. Truhachev, D. J. Costello, Jr., and K. S. Zigangirov, On the minimum distance of codes with parity check matrices constructed from permutation matrices, Submitted to the Problems of Information Transmission (Problemy Peredachi Informatsii), July 2004. 50. M. Lentmaier, D. Truhachev, and K. S. Zigangirov, On the theory of low density convolutional codes II, Problems of Information Transmission (Problemy Peredachi Informatsii), vol. 37, pp. 1535, Oct.-Dec. 2001. 51. K. Engdahl and K. S. Zigangirov, On the theory of low density convolutional codes I, Problems of Information Transmission (Problemy Peredachi Informatsii), vol. 35, no. 4, pp. 295310, Oct.-Dec. 1999. 52. M. Luby, M. Mitzenmacher, A. Shokrollahi, D. Spielman, and V. Stemann, Practical erasure resilient codes, in Proceedings of the 29th annual ACM Symposium on Theory of Computing, ((STOC)), pp. 150159, 1997. 53. M. Lentmaier, D. V. Truhachev, K. S. Zigangirov, and D. J. Costello, Jr., An analysis of the block error probability performance of iterative decoding, submitted to IEEE Transactions on Information Theory, Apr. 2004. 54. P. Oswald and A. Shokrollahi, Capacity-achieving sequences for the erasure channel, IEEE Transactions on Information Theory, vol. IT-48, pp. 3017 3028, December 2002. 55. Y. Kou, S. Lin, and M. Fossorier, Low density parity-check codes based on nite geometries: a rediscovery and new results, IEEE Transactions on Information Theory, vol. IT-47, no. 7, pp. 27112736, Nov. 2001.

156