1548
zyxwvutsrqponmlkjih
EEE
TRANSACTIONS ON INFORMATTON THEORY,
VOL.
38,
NO.
4,
SEPTEMBER
1992
zy
A
Direct Geometrical Method for Bounding the Error Exponent for
Any
Specific Family of Channel CodesPart
I:
Cutoff Rate Lower Bound for Block Codes
Dejan
E.
LaziC,
zyxwvutsr
ember,
IEEE,
and Vojin Senk,
Member,
IEEE
AbstractA
direct, general and conceptually simple geometri cal method for determining lower and upper bounds on the error exponent of any specific family of channel block codes is presented. It is considered that a specific family of codes is characterized by
a
unique distance distribution exponent, de fined
as
the asymptotic (in codelength) negative logarithm
of
the expected (according to the codewords
a
zyxwvutsr
riori
probabilities) Bhattacharyya distance distribution, normalized by the code length. Depending on the channel characteristics, every such family has its own error exponent. The maximum error exponent over all the realizable families of channel block codes
is
the usual channel error exponent. The tight linear lower bound of slope

zyxwv
on the code family error exponent represents the code family cutoff rate bound. It is always a minimum of a sum of three functions; the first of these functions depends on the asymptotic Bhattacharyya distance characteristics of the code family, the second on the channel characteristics, and the third is the negative value
of
the code rate. The intrinsic asymptotic properties of channel block codes are revealed by analyzing these functions and their relationships. It is shown that the wellknown random coding technique for lowerbounding the channel error exponent, when adequately interpreted, is but
a
special case of this general method. The (indirect) random coding technique uses the notion of an ensemble of all possible codes, thus concealing the requirements that a code family should meet in order to have
a
positive error exponent and at best attain the channel error exponent. These requirements are now stated in a limpid way using the (direct) distance distribu tion method presented. Index TermsChannel coding theorem, error exponent, non random coding argument, distance distribution.
1.
INTRODUCTION
VER
since Claude Shannon published his famous
E
948 papers [ll, information theorists used and devel
Manuscript received October
IO,
1990; revised September
6,
1991. This work was presented in part at the
IEEE
International Symposium on Information Theory, Budapest, Hungary, June 2428, 1991. D.
E.
LaziC
was
with Universitit Karlsruhe, Fakultit fur Informatik, Institut fur Algorithmen und Kognitive Systeme,
zyxwvutsr
m
Fasanengarten
5
(Geb. 5034), D7500 Karlsruhe, FR Germany, Alexander von
Humboldt
Fellow on leave
from
Faculty of Technical Sciences, Institute for
Com
puter Science, Control and Measurements, Trg Dositeja 0bradovii.a
6,
21000 Novi Sad, Yugoslavia. He is now with Universitat Karlsruhe, Fakultat fur Informatik, Institut
fur
ProzeRrechentechnik und Robotik (Geb. 4028), D7500 Karlsruhe, FR Germany,
as
a
Fellow
of
European CommyityDGXII.
V.
Senk is with the Faculty
of
Technical Sciences, Institute for Computer Science, Control and Measurements, Trg Dositeja Obradoviia
6,
21000 Novi Sad, Yugoslavia.
IEEE
Log
Number 9107507.
oped his random coding technique
of
obtaining an asymp totic (in code length) upper bound on the probability
of
block decoding error for the optimal channel block code. Formally,
it
consists
of
calculating the average probability
of
block decoding error over the ensemble of all sets of
M
codewords with dimensionality (length)
zy
hat are possi ble over an encoding space. This technique was enthusias tically welcomed
as
the “way out of the impasse” and frequently designated as the central technique
of
informa tion theory. For
a
large number of coding channels it has really succeeded in obtaining lower bounds
on
channel error exponents (reliability functions) defined
as
where
R
is the code rate (measured in bits/symbol), and
P,
opt(
R,
N)
s the smallest possible probability of block decoding error for codes of code rate
R
and length
N
used on the channel considered. Thus, the upper bound
on
Peopt(R,
N)
s given by
PeOpt(R
P,,,,l(R>
N)
=
exp2[N.E(R)
+
o(N)],
exp2(x)
=
2”, (1b) where
o(N)
s a term of order less than
N;
s
N
+
zy
C
it becomes small relative to
N.
(R).
Three aspects of this technique were especially satisfac tory:
1)
the obtained lower bound
on
E(R)
was positive for all code rates below the channel capacity,
R,,
and equal to zero at
R,,
2)
it
was tight in the most important range of code rates, i.e., it agreed with the upper bound on
E(R)
for all code rates above
R,,,,
(critical rate), and below
R,.
(R,,,,
is typically well below
Rc),
nd
3)
it
guaranteed that almost every code of code rate
R
and great enough
N
atisfied (lb). The most unsatisfactory aspect
of
the random coding technique is that
it
cannot determine the performance of
00189448/92 03.00
0
Y92 IEEE
11
LAZIC
AND
SENK:
DIRECT GEOMETRICAL METHOD
zyxwvutsrq
OR
BOUNDING THE
ERROR
EXPONENT
zyxwvu
549
a specific family of channel block codes used
on
the channel considered, nor can it determine the require ments that a specific family of channel block codes should meet in order to attain the probability of error guaranteed by Ob). In order to overcome this problem, it is necessary to define the code family error exponent where
zyxwvutsr
?
s a specific family of channel block codes with a unique distance distribution exponent, defined as the asymptotic (in
zyxwvutsrqp
)
negative logarithm of the expected (according to the codewords a priori probabilities) Bhat tacharyya distance distribution, normalized by the code length, and
zyxwvutsrqpo
,,(R,
N)
s the probability of block decod ing error for the code of code rate
R
and dimensionality
N
that belongs to
9.
ince
P,,,(R,
N)
cannot be smaller than
Peopt(R,
),
it is obvious that
0
5
E(R)g
5
E(R).
(3)
It is the aim of this paper to present a new technique for the derivation
of
the tight linear lower bound of slope
1
on the code family error exponent,
E,(R),.
This direct method does not use the random coding argument, i.e., ensemble averaging, and is applicable to any specific family
35’
of channel blocks codes decoded using the maximum likelihood rule. The bound thus obtained has the form
Eo(R)g
=
R,, R,
zyxwvu
4)
where
R,,
is a function independent of
R
that repre sents the cutoff rate of the code family
on
the coding channel considered. The bound (4)
is
called the cutoff rate lower bound
on
the code family error exponent. The general distance distribution method for bounding
E(R),
is presented in Section
11,
and used for the deter mination of
,(RI,.
In Section
zyxwvuts
11,
this method is applied to derive
,(RI,
for several families of spherical and binary codes used on the additive white Gaussian noise (AWGN) channel and binary symmetric channel
(BSC),
as well as to establish the impact of the minimum distance on
E,(R),.
zyxwvutsr
n
improvement of the method for obtaining
<R)a
is analyzed in a companion paper [2] (hereafter denoted as Part 11). This improvement gives the exact solution for
E(
R),
of some important families of channel block codes, and tight solution at low code rates for any specific family of block codes. It contains the full geometrical explana tion of all the phenomena encountered, including the impact of the minimum distance
on
E(R),,.
11.
DISTANCE ISTRIBUTION ETHOD
ND
ITS
APPLICATION
TO THE
DETERMINATION
F
THE
CUTOFF
RATE
LOWER
BOUND
N
E(R),
This method will be demonstrated on the standard model for studying the reliability of communications over noisy transmission channels. The model consists of a coding channel,
Z’,
with an encoder at its input and a decoder at its output. When the input to the channel is an ntuple from
X”,
=
1,2;..,
it
outputs a random ntuple from
Y“,
where input and output alphabets
X
nd
Y
may be of different cardinality,
1x1
=
q,
IYI
=
q’,
1
<
q,
q’
5
x
A
coding channel is completely defined by
X,
Y
and the set of likelihood functions,
{P[y‘”’
x‘“)],
x “)
E
X“,
y(”)
E
Y”,
n
=
1,2;..). The encoder generates a code word of the channel block code
B(R,
N,P)
=
{xm
x,,,
€
PC
”,
m
=
1,2;..,
M},
where
zyx
is the available encod ing space. The decoder uses the received random vector
y
=
(yl;..,
yN)
o produce the estimate
3
of the transmit ted codeword from
B(
R,
N,
29
according to a convenient decoding rule. Denote by
e,(x,)
the probability that
3
equals
x,
when
x,,
m
f
,
is
sent over
5
i.e.,
e,,(x,)
~~P[i=x,Ix,,j#rn],
x,, x,
E
B( R,
N,Z),
m
=
l;..
,M.
(5)
This quantity will be called the error effect of the code word
x,
on the probability of erroneous decoding when
x,,
is sent over since
M
P,,
=
P[i
z
x, x,,]
=
e, x,),
r
=
I,...,
M.
I=
I
(6)
The overall probability of block decoding error for
B(R,
N,X),
used on is
M
M
zyx
P,(R>N)
=
c
P[x,lP,,,
=
c c
P[x,Ie,,(x,).
m=
1
m=l
/=I
J
Wl
(7)
An upper bound,
P,(R,
N),
on
P,(R,
N)
for any
B(R,
N,Z)
used on can be obtained upper bounding
e,(x,)
for each pair of codewords from the code. Sorting the
M(M
I)
upper bounds
on
the error effects of each codeword
on
all others by descending order,
L
different values of such bounds are obtained
(1
I
2
M(M
1)).
Denote by
S
he Ith value of the upper bound
on
the error effec_t
(e,
>
e,,
,,
=
l;..,
L

l),
where there are
A?,
(1
5
M,
5
M(M
1_>)
ounds
on
it that have the value
S
out of which
MnII
0
5
M,,
5
M
1) have the sole influence
on
the determination of
P,,.
P,(R,
N)
s thus overbounded by
Mi
P,(R,N)
=
c
CP[x,I~,,~,
m=
I
I=
1
iM
=
e4 c
P[.,ln;l,,.
(8)
l=l
m=l
Denoting by
(A?,)
the expected number of codewords in
B(R,
N,Z)
hat have the same upper bound
on
the error
1550
zyxwvutsrqponmlkjih
EEE TRANSACTIONS
ON
INFORMATION
THEORY,
VOL.
38,
NO.
4,
SEPTEMBER
1992
effect on an another codeword, evaluated as
zyxwvuts
m=l
zyxwvutsrqpo
x,
E
zyxwvutsrq
(R,N,X),
zyxwvuts
=
l;..
zyxwv
L
9)
one obtains that
zyxwvuts
FS,(R,N)
=
LQ/>Z,.
(loa)
I=
1
It is obvious that using the same sorting procedure one can also obtain the lower bound on
P,(R,
N)
whose form is
c
FJR,N)
=
c
M>g/,
(lob)
I=
zyxwvutsrqponmlkjihgf
where
&,
(MI>,
nd
e,
are defined analogously to their counterparts in (loa). Upper and lower bounds on the error effects are best determined and sorted according to a convenient distance between different codewords. Such a distance will be introduced in the sequel, assuming that a maximum likelihood decoder (MLD) is used. MLD selects as its estimate the codeword
x,
for which the likelihood function
P[y
x,]
is largest. Though its decision is optimum only when all the codewords are equally likely, it
is
often robust in the sense that it gives the same or nearly the same
P,(R,
N)
egardless of the
a
priori probability distribution of the codewords,
{P[
,],
m
=
l;..,
MI.
MLD sets
3
=
x,
E
B(R,
N,Z)
if
P[y
x,]
>
P[y
x,),
j
=
I;..,
M,
J
f
m.
Ties are re solved arbitrarily, for instance in the favor of smaller index. This decision criterion can be given a geometrical interpretation using the appropriate decision regions (also called Voronoi regions), that perform the partitioning of the whole decoding space
YN.
MLD sets
2
=
x,,
if
and only
if
the channel output vector
y
belongs to the Voronoi region
V(x,),
defined as
zyxwvuts
VX,
n
qx,
I
x,),
qx,
I
XI)
I=
I
I'm
(smaller index tie resolving rule is assumed). An attempt to visualize the partition
of
a part of the channel output space
YN
nto Voronoi regions is shown in Fig.
1.
The error effect
of
the codeword
x,
on the probability of erroneously decoding when codeword
x,
is sent through the channel can be bounded using the
two
iso lated codewords bound (see Fig. 1)
Fig.
1.
Visualization
of
the partition
of
a part
of
the channel output space
Y
into Voronoi regions, and the expansion of the Voronoi region
Wx,)
into
V(x,
Ix,)
that enables the determination
of
the
two
isolated codewords upper bound on the error effect.
y,
denotes the most probable channel output when the codeword
x,
has been trans mitted over
it.
that can be further overbounded by the Bhattacharyya upper bound
[3,
pp. 62631
C x,)
=
exp2[
~d,(X,,X,)],
(13)
where is the normalized (by
N
hattacharyya distance. It is readily seen that
dR(xn,,
,)
depends on
x,, x
,
and the characteristics of the channel considered. If YN1is contin
uous,
the sum in
(14)
is substituted with an integral. For the AWGN channel, the normalized Bhattacharyya distance is equal
to
[3,
p.
631
4,( 9,)
=
[SNR/(81n(2))ld~(x,,x,),
zy
15)
where
4
is the squared Euclidean distance normalized
by
the expected codeword energy,
E
=
C,
P[x,
IIx,~
2
and SNR is the signaltonoise ratio (ratio of
E
and noise energy,
EAWGN
Nu2,
where
(+?
is the variance of the AWGN component). For the BSC the normalized Bhat tacharyya distance
is
equal to
gA xni,x,)
=
[1dJ.lp lp)]d, xrn,xj),
(16)
where
p
is the crossover probability of the BSC, and
dH
denotes the normalized (by
N
amming distance. We can now define the expected Bhattacharyya dis tance distribution of the block code
B(R,
N,Z)
as
M,,
P,
=
Pl(dsr,
R,
N
P[x,]
M1
m=
1
Here
L
5
M(M
1)/2 is the number of different nor malized Bhattacharyya distances in the code
(dR1
<
ds2
<
<
&),
M,nl
is
the number
of
codewords
on dis
I
11
uzic
AND
SENK:
DIRECT
GEOMETRICAL
METHOD FOR
BOUNDING
THE
ERROR EXPONENT
1551
tance
zyxwvutsrqpo
,,
from
zyxwvutsr
,,
and
(MI)
s the expected (according to
{P[x,],
m
zyxwvutsrqp
l;*.,M}) number of codewords on dis tance
d,,
from a codeword in
zyxwvutsrq
(R,
N,t% ).
EBDD is the set of relative frequencies of the expected (according to
{P[x,],
m
=
l,...,
zyxwvutsrq
})
umbers of codewords
on
distance
d,,
from a codeword in
B(R,
N,Z).
It is obvious that these relative frequencies sum up to
1.
In the case when
B(R,
N,2)
is distance invariant, i.e., when the set of Bhattacharyya distances is the same from any codeword in the code, EBDD is independent of
zyxwvut
P[n,],
zyxwvu
=
1;,
MI.
Note also that EBDD remains unchanged
if
the normal ized Bhattacharyya distance is substituted by any mono tonically increasing function of
&.
For instance,
LI
in the case of the AWGN channel and
4
in the case
of
the BSC are suitable for this substitution, as can be seen from
(15)
and (16). The upper and lower bounds on the error effect (used in (10)) can always be simply related to the normalized Bhattacharyya distance
d,,
(a simple upper bound is given by (13)). Generally, introducing these bounds, denoted by
e',
zyxwvutsrq
'(&,,
R,
zyxwvuts
)
nd
e,
e(d,,,
R,
NI,
together with (17) into (10) and overbounding
(M
1) by
M
one obtains that the form of upper and lower bounds on
P,(R,
N)
s
L
2
e(
R,
N,
)
P(dB,,
R7
N,
I=
1
<
Few
N)
L
kf
c
P(d,,,
R,
N)
l=
1
*e (dB,,
R,
N),
(18)
I
since
L
=
=
&
and
M,)
=
M

l)P(dB,,
R,
N)
=
k,)
zyxwvut
y,).
ntroducing the notion of the expected Bhattacharyya distance density,
(19)
p(dB,
R7
4/31
5
dB
dRL9
0,
elsewhere, ebdd
2
these bounds attain the form
2
ee(~,
)
=
(M
I)/ ~P(W,
R,
N)
zyxwvut
A1
e'(x,R,N)&.
(20) Since EBDD is essentially discrete for any block code,
(20)
has practical sense only if a function
p(d,,
R,
N),
continuous in
4
can be associated to EBDD of
B(R,N,Z),
whose features are that the difference be tween bounds in (18) and
(20)
s negligible. This is always the case when
L
00,
if the associated continuous func tion represents an adequate interpolation of EBDD of
B(R,
N,Z).
L
+
CD
implies
M
+
00,
which further implies that
R
for
R
finite. Only this later case is
of
interest. The method
of
obtaining the upper and lower bounds
on
P,(R,
N)
sing (18) and (20) will be called the distance distribution method. The expressions (18) and (20) are, as can be easily seen from their derivation, absolutely gen eral and can be applied to any channel, regardless of whether random or deterministic codes are considered. For random codes EBDD corresponds to the probability distribution of the expected number
of
codewords on the normalized Bhattacharyya distance
d,,
from a codeword in the code. Naturally, the bounds (18) and (20) will have sense only
if
the existence of a block code with underlying EBDD (ebdd) can be proved. Finally, it can be observed that
(18)
and
(20)
represent linear transformations of EBDD and ebdd into upper and lower bounds on
P,(R,
N)
that are relatively simple to evaluate, as will be seen in the sequel. Since the aim of this paper is obtaining the cutoff rate lower bound on the code family error expo nent, from now on we will consider only the upper bound on
P,(R,
NI,
though the expressions analogous to those derived in the rest of this section are also valid for the lower bound on it. for
N
finite, and that
N
The expression (18) can always be overbounded by while an overbound on (20) is Applying (21a) to all the members of a specific family
9
of channel block codes, and introducing it into (21, the lower bound on
E(R),
is obtained as supposing that the limit in
(22)
exists. Note that the term
(
/N)ld(
L)
is omitted, since when
L
exponentially grows with
N
all the conditions for switching to continu ous case (21b) are met, and that introducing (21b) into (2) gives (22) again (the only difference is that
P(&,
R,
N)
is substituted by p(&,
R,
NI).
Thus, in what follows, the discrete and continuous case need not be treated sepa rately. We will use the continuous denotation in the sequel. It is convenient, in order to proceed with the analysis of (22), to define the distance density exponent (dde) (or the
I
11
1552 IEEE
TRANSACTIONS
ON
INFORMATION
THEORY,
VOL.
38,
NO.
zyxw
,
SEPTEMBER 1992
distance distribution exponent (DDE) in the discrete case) as
zyxwvutsrq
\E,
elsewhere, where
&,
=
limN+m{cfBl} nd
zyxwvutsr
jBL
=
1imN+%{dBL}, s well as the lower bound on the error effect exponent (EEE) as Supposing that the limits in (23) and (24) exist, and that (23) specifies the code family
zyxwvut
,
xchanging lim and min operations in
zyxwvutsrqp
22)
gives the following lower bound on
zyxwvut
(
R),
E WB
=
<?in
{E,(d,,R)
E,(dB,
R,g)
zy
.
(25)
zyxwvut
BI
~
B
5
I
Here,
E,(&,, R)
s dependent on the code family charac teristics (that may, but often does not, depend on the code rate), and
&e(d,,
R,
9
s dependent
on
the channel characteristics and, eventually, the code family character istics and the code rate, subject to the choice
of
the way
of
overbounding
Z .
Such an overbound that depends on
5
B
nd
R
is used in Part 11. The expression
(25)
is the basic formula for evaluating lower (upper) bounds on
E(R),,
that displays the intrin sic dependence of the code family error exponent on the characteristics of
B
,
and on
R.
This expression is very simple, general and conspicuous. In order to make it even simpler, at least in notation, denote by
&B.9eff
that nor malized Bhattacharyya distance at which the minimum in
(25)
is obtained,
so
that
awL3
=WdB.@cff?R)
MdB.@Cff?R>9)
R.
(26)
Maximizing
E(R),
over all realizable families of block codes, one would obtain the lower bound on the channel error exponent. It is now easily seen that the usual ran dom coding technique for bounding the channel error exponent when adequately interpreted is but a special case of this general technique. Namely, in that case the DDE (dde) is defined for codes whose codewords are drawn at random according to an arbitrary probability distribution on
2,
nd then optimized over all possible probability distributions on
zyxwvuts
he random coding tech nique is usually interpreted as averaging
P,(R,
N)
over an ensemble of all possible codes of given cardinality and dimension, though it is almost never carried out in that way, but according to the procedure just described. This point will be discussed again in the conclusion, after exhibiting the power of the distance distribution method
on
several examples usually encountered in the literature. The simplest lower bound on EEE is obtained introduc ing (13) into (24),
so
that
Ce:e(d,,
R,
9
EAdR)
=
dB,
(27) i.e., it represents simply the normalized Bhattacharyya distance. This bound does not depend on R or
9.
There are families
9
f block codes whose DDE (dde) is not a function of the code rate. Having such a code rate invariant family, and using
(271,
the corresponding lower bound on
E(R),
is, according to (251, given by Since neither
E,(d,)
nor
dR
depend on
R,
his is a linear function of R, of slope

1.
Since
(28)
is always tight in a certain range of code rates, as shown
in
Part
zy
1,
it is the desired cutoff rate lower bound
F,(R),
on
E(R),.
Ac cording to (26), expression (28) can also be written as
e0(R)L8
=
E (d;,df)
+
d;;*,ff

R
e
R ,

R,
(29) where
dg9
cff
is that normalized Bhattacharyya distance at which the minimum in (28)
is
obtained, and
R,,
=
E
8( deff)
+
dg
eft
represents the cutoff rate of the family
9
sed on
g.
bviously,
if
E,(&,)
is differen tiable and
dgBeff
s in the interval
[ ,, I,
it satisfies the relation Maximizing (28) over all realizable code rate invariant families
of
block codes, one would obtain the cutoff rate lower bound on error exponent for the channel itself. Fig. 2 gives a simple graphical interpretation of the presented method of obtaining
Ro~*
or a typical code rate invariant family of block codes that have a unique dde. It is noteworthy that the asymptotic minimum Bhat tacharyya distance,
&,,
of
9
oes not have any influence on Ro,,, unless
d;;,,cff
is equal to
l.
Examples of code rate invariant families
of
spherical and binary codes, used in the AWGN and binary symmet ric channel will be used in the next section to illustrate the power of the distance distribution method in obtain ing the appropriate cutoff rate lower bounds on the code family error exponent.
111.
EXAMPLES
F
THE
APPLICATION
F THE
DETERMINATION
F
THE
CUTOFF ATE LOWER DISTANCE ISTRIBUTION ETHOD
OR
BOUND
N
CODE FAMILY RROR
XPONENT
In this section, the distance distribution method will be applied to determination of the cutoff rate lower bound on
E(
R),
for some important code rate invariant families of channel block codes used on the
AWGN
and binary symmetric channels. Among the most important families of block codes for any symmetric channel and encoding space are those that are uniformly distributed over
F,
I
I1