Digital Signal Processing: Computer Science Perspective 2nd Ed
Digital Signal Processing: Computer Science Perspective 2nd Ed
A Computer SciencePerspective
A Wiley-Interscience Publication
JOHN WILEY & SONS,INC.
New York l Chichester l Weinheim l Brisbane l Singapore l Toronto
Copyright 2000 by John Wiley & Sons, Inc. All rights reserved.
This publication is designed to provide accurate and authoritative information in regard to the
subject matter covered. It is sold with the understanding that the publisher is not engaged in
rendering professional services. If professional advice or other expert assistance is required, the
services of a competent professional person should be sought.
ISBN 0-471-20059-X.
For more information about Wiley products, visit our web site at www.Wiley.com.
Stein, Jonathan Y.
Digital signal processing: a computer science perspective/Jonathan Y. Stein
p. cm.
A Wiley-Interscience publication.
Includes bibliographical references and index.
ISBN 0-471-29546-9 (cloth: alk. paper)
1. Signal processing Digital techniques. I. Title.
10 9 8 7 6 5 4 3 2 1
To Ethel, Hanna and Noga
Contents
Preface XV
1 Introductions 1
1.1 Prehistory of DSP ........................ 2
1.2 Some Applications of Signal Processing ............ 4
1.3 Analog Signal Processing .................... 7
1.4 Digital Signal Processing .................... 10
2 Signals 15
2.1 Signal Defined .......................... 15
2.2 The Simplest Signals ...................... 20
2.3 Characteristics of Signals .................... 30
2.4 Signal Arithmetic ........................ 33
2.5 The Vector Space of All Possible Signals ........... 40
2.6 Time and Frequency Domains ................. 44
2.7 Analog and Digital Domains .................. 47
2.8 Sampling ............................. 49
2.9 Digitization ............................ 57
2.10 Antialiasing and Reconstruction Filters ............ 62
2.11 Practical Analog to Digital Conversion ............ 64
vii
...
VIII CONTENTS
5 Noise 161
5.1 Unpredictable Signals ...................... 162
5.2 A Naive View of Noise ..................... 164
5.3 Noise Reduction by Averaging ................. 171
5.4 Pseudorandom Signals ...................... 174
5.5 Chaotic Signals .......................... 180
5.6 Stochastic Signals ........................ 192
5.7 Spectrum of Random Signals .................. 198
5.8 Stochastic Approximation Methods .............. 202
5.9 Probabilistic Algorithms .................... 203
6 Systems 207
6.1 System Defined .......................... 208
6.2 The Simplest Systems ...................... 209
6.3 The Simplest Systems with Memory .............. 213
6.4 Characteristics of Systems ................... 221
6.5 Filters ............................... 226
6.6 Moving Averages in the Time Domain ............. 228
6.7 Moving Averages in the Frequency Domain .......... 231
6.8 Why Convolve? ......................... 237
CONTENTS ix
7 Filters 271
7.1 Filter Specification ........................ 272
7.2 Phase and Group Delay ..................... 275
7.3 Special Filters .......................... 279
7.4 Feedback ............................. 289
7.5 The ARMA Transfer Function ................. 293
7.6 Pole-Zero Plots .......................... 298
7.7 Classical Filter Design ...................... 303
7.8 Digital Filter Design ....................... 309
7.9 Spatial Filtering ......................... 315
8 Nonfilters 321
8.1 Nonlinearities .......................... 322
8.2 Clippers and Slicers ....................... 324
8.3 Median Filters .......................... 326
8.4 Multilayer Nonlinear Systems .................. 329
8.5 Mixers .............................. 332
8.6 Phase-Locked Loops ....................... 338
8.7 Time Warping .......................... 343
9 Correlation 349
9.1 Signal Comparison and Detection ............... 350
9.2 Crosscorrelation and Autocorrelation ............. 354
9.3 The Wiener-Khintchine Theorem ................ 357
9.4 The Frequency Domain Signal Detector ............ 359
9.5 Correlation and Convolution .................. 361
9.6 Application to Radar ...................... 362
9.7 The Wiener Filter ........................ 365
9.8 Correlation and Prediction ................... 369
9.9 Linear Predictive Coding .................... 371
9.10 The Levinson-Durbin Recursion ................ 376
9.11 Line Spectral Pairs ....................... 383
9.12 Higher-Order Signal Processing ................. 386
X CONTENTS
IO Adaptation 393
10.1 Adaptive Noise Cancellation .................. 394
10.2 Adaptive Echo Cancellation .................. 400
10.3 Adaptive Equalization ...................... 404
10.4 Weight Space ........................... 408
10.5 The LMS Algorithm ....................... 413
10.6 Other Adaptive Algorithms ................... 420
Part IV Applications
Bibliography 829
Index 849
Preface
I know what you are asking yourself--there are a lot of books available
about DSP, is this book the one for me? Well that depends on who you
are. If
l you are interested in doing research and development in one of the
many state-of-the-art applications of DSP, such as speech compression,
speech recognition, or modem design,
0 your main proficiency is in computer science, abstract mathematics,
or science rather than electronics or electrical engineering,
l your math ematical background is relatively strong (flip back now to
the appendix-you should be comfortable with about half of what you
see there),
then you are definitely in the target group of this book. If in addition
l you dont mind a challenge and maybe even enjoy tackling brain-
teasers,
l youre looking for one comprehensive text in all aspects of DSP (even
if you dont intend reading all of it now) and dont want to have to
study several different books with inconsistent notations, in order to
become competent in the subject,
l you enjoy and learn more from texts with a light style (such as have
become common for computer science texts) rather than formal, dry
tomes that introduce principles and thereafter endlessly derive corol-
laries thereof,
then this is probably the book you have been waiting for.
This book is the direct result of a chain of events, the first link of which
took place in mid-1995. I had been working at a high-tech company in Tel
xv
xvi PREFACE
Aviv that was a subsidiary of a New York company. In Tel Aviv it was rela-
tively easy to locate and hire people knowledgeable in all aspects of DSP, in-
cluding speech processing, digital communications, biomedical applications,
and digital signal processor programming. Then, in 1995, I relocated to a
different subsidiary of the same company, located on Long Island, New York.
One of my first priorities was to locate and hire competent DSP software
personnel, for work on speech and modem signal processing.
A year-long search turned up next to no-one. Assignment agencies were
uncertain as to what DSP was, advertisements in major New York area
newspapers brought irrelevant responses (digital design engineers, database
programmers), and, for some inexplicable reason, attempts to persuade more
appropriate people from Silicon Valley to leave the California climate, for
one of the worst winters New York has ever seen, failed.
It struck me as rather odd that there was no indigenous DSP population
to speak of, in an area noted for its multitude of universities and diversity
of high-tech industries. I soon found out that DSP was not taught at under-
graduate level at the local universities, and that even graduate-level courses
were not universally available. Courses that ulere offered were Electrical En-
gineering courses, with Computer Science students never learning about the
subject at all. Since I was searching for people with algorithm development
and coding experience, preferably strong enough in software engineering to
be able to work on large, complex software systems, CS graduates seemed to
be more appropriate than EEs. The ideal candidate would be knowledgeable
in DSP and would in the target group mentionedabove.
Soon after my move to New York I had started teaching graduate level
courses, in Artificial Intelligence and Neural Networks, at the Computer
and Informations Sciences department of Polytechnic University. I inquired
of the department head as to why a DSP course was not offered to Computer
Science undergraduates (it was being offered as an elective to Electrical En-
gineering graduate students). He replied that the main reason was lack of
a suitable teacher, a deficiency that could be easily remedied by my volun-
teering.
I thus found myself volunteered to teach a new Computer Science un-
dergraduate elective course in DSP. My first task was to decide on course
goals and to flesh out a syllabus. It was clear to me that there would be lit-
tle overlap between the CS undergraduate course and the EE graduate-level
course. I tried to visualize the ideal candidate for the positions I needed to
fill at my company, and set the course objectives in order to train the perfect
candidate. The objectives were thus:
PREFACE xvii
Objectives defined, the next task was to choose a textbook for the course.
I perused web sites, visited libraries, spoke with publisher representatives at
conferences, and ordered new books. I discovered that the extant DSP texts
fall into three, almost mutually exclusive, categories.
About 75% of the available texts target the EE student. These books
assume familiarity with advanced calculus (including complex variables and
ordinary differential equations), linear system theory, and perhaps even
stochastic processes. The major part of such a text deals with semirigor-
ous proofs of theorems, and the flavor and terminology of these texts would
certainly completely alienate most of my target group. The CS student, for
example, has a good basic understanding of derivatives and integrals, knows
a little linear algebra and probably a bit of probability, but has little need
for long, involved proofs, is singularly uninterested in poles in the complex
plane, and is apt to view too many integral signs as just so many snakes,
and flee in terror from them.
In addition, these type-one texts ignore those very aspects of the subject
that most interest our target students, namely algorithm design, compu-
tational efficiency and special computational architectures, and advanced
applications. The MAC instruction and Harvard architecture, arguably the
defining features of digital signal processors, are generally not even men-
tioned in passing. Generally only the FFT, and perhaps the Levinson-Durbin
recursion, are presented as algorithms, and even here the terminology is of-
ten alien to the CS students ear, with no attention paid to their relation
with other problems well known to the computer scientist. The exercises
...
XVIII PREFACE
Every section ends with a number of exercises that are designed to be en-
tertaining and enriching. Some of these should not be difficult for the reader
who understands the section, being designed to reinforce basic understand-
ing of the material. Many are somewhat challenging, complementing the
text, extending the theory, or presenting actual applications of the subject
studied. Some are only loosely defined; for these one can give a quick an-
swer, or develop them into a term project. Others introduce new material
that will ease the understanding of the following sections, as well as widening
the readers DSP horizons.
I purposely avoid taking sides on the divisive issue of programming lan-
guage and environment for algorithm design and test on general-purpose
computers. Realizing that C, MATLAB, SPW, Mathematics and the like
will all have their staunch supporters, and all have their strengths and
weaknesses, I leave it to the student or instructor to select that language
with which they are the most comfortable. Every seasoned programmer is
most effective in his or her native language, and although some languages
are obviously better DSP environments than others, the difference can be
minimized by the use of appropriate libraries.
Although the book was written to serve as a course textbook, it may be
used by non-students as well. DSP practitioners are like master craftsmen;
when they are called upon to construct some object they must exploit their
box of tools. Novices have only a few such tools, and even these may not
be sufficiently sharp. With time more tools are acquired, but almost all
craftsmen tend to continue using those tools with which they have the most
experience. The purpose of this book is to fill the toolbox with tools, and to
help the DSP professional become more proficient in their proper use. Even
people working in the field several years will probably find here new tools
and new ways of using tools already acquired.
I would like to thank my students, who had to suffer through courses
with no textbook and with continually changing syllabus, for their com-
ments; my colleagues, particularly Yair Karelic, Mauro Caputi, and Tony
Grgas, for their conscientious proofreading and insights; my wife Ethel for
her encouragement (even allowing me untold late-night sessions banging
away at the keyboard, although she had long ago banished all computers
from the house); and our two girls, Hanna and Noga, who (now that this
book is complete) will have their father back.
1
2 INTRODUCTIONS
these functions that we have come to grasp their extreme complexity. Due
to this inherent complexity, researchers and implementors attempting to
mechanize these functions have turned to the strongest and most intelligent
devices available. The most sophisticated and capable invention humankind
has devised to date is the digital computer. For this reason it is natural
that much of the state-of-the-art signal processing is performed digitally. In
this, the first chapter, we introduce signal processing, and more specifically
Digital Signal Processing, which from now on we shall call DSP.
In order to acquaint the reader with the concept of using digital technol-
ogy in order to process signals, we will first trace the early history of signal
processing. We then jump ahead to a survey of state-of-the-art applications,
in order to convince the reader that the problem is still alive and interesting.
Next we introduce the concept of signal processing by demonstrating analog
signal processing on a simple example. Finally we present the basic ideas
behind the use of computers in signal processing.
velopment of the telegraph and telephone, which used the electrical signals,
and radio, which used electromagnetic waves. The objects being manipu-
lated remain physically existing quantities, although they became less and
less tangible.
The final stage (so far) in humankinds development entailed learning
to process information. Unlike material objects and signals, information is
entirely abstract and cannot really be said to exist in the physical world.
Information is like ideas, and while it can be quantified it is not held back
by physical limitations. The seeds of information-processing were sown with
the invention of writing and arithmetic, philosophy and algebra, art and
logic, but were brought to full fruition with the invention of the digital
computer. The computer can transcend nature by predicting physical phe-
nomena before they occur, simulating worlds that cannot exist, and creating
new information where none was before.
The marriage of the last two developments in mankinds history, i.e., uti-
lizing digital computation for the purpose of processing of signals in the real
world, is the objective of DSP. While perhaps not a major milestone in the
history of humankind, DSP is a significant enough endeavor to warrant study
by all interested in manipulating their world using information-processing
techniques.
EXERCISES
1.1.l Does Digital Signal Processingmean the processingof digital signals or the
digital processing of signals ?
1.1.2 What possible relationships might there be between DSP and the following
computer science fields?
1. Numerical Analysis
2. Compiler Design
3. Operating Systems
4. Database Programming
5. Artificial Intelligence
1.1.3 Listen to an extremely weak station on an AM radio. Can you understand
what is being said? Would you be able to understand were the language
spoken to be one in which you are not completely proficient? What happens
if there are interference and whistles? Other radio stations? Does the same
happen with an FM radio station ? Repeat the above experiment with a
shortwave radio. Find stations using SSB modulation. What happens if you
do not tune the signal in properly? Sit in a cocktail party where many groups
of people are talking. Focus on one conversation after another. How well can
you separate out voices? What have you learned from this exercise?
4 INTRODUCTIONS
Radar signals usually have very high bandwidths, and consequently require
very fast processing rates. Sonar bandwidths are much lower than those
of radar, but the processing power required is high due to the interference
being stronger, and the return signals being weaker and more distorted.
Multipath reception complicates the location effort and often arrays of sen-
sors are employed and beamforming used. Electronic intelligence (ELINT)
and electronic warfare (EW) exploit interception of radar signals in order to
detect/identify and to deceive/defeat the radar system, respectively.
Seismology. Seismic signal analysis is used by the oil and gas industries
in the exploration of subsurface hydrocarbon reserves; by government agen-
cies for nuclear detonation detection; and by long-term planning authorities
for investigation of subsurface geological formations and their significance
to architecture and urban development. Signals passively collected during
naturally occurring seismic events such as earthquakes and volcanic erup-
tions may aid in their detection, epicenter location, and prediction. During
active exploration such seismic disturbances must be initiated, for example,
by setting off high-energy charges (although environmental considerations
may mandate the use of lower energy sources such as acoustic speakers). The
seismic waves are scattered by interfaces between different geological strata,
and collected at the earths surface by an array of seismometers. Thus multi-
ple seismic signals must be digitized and processed to lead to source location
or mapping of the geological strata.
EXERCISES
1.2.1 What other areas utilize signal processing? List several applications not on
the above list. Research at a library or search the Internet.
1.2.2 What areas may potentially benefit from signal processing, but are not Yet
using it? Write up a detailed description and submit to the patent office.
1.2.3 Consider a mobile robot able only to avoid obstacles, and to locate an electric
outlet when its batteries are low. What technologies would be needed to
implement such a robot? Where is DSP needed? Now give the robot the
ability to receive verbal commands, to retrieve objects, and to keep its owner
informed. What DSP is needed now?
1.2.4 Dual Tone Multi Frequency (DTMF) tones consists of two frequencies. The
same is true for dial tone and ring-back tone. What simple tunes can you play
recognizably using DTMF? Who answers the phone when you do? Why are
two different frequencies used-wouldnt it be easier to use only one? (Hint:
Take the expense of incorrectly routed calls into account.)
1.3. ANALOG SIGNAL PROCESSING 7
1.2.5 Using a computer with multimedia capabilities, record some naturally spoken
speech and observe its waveform graphically. Does this picture contain all the
information we can obtain from listening to the speech? You can easily find
long silences and tell the difference between whispering and screaming. Try to
tell where the words begin and end. Can you differentiate between male and
female voices? Can you guess what is being said? Assuming you answered in
the affirmative to the first question, where exactly is the information?
1.2.6 You are given the job of saving the several megabytes of information from
an old computer, about to be discarded. The computer has no serial output
ports or modem, but does have an analog output that can produce 256 dif-
ferent voltage levels. What is the simplest encoding method for outputting
information? How can you decode and store the information (you can use
any readily available computer or peripheral)? How fast can you go? Do you
think it can be decoded this fast? What are the real limitations? What hap-
pens if background noise is recorded along with the signal? This is the basic
idea behind the download path of the so-called PCM modem that achieves
56 Kb/s over telephone lines.
1.2.7 Same problem but this time the computer has an internal speaker and can
generate tones of different frequencies (all of the same amplitude). You may
decide to convert the data to be saved, byte by byte, into one of 256 tones, and
to record the tones onto an audio cassette. Design a transmitter (modulator)
for this case (try writing a program). What do you need to decode this
information (demodulator)? How fast can you go? Perhaps you decide to
convert the data to be saved, bit by bit, into one of only two tones. What
do the modulator and demodulator look like now? This is the FSK modem,
capable of 300 b/s on phone lines.
\
DELAY
l
One way of producing the desired effect is depicted in Figure 1.1, with
the input, intermediate signals, and output drawn in Figure 1.2. The first
step is to pass the signal (waveform X) through a capacitor that acts as a
DC blocker. This ensures that the signal values are centered around zero
voltage (waveform A). Next we put the signal through a hard limiter. This is
an amplifier driven to its maximum amplification, so that its output will be
+v max for any positive input, and -Vmax for any negative input (waveform
B). Next we split the signal so that it traverses two paths, one slightly de-
layed with respect to the other (this delay determines the width of the pulse
to be obtained). The delayed signal is now subtracted from the nondelayed
version, producing an output that is almost always zero (waveform C). The
subtraction is performed, once again, using an amplifier, this time a difer-
ential amplifier that has noninverting and inverting inputs. The amplifiers
output is nonzero and positive at the leading edge of the square wave (since
1.3. ANALOG SIGNAL PROCESSING 9
there we have +I(,,, - -V,,,) and nonzero and negative one half-cycle
later. This latter artifact is eliminated by the use of a half-wave rectifier,
a component that only passes positive voltages, suppressing negative ones.
The final result (waveform Y) is a narrow positive pulse locked to the leading
edge of the original sine, as required.
EXERCISES
1.3.1 The above example assumes the existence of a delay element, which may be
quite difficult to implement. For high-frequency signals, a long piece of cable
may be used, relying on the finite speed of propagation of the signal through
the cable to introduce the time delay. For low frequencies, even extremely
long lengths of cable introduce delays that are insignificant fractions of the
period. Assume you have an analog differentiator, a device whose output is
the derivative of its input. How would you use it in our sine to pulse converter?
What would the output pulse look like?
1.3.2 The device we described above is basically a zero crossing detector, a device
that determines when the signal goes through zero voltage. We can avoid the
need for a rectifier if we employ a peak picker, which outputs pulses at the
maxima of the input signal. How can a peak picker be implemented given a
differential amplifier and a reference voltage source?
1.3.3 How can an integrator (a device whose output is the integral of its input) be
used to solve our problem?
10 INTRODUCTIONS
1.3.4 Assume you have a digital representation of the input; that is, a sequence of
voltage measurements uniformly spaced in time. Write software routines for
the zero crossing detector and the peak picker, assuming that the sine wave
is sampled very densely (many equally-spaced samples per cycle). Will your
routines work if the sampling rate is much lower, for example, eight samples
per cycle? Four samples per cycle?
antialiasing reconstruction
- filter - A/D = DSP Processor = WA = filter r
The purpose of the filters will only become clear later on (see Sec-
tion 2.10). The blocks marked A/D and D/A represent devices that convert
Analog signals into Digital ones, and vice versa. These devices allow us to
translate signals in the physical world into sequences of numbers that com-
puters can accept as input and process, and to convert sequences of numbers
output by computers back into physical signals.
The heart of the system is the digital signal processor, which we shall
usually just call the DSP. (This double use of the acronym DSP should not
be confusing, with the differentiation between the processor and process-
ing being easily understood from context.) You may think of the DSP as
basically a computer that performs the needed computation. It may be a
general-purpose computer, such as a desktop workstation, readily available
and easily programmed. Or it may be special purpose digital hardware de-
signed specifically for the task at hand. Intermediate between these extremes
is a general-purpose programmable digital signal processor. These DSP chips
are similar to microprocessors, with arithmetic capabilities, memory access,
input and output ports, etc. However, as we shall discuss in Chapter 17,
they are augmented with special commands and architecture extensions in
order to make them particularly efficient for computations of the type most
1.4. DIGITAL SIGNAL PROCESSING 11
0 greater functionality,
l accuracy and reproducibility,
l modularity and flexibility,
l increased price/performance, and
l reduced time-to-market.
There are, admittedly, a few drawbacks associated with the use of DSP.
The most notable of these are:
As a result of the first problem many applications, for example, those dealing
with real-time processing of high bandwidth signals, cannot yet be handled
digitally. The second shortcoming is more a hindrance than a true impedi-
ment, compelling us to analyze our numeric algorithms more cautiously. The
third drawback is actually a favorable opportunity for students of DSP. It
ensures a steady demand for competent DSP personnel for many years to
come.
EXERCISES
1.4.1 Some DSP practitioners rarely deal with DSP theory at all, rather are
experts at programming DSPs for control and general algorithmic applica-
tions, rather than as a DSP. The acronym DSP appears four times in this
sentence. Explain which of the various meanings (processing, block diagram
function, programmable processor) best matches each.
1.4.2 Figure 1.3 depicts a situation with exactly one input signal and one output
signal. Describe an application with no inputs and one analog output. Two
analog inputs and one output. One input and two outputs. Can there be
useful applications with no outputs?
1.4.3 An amplifier increases the magnitude of a signal, while an attenuator de-
creases the magnitude. An inverter inverts the polarity of a signal, while
a clipper limits the magnitude of a signal. A DC blocker shifts the average
to zero. What mathematical functions are performed by these components?
Code a routine for each.
1.4.4 What differences do you expect to find between DSPs and conventional
CPUS?
1.4.5 Are there functions that can be performed in analog electronics, but cannot
be performed in DSP?
1.4.6 Compare digital Compact Disc (CD) technology with the older Long Playing
(LP) records. Explain why CD technology has totally replaced LPs by con-
sidering sound quality, playing duration, noise, stereo separation, the effect
of aging: media, and the ability to make copies.
Part I
Signal Analysis
2
Signals
We are now ready to commence our study of signals and signal processing
systems, the former to be treated in Part I of this book and the latter in Part
II. Part III extends the knowledge thus gained by presentation of specific
algorithms and computational architectures, and Part IV applies all we will
have learned to communications and speech signal processing.
At times one wants to emphasize signals as basic entities, and to consider
systems as devices to manipulate them or to measure their parameters. The
resulting discipline may then be called signal analysis. At other times it is
more natural to consider systems as the more fundamental ingredients, with
signals merely inputs and outputs to such systems. The consequence of this
viewpoint is called signal processing. This term is also most commonly used
when it is not clear which aspect one wishes to stress.
In this chapter we introduce the concept of a signal. We will see that
there are analog signals and digital signals, and that under certain conditions
we can convert one type into the other. We will learn that signals can be
described in terms of either their time or frequency characteristics, and that
here too there are ways to transform one description into the other. We
present some of the simplest signals, and discover that arbitrary signals can
be represented in terms of simple ones. On the way we learn how to perform
arithmetic on signals, and about the connection between signals and vectors.
15
16 SIGNALS
Definition: signal
An analog signal s is a finite real-valued function s(t) of a continuous variable
t (called time), defined for all times on the interval -oo < t < +oo. A digital
signal s is a bounded discrete-valued sequence sn with a single index n (called
discrete time), defined for all times n = -oo . . . + 00. n
Definition: energy
The energy of an analog or digital signal s is defined to be
E, = O s2 (t)dt A D E, = 5 s;
s --oo (24
7X=--00
the sum (or integral for the analog case) of the signals values squared.
we shant violate at some time or other. In practice entities are more often
considered signals because of the utility in so doing, rather than based on
their obeying the requirements of this definition (or any other).
In addition to all its possibly ignorable requirements, our definition also
leaves something out. It is quiet about any possible connection between ana-
log and digital signals. It turns out that a digital signal can be obtained from
an analog signal by Analog to Digital conversion (the A/D of Figure 1.3)
also known as sampling and digitizing. When the sampling is properly car-
ried out, the digital signal is somehow equivalent to the analog one. An
analog signal can be obtained from a digital signal by Digital to Analog
conversion (the D/A block), that surprisingly suffers from a dearth of al-
ternative names. Similar remarks can be made about equivalence. A/D and
D/A conversion will be considered more fully in Section 2.7.
EXERCISES
2.1.1 Which of the following are signals? Explain which requirement of the def-
inition is possibly violated and why it is acceptable or unacceptable to do
so.
1. the height of Mount Everest
2. (eit + ewi>
3. the price of a slice of pizza
4. the sin? function F
5. Eulers totient function 4(n), the number of positive integers less than
n having no proper divisors in common with n
6. the water level in a toilets holding tank
7. [tJ the greatest integer not exceeding t
8. the position of the tip of a mosquitos wing
9. fi
10. the Dow Jones Industrial Average
11. sin($)
12. the size of water drops from a leaky faucet
13. the sequence of values zn in the interval [0 . . . l] defined by the logistics
recursion zn+l = Xx, (1 - xn) for 0 5 X 5 4
2.1.2 What is the power of s(t) = Asin( The RMS value?
2.1.3 A signals peak factor is defined to be the ratio between its highest value and
its RMS value. What is the peak factor for s(t) = Asin( The sum of N
sinusoids of different frequencies?
20 SIGNALS
2.1.4 Define a size measure M for signals diRerent from the energy (or RMS value).
This measure should have the following properties.
l The zero signal must have zero measure MO = 0, and no other signal
should have zero measure.
l If signal y is identical to signal x shifted in time then M3, = M,.
0 If yn = CYX~for all times, then Mg > Mz if cy > 1 and My < M, if
cy < 1.
l If yn > x,., almost all of the time, then My > M,.
What advantages and disadvantages does your measure have in comparison
with the energy?
s(t) = 1 A D sn = 1 (2.3)
Although this is the simplest signal we can imagine, it has infinite energy,
and therefore violates one of the finiteness constraints. Hence technically it
isnt really a signal at all! Arbitrary constant signals can be obtained by
multiplying the unit constant signal by appropriate values. The constant
signal, depicted in Figure 2.1, although admittedly trivial, can still be use-
ful. We will often call it Direct Current (DC), one of the many electronics
--i- I--
Figure 2.1: The constant signal. In (A) we depict the analog constant and in (B) the
digital constant.
2.2. THE SIMPLEST SIGNALS 21
Figure 2.2: The unit step signal. In (A) we depict the analog step u(t) and in (B) the
digital step un.
terms imported into signal processing. The gist is that a batterys voltage
is constant, w(t) = VO, and consequently induces a current that always flows
in one direction. In contrast the voltage from a wall outlet is sinusoidal,
w(t) = VOsin(&), and induces an Alternating Current (AC).
We cannot learn much more from this signal, which although technically
a function of time in reality is not time dependent at all. Arguably the
simplest time-dependent signal is the unit step, which changes value at only
one point in time (see Figure 2.2). Mathematically, the analog and digital
unit step signals are:
A D y E;: (2.4)
-
A f B s
0
Fit -72
Figure 2.3: The unit impulse. In (A) we depict an analog impulse of unity width. In (B)
the digital unit impulse 6,,0.
symbol for this signal, we utilize one known as the Kronecker delta 6n,m.
This doubly indexed entity is defined to be one, if and only if its indices are
equal; otherwise it is zero. In terms of the Kronecker delta, the digital unit
impulse is sn = &a.
The full Kronecker delta corresponds to a Shifted Unit Impulse (SUI)
Sn = 6 n,rn (2 . 5)
that is zero for all times except for time n = m, when it equals one. The
importance of the set of all SUIs will become clear in Section 2.5.
One might similarly define an analog unit impulse by subtracting analog
unit steps, obtaining the Figure 2.3.A. This analog signal flagrantly displays
two jump discontinuities, but by now that should not make us feel uncom-
fortable. However, this is not the signal usually referred to as the analog unit
impulse. There is no profound meaning to the width of this signal, since in
the analog world the meaning of a unit time interval depends on the time
units! What is meaningful is the energy of the impulse, which is its ampli-
tude squared times its width. There are good reasons to expect that once
the width is small enough (i.e., small compared to all significant times in
the problem) all impulses with the same energy will have basically the same
effect on systems. Accordingly, when one speaks of a unit impulse in the
analog domain, conventionally this alludes to a unit energy impulse. Of
course the unit width impulse in Figure 2.3 is a unit impulse in this sense;
but so are all the others in Figure 2.4.
The unit energy impulses in the figure are given by:
0 (tl>T
I(t) =
{ & Itl < T
2.2. THE SIMPLEST SIGNALS 23
S
t
t
Figure 2.4: Analog unit energy impulses. Since all of these signals have the same energy,
the height increases as the width decreases. The vertical arrow is a symbolic way of
designating Diracs delta function.
and clearly there can be no such function! However, Diracs delta is such an
extremely useful abstraction, and since its use can be justified mathemati-
cally, we shall accept it without further question. Indeed, Diracs delta is so
useful, that when one refers without further qualification to the analog unit
impulse, one normally means b(t).
The next signal we wish to discuss is the square wave 0 (t), depicted in
Figure 2.5.A. It takes on only two values, 33, but switches back and forth
between these values periodically. One mathematical definition of the analog
square wave is
14 is even
q (t) = 11 1tJis odd (2.7)
24 SIGNALS
Figure 2.5: Three periodic analog signals. In (A) we depict the square wave, in (B) the
triangle wave and in (C) the sawtooth.
where LtJ (pronounced floor of t) is the greatest integer less than or equal
to the real number t. We have already mentioned that this signal has an
infinite number of jump discontinuities, and it has infinite energy as well!
Once again we can stretch and offset this signal to obtain any two levels,
and we can also change its period from unity to T by employing q (t/T).
We can further generalize the square wave to a rectangular wave by having
it spend more time in one state than the other. In this case the percentage
of the time in the higher level is called the duty cycle, the standard square
wave having a 50% duty cycle. For digital signals the minimal duty cycle
signal that is not a constant has a single high sample and all the rest low.
This is the periodic unit impulse
(2.8)
where the period is P samples.
Similarly we can define the analog triangle wave n(t) of Figure 2.5.B
and the sawtooth 7(t) of Figure 2.5.C. Both, although continuous, have
slope discontinuities. We leave the mathematical definitions of these, as well
as the plotting of their digital versions, to the reader. These signals pop
up again and again in applications. The square wave and its close brethren
are useful for triggering comparators and counters, the triangle is utilized
when constant slope is required, and the sawtooth is vital as the time base
of oscilloscopes and the raster scan in television. Equipment known as
function generators are used to generate these signals.
2.2. THE SIMPLEST SIGNALS 25
Figure 2.6: Sinusoidal signals. In (A) we depict the analog sinusoid with given amplitude,
frequency and phase. In (B) the digital sinusoid is shown.
Of course the most famous periodic signal is none of these, but the sine
and cosine functions, either of which we call a sinusoid.
The connection between the frequency f of an analog sinusoid and its period
T can be made clear by recalling that the sine function completes a full cycle
after 27r radians. Accordingly, the frequency is the reciprocal of the period
=-
f f
and its units must be fu2l cycles per second, also known as Hertz or Hz.
The period represents the number of seconds per cycle while the frequency
in Hz describes the number of full cycles per second. Since discrete time n
carries no units, the digital frequency fd will be essentially a pure number.
The periodicity of digital sinusoids will be discussed later.
In order to avoid factors of 2n we often rewrite equation 2.9 as follows.
depicted in Figure 2.7 for negative A. For positive A and any finite time this
function is finite, and so technically it is a well-behaved signal. In practice
the function explodes violently for even moderately sized negative times,
and unless somehow restricted does not correspond to anything we actually
see in nature. Mathematically the exponent has unique qualities that make
it ideal for studying signal processing systems.
We shall now do something new; for the first time we will allow complex-
valued functions. We do this by allowing the constant in the argument of the
exponential to be a pure imaginary number A = iw, thus radically chang-
2.2. THE SIMPLEST SIGNALS 27
0 s
0 t
B l
..
.o l .0
+n
Figure 2.7: Exponentially decreasing signals. In (A) we depict the analog exponential,
in (B) the digital.
ing the character of the signal. Recalling the remarkable identity (equa-
tion (A.7))
eicp= cos(cp) + isin
we see that exponentials with imaginary coefficients are complex sinusoids.
When we deal with complex signals like s(t) = AeiWt, what we really mean
is that the real-world signal is the real part
while the imaginary part is just that-imaginary. Since the imaginary part
is 90 (one quarter of a cycle) out of phase with the real signal, it is called
the quadrature component. Hence the complex signal is composed of in-phase
(real) and quadrature (imaginary) components.
At first it would seem that using complex signals makes things more
complex but often the opposite is the case. To demonstrate this, consider
what happens when we multiply two sinusoidal signals s1(t) = sin(wl t) and
$2 v> = sin(o2t). The resulting signal is
which combines the exponential with the sinusoid. For negative X, this is a
damped sinusoid, while for positive X it is an exponentially growing one.
Summarizing, we have seen the following archetypical simple signals:
unit constant s(t) = 1 Sn- -1
unit step s(t) = u(t) Sn = Un
unit impulse s(t) = d(t) Sn = 6,,0
square wave s(t)= q (C3t) Sn = q l(W,n)
sinusoid s(t) = Asin(wt + 4) sn = A sin(wdn + 4)
damped sinusoid s(t) = Aevxt sin(wt + 4) sn = Aa+ sin(wdn + 4)
real exponential s(t) = AeXt Sn = Cyn
complex sinusoid s(t) = Ae1(Wt+4) Sn = Ae%W++>
damped complex sinusoid s(t) = Ae(x+lw)t sn = Ae@+iwd>n
EXERCISES
2.2.1 Thomas Alva Edison didnt believe that AC electricity was useful, since the
current first went one way and then returned. It was Nikola Tesla who claimed
that AC was actually better than DC. Why was Edison wrong (hint: energy)
and Tesla right (hint: transformers)?
2.2.2 In the text we depicted digital signals graphically by placing dots at signal
values. We will usually use such dot gruphs, but other formats are prevalent
as well. A comb gruph uses lines from the time axis to the signal point; a slint
graph (straight line interpolation) simply connects successive signal values;
comb-dot and slint-dot combinations are useful when the signal takes on zero
values. These formats are depicted in Figure 2.8. Write general routines for
plotting digital signals in these formats in whatever computer programming
language you usually use. Depending on your programming language you
may first have to prepare low-level primitives. Plot the digital sinusoidal
signal sn = sin(w,n) for various frequencies CJin all of these formats. Decide
which you like the best. You may use this format from now on.
2.2. THE SIMPLEST SIGNALS 29
Figure 2.8: Different formats for graphical representation of digital signals. In (A) we
depict a signal using our usual dot gruph. In (B) th e same signal is plotted as a comb graph.
In (C) it is graphed as a Ant graph. (D) and (E) are comb-dot and slint-dot representations
respectively.
2.2.3 Give mathematical definitions for the analog triangle signal A(t) of Fig-
ure 2.5.B and for the analog sawtooth saw(t) of Figure 2.5.C.
2.2.4 What is the integral of the square wave signal? What is its derivative?
2.2.5 Using your favorite graphic format plot the digital square wave, triangle wave
and sawtooth, for various periods.
2.2.6 Perform the following experiment (you will need an assistant). Darken the
room and have your assistant turn on a pen-flashlight and draw large circles
in the air. Observe the light from the side, so that you see a point of light
moving up and down. Now have the assistant start walking while still drawing
circles. Concentrate on the vertical and horizontal motion of the point of light,
disregarding the depth sensation. You should see a sinusoidal signal. Prove
this. What happens when you rotate your hand in the opposite direction?
What can you infer regarding negative frequency sinusoids?
2.2.7 Diracs delta function can be obtained as the limit of sequences of functions
other than those depicted in Figure 2.4. For example,
0 t<o
asymmetric unit impulses ZT@) = & O<t<T
0 t>T
Gaussian functions Gu (4
sin(wt)
Sine functions $sinc,(t) =-
7rt
Lorentzian functions
d(t)=&I
2.2.10 Show that
O f(t)b(t - t)dt = f(t)
J -CO
both graphically and by using basic calculus. From this result show that J(t)
must be zero for all nonzero arguments. Compare the above relation with the
Fourier identity
l deterministic or stochastic
l if deterministic: periodic or nonperiodic
l if stochastic: stationary or nonstationary
l of finite or infinite time duration
l of finite bandwidth or of full spectrum
(at least over short time scales-but see Section 5.5) and well-behaved math-
ematically. Stochastic signals are generated by systems that contain random-
ness (see Section 5.6). At any particular time the signal is a random variable,
(see Appendix A.13), which may have well defined average and variance, but
is not completely defined in value. Any particular sequence of measurements
of the signals values at various times captures a specific instantiation of the
stochastic signal, but different sequence of measurements under the same
conditions would retrieve somewhat different values.
In practice we never see a purely deterministic signal, since even the
purest of deterministic signals will inevitably become contaminated with
noise. Pure noise is the name we give to a quintessential stochastic signal,
one that has only probabilistic elements and no deterministic ones. When a
deterministic signal becomes contaminated with additive noise, as depicted
in Figure 2.9,
dt> = dt> + n(t)
we can quantify its noisiness by the Signal to Noise Ratio (SNR). The
SNR is defined as the ratio of the signal energy to the noise energy, and is
normally measured in dB. (equation (A.16))
J%
SNR(dB) = 10loglo z;r = IO (log,a & - loglo En) (2.13)
Figure 2.9: Deterministic signal (simple sine) with gradually increasing additive noise.
In (A) the deterministic signal is much stronger than the noise, while in (D) the opposite
is the case.
32 SIGNALS
When measuring in, we usually talk about the signal as being above the
noise by SNR(dB).
Not all the signals we encounter are stochastic due solely to contami-
nation by additive noise. Some signals, for example speech, are inherently
stochastic. Were we to pronounce a single vowel for an extended period of
time the acoustic signal would be roughly deterministic; but true speech is
random because of its changing content. Speech is also stochastic for an-
other reason. Unvoiced sounds such as s and f are made by constricting air
passages at the teeth and lips and are close to being pure noise. The h sound
starts as noise produced in the throat, but is subsequently filtered by the
mouth cavity; it is therefore partially random and partially deterministic.
Deterministic signals can be periodic, meaning that they exactly repeat
themselves after a time known as the period. The falling exponential is not
periodic, while the analog sine Asin(2rft), as we discussed above, is peri-
odic with period T = $. The electric voltage supplied to our houses and the
acoustic pressure waves from a flute are both nearly perfect sinusoids and
hence periodic. The frequency of the AC supplied by the electric company
is 60 Hz (sixty cycles per second) in the United States, and 50 Hz (fifty
cycles per second) in Europe; the periods are thus 16; and 20 milliseconds
respectively. The transverse flutes used in orchestral music can produce fre-
quencies from middle C (524 Hz) to about three and a half octaves, or over
ten times, higher!
While the analog sinusoid is always periodic the digital counterpart is
not. Consider an analog signal with a period of 2 seconds. If we create a
digital sinusoid by sampling it 10 times per second, the digital signal will
be periodic with digital period 20. However, if we sample at 10.5 times per
second, after 2 seconds we are a half-second out of phase; only after four
seconds, (i.e., 21 samples) does the digital signal coincide with its previous
values. Were we to sample at some other rate it would take even longer for
the digital version to precisely duplicate itself; and if ratio of the period to
the sampling interval is not rational this precise duplication will never occur.
Stochastic signals may be stationary, which means that their probabilis-
tic description does not change with time. This implies that all the signals
statistics, such as the mean and variance, are constant. If a stochastic signal
gets stronger or weaker or somehow noisier with time, it is not stationary. For
example, speech is a stochastic signal that is highly nonstationary; indeed
it is by changing the statistics that we convey information. However, over
short enough time intervals, say 30 milliseconds, speech seems stationary
because we cant move our mouth and tongue this fast.
2.4. SIGNAL ARITHMETIC 33
EXERCISES
2.3.1 Look closely at the graphs of the digital sinusoid sn = sin(wn) that you
prepared in exercise 2.2.2. When is the digital sinusoid periodic? Under what
conditions is the period the same as that of the analog sinusoid? Verify the
statement in the text regarding nonperiodic digital sinusoids.
2.3.2 The purpose of this exercise is to examine the periodicity of the sum of two
analog sines. For example, the sum of a sine of period 4 seconds and one of
period 6 seconds is periodic with period 12 seconds. This is due to the first
sine completing three full periods while the second competes two full periods
in 12 seconds. Give an example of a sum that is not periodic. Give a general
rule for the periodicity. What can be said about cases when the sum is not
exactly periodic?
2.3.3 Plot analog signals composed of the sum of two sinusoids with identical am-
plitudes and frequencies jr and f2. Note that when the frequencies are close
the resultant seems to have two periods, one short and one long. What are
the frequencies corresponding to these periods? Prove your assertion using
the trigonometric identities.
Y = ax Y = ax
means AD means (2.14)
y(t) = ax(t) Vt yn = ax, Vn
that is, we individually multiply every signal value by the real number.
It might seem overly trivial even to define this operation, but it really is
important to do so. A signal is not merely a large collection of values, it is
an entity in its own right. Think of a vector in three-dimensional space (a
3-vector). Of course it is composed of three real numbers and accordingly
doubling its size can be accomplished by multiplying each of these numbers
by two; yet the effect is that of creating a new 3-vector whose direction is the
same as the original vector but whose length is doubled. We can visualize
this operation as stretching the 3-vector along its own direction, without
thinking of the individual components. In a similar fashion amplification
of the signal should be visualized as a transformation of the signal as a
whole, even though we may accomplish this by multiplying each signal value
separately.
We already know that multiplication of a signal by a real number can
represent an amplification or an attenuation. It can also perform an inversion
Y = -X Y = -X
means AD means (2.15)
Y(t) = -x(t) vt Yn = -57-t h
y=Revz y = Revz
means AD means (2.16)
Y(t) = x(-t) vt Yn =X-n Vn
which makes it run backwards in time. If you whistle a constant note it will
sound the same when reversed, but if you whistle with ascending pitch the
reversed signal will have descending pitch. This operation has no counterpart
for n-vectors.
Frequently we will need to add two signals,
z =x+y 2 =x+y
means AD means (2.17)
z(t) = x(t) + y(t) vt zn = Xn +yn Vn
one simply adds the values. This is the familiar addition of two n-vectors,
and is the similar to the addition of complex numbers as well. Signal addition
is commutative (x + y = y + x) and associative (x + (y + Z) = (x + y) + Z)
and adding a signal to its inversion yields the zero signal. Hence signals, like
real numbers, complex numbers, and n-vectors, obey all the normal rules of
arithmetic.
We will also need to multiply two signals, and you have probably already
guessed that
x = xy x=xy
means AD means (2.18)
z(t) = x(t) y(t) vt Xn=Xnyn Vn
one simply multiplies value by value. Multiplication of a signal by a num-
ber is consistent with this definition of multiplication-just think of the
number as a constant signal. However, this multiplication is different from
multiplication of 3-vectors or complex numbers. The usual dot product
multiplication of two 3-vectors yields a scalar and not a 3-vector. There is
a cross or vector product kind of multiplication that yields a vector, but it
doesnt generalize to n-vectors and it isnt even commutative. Multiplication
of complex numbers yields a complex number, but there
r=x-y r =x:y
means AD means (2.19)
r = JZ& x(t)y(t)dt r = C~iLcoXnYn
This is the proper definition for real signals; although it can be extended for
complex signals. The energy of a signal is the dot product of the signal with
itself, while the dot product of two different signals measures their similarity
(see Chapter 9). Signals for which the dot product vanishes are said to be
orthogonal, while those for which it is large are said to be strongly correlated.
For digital signals there is another operator known as the time advance
operator z,
Y = 2x means Yn = G-b+1 Vn (2.20)
which would certainly be meaningless for vectors in space. What meaning
could there possibly be for an operator that transforms the x coordinate
of a vector into the y coordinate? However, signals are not static vectors;
they are dynamic entities. The time variable is not a dummy variable or
index; it is physical time. We can always renumber the axes of a vector,
thus scrambling the order of elements, and still understand that the same
physical vector is described. For signals such an action is unthinkable. This
is the reason that Rev(x) had no vector counterpart. This is the reason that
our original definition of signal emphasized that the independent variable or
index was time.
You can think of z as the just wait a little while and see what happens
operator. For digital signals the natural amount of time to wait is one unit,
from n to n + 1. If we wish to peek further forward in time, we can do so.
For example, we can jump forward two units of time by first advancing one
unit and then one more
and so on.
We may also wish to go backwards in time. This doesnt require us to
invent a time machine, it just means that we wish to recall the value a
signal had a moment ago. A little reflection leads us to define the time delay
operator z-l
y = z-lx means Yn = Xn-1 Vn (2.21)
2.4. SIGNAL ARITHMETIC 37
so that z z-l x = z-l z x = x. The operator z-l will turn out to be even more
useful than z, since it is usually easier to remember what just happened than
to predict what is about to occur. The standard method for implementing
the digital delay of L units of time is through a FIFO buffer of length L.
A signal value that enters the FIFO at time n exits at time n + L, and so
the output of the FIFO is delayed exactly L time units with respect to its
input. When used in this fashion the FIFO is called a delay line.
We can make these operators more concrete with a simple example. In
exercise 2.1.1.13 we introduced a family of recursively defined signals, often
called the logistics signals
where the xn are all in the range 0 2 X~ 5 1. In order to enforce this last
restriction we must restrict X to be in the range 0 5 X < 4. A particular
signal in this family is determined by giving x0 and X. It is most instructive
to generate and plot values for various x0 and X, and the reader will be
requested to do so as an exercise. In this case the operation of the time
advance operator can be simply specified
zx = Xx(1 - 2)
As, = sn - s~-~
The finite difference operator for digital signals is vaguely similar to the dif-
ferentiation operator for continuous signals. Common characteristics include
linearity and the fact that they are identically zero only for a constant. A is a
38 SIGNALS
linear operator since for any two signals II: and y, A@+ y) = AZ+ Ay and for
any number c and signal 2, Acx = cAx. As = 0 (the zero signal) if and only
if the signal is constant. In other ways finite differences are similar to, but
not identical to derivatives. For example, A(xy) = xAy + Ax z-l y. In some
things finite differences are completely different, e.g., Aan = o?( 1 - CV-l).
This last example leads us to an important property of the time delay
operator. For the exponential signal sn = enn it is easy to see that
so that
z-is = eBAs
i.e., the operation of time delay on the exponential signal is equivalent to
multiplication of the signal by a number. In linear algebra when the effect
of an operator on a vector is to multiply it by a scalar, we call that vector
an eigenvector of the operator. Similarly we can say that the exponential
signal is an eigensignal of the time delay operator, with eigenvalue e-*
The fact that the exponential is an eigensignal of the time delay operator
will turn out to be very useful. It would have been even nicer were the
sinusoid to have been an eigensignal of time delay, but alas equation (A.23)
tells us that
EXERCISES
2.4.1 Show that the exponential signal s, = AeAn is an eigensignal of the time
advance operator. What is its eigenvalue? The real sinusoid sn = A sin(wn+qb)
is the eigensignal of an operator that contains z-l and zV2. Can you find this
operator?
2.4.2 What is the effect of the time advance operator on the unit impulse? Express
the general SUI Sn,m in terms of Sn,a and the time delay operator.
2.4. SIGNAL ARITHMETIC 39
2.4.3 Compare the energy of a time delayed, advanced, or reversed signal with
that of the original signal. What is the energy of y = ax in terms of that
of x? What can be said about the energy of the sum of two signals? For
example, consider summing two sinusoids of the same frequency but different
amplitudes and phases. What about two sinusoids of different frequencies?
Why is there a difference between these two cases?
2.4.4 Plot the logistics signal of equation (2.22) using several different x0 for each
X. Try X = 0.75 and various x0---what happens after a while? Next try
X = 1.5,2.0, and 2.75. How is the long time behavior different? Can you
predict the behavior as a function of X? Are there any starting points where
the previous behavior is still observed? Next try X = 3.2,3.5,3.55,3.5675,
and 3.75. What is the asymptotic behavior (for almost all x0)?
2.4.5 Using the program from the previous exercise try X = 3.826,3.625 and 3.7373.
What is the asymptotic behavior? Try X = 4. How is this different?
2.4.6 Canons are musical compositions composed of several related voices heard
together. The canonical relations require the voices to repeat the theme of
the first voice:
time offset: after a time delay,
key shift: in a different key,
diminution: at twice normal speed,
augmentation: at half normal speed,
inversion: with high and low tones interchanged,
crab order: time reversed,
or with combinations of these. Describe the signal processing operators that
transform the basic theme into the various voices. In order for the resulting
canon to sound pleasing, at (almost) every instant of time the voices must
be harmonically related. Can you write a program that composes canons?
2.4.7 In the text we discussed the usefulness of considering a signal as a single
entity. This exercise deals with the usefulness of considering a signal as a col-
lection of values. A streaming signal is a digital signal that is made available
as time progresses. When the signal is not being streamed one must wait for
the signal to be completely prepared and placed into a file before processing.
Explain the usefulness of streaming digital audio. In certain computer lan-
guages a stream is defined to be a sequentially accessed file. Compare this
use of stream with the previous one.
40 SIGNALS
Since signals form a vector space, the theorems of linear algebra guar-
antee that there is a basis {vk}, i.e., a set of signals in terms of which any
signal s can be expanded.
S =
c ckvk (2.24)
k
The use of the summation sigma assumes that there are a finite or denu-
merable number of basis signals; when a nondenumerable infinity of basis
signals is required the sum must be replaced by integration.
s= c(k)v(k) dk (2.25)
s
From linear algebra we can show that every vector space has a basis, but
in general this basis is not unique. For example, in two-dimensional space
we have the natural basis of unit vectors along the horizontal z axis and
vertical y axis; but we could have easily chosen any two perpendicular direc-
tions. In fact we can use any two nonparallel vectors, although orthonormal
vectors have advantages (equation (A.85)). Similarly, for the vector space of
signals there is a lot of flexibility in the choice of basis; the most common
choices are based on signals we have already met, namely the SUIs and the
sinusoids. When we represent a signal by expanding it in the basis of SUIs
we say that the signal is in the time domain; when we the basis of sinusoids
is used we say that the signal is in the frequency domain.
We are not yet ready to prove that the sinusoids are a basis; this will be
shown in Chapters 3 and 4. In this section we demonstrate that the SUIs are
a basis, i.e., that arbitrary signals can be uniquely constructed from SUIs.
We start with an example, depicted in Figure 2.10, of a digital signal that
is nonzero only between times n = 0 and n = 8. We build up this signal
by first taking the unit impulse 6,,0, multiplying it by the first signal value
SO,thereby obtaining a signal that conforms with the desired signal at time
42 SIGNALS
1 c so ha,0
+ Sl &,l
:I, + s2 ha,2
+- + s3 &x,3
+- + $4 &,4
+ c + s5 sn,5
+A + s6 &a,6
+ s7 &,7
+I
+1 + s8 &x,8
=w = sn
Figure 2.10: Comb-dot graph depicting building up a digital signal from shifted unit
impulses.
n = 0 but which is zero elsewhere. Next we take the shifted unit impulse
6n,l, which is nonzero only for n = 1, and multiply it by ~1, thus obtaining a
signal that agrees with sn for n = 1 but is otherwise zero. Adding together
these two signals we obtain a signal that is identical to the desired signal
both at time n = 0 and at time n = 1 but otherwise zero. We proceed in a
similar fashion to build up a signal that is identical to the desired signal for
all times.
In a similar fashion we can expand any digital signal in terms of SUIs
cm
Sn =
c %-n&,7-n (2.26)
n=--00
thus proving that these signals span the entire space. Now, it is obvious that
no two SUIs overlap, and so the SUIs are orthogonal and linearly indepen-
dent (no 6n,m can be expanded in terms of others). Therefore the SUIs are a
linearly independent set that spans the entire space, and so they are a basis.
Hence we see that the SUIs form a basis of the vector space of digital
signals. Since there are (denumerably) infinite signals in the basis, we see
that the vector space of signals is of infinite dimension. Similar statements
are true for analog signals as well. In Figure 2.11 we demonstrate approxi-
mating a function using shifted unit width analog impulses. We leave it for
the reader to complete the argument to show that any analog signal can be
2.5. THE VECTOR SPACE OF ALL POSSIBLE SIGNALS 43
rl
sou(t)
+- + Sl u(t - 1)
+ + s2 u(t - 2)
l-l
I + s3 u(t - 3)
+ U
t
+ s4 u(t - 4)
+ b
+ s5 u(t - 5)
+- + s6 U(t - 6)
+- +
+
s7 u(t - 7)
+- $3 U(t - 8)
=-q&I+ = Sn(t>
Figure 2.11: Building up an analog signal from shifted unit width impulses.
EXERCISES
2.5.1 Show that the triangle inequality is obeyed for signals.
ccSl,- 2(X81,
- +
s3J2 52J2 cc s2, - s3J2
2.5.2 Show that the set of digital signals of finite time duration is a finite dimension
vector space.
2.5.3 Express a general digital signal x, as a sum involving only the impulse at
time zero and time delay operators.
2.5.4 Lets try to approximate the S-vector v = (v,, vy, v,) by a vector parallel to
the x axis a,?. The best such approxrmation requires that the error vector
e=v-cxa, 2 l!k of minimal squared length. Show that this criterion leads to
a Z = v, acd that the error vector lies entirely in the y-z plane. Similarly,
show that best approximation of v by a vector in the x-y plane a, 2 + oy$,
requires cyZ = vZ and or, = vy , a<d for the error vector must be parallel co
the z axis. When can the error become zero?
44 SIGNALS
2.5.5 The previous exercise leads us to define the coefficients vi as those real num-
bers that minimize the approximation error. Use this same approach to find
the expansion of a given signal s(t) in terms of a set of normalized signals
vk(t), by requiring the error signal to be of minimal energy. Show that this
approach demystifies the use of equation (2.19) as the dot product for signals.
2.5.6 Show how to expand analog signals in terms of shifted Dirac delta functions,
by starting with Figure 2.11 and sending the impulse width to zero.
2.5.7 Explain why the set of all analog signals forms a vector space. What new
features are there? What is the dimensionality of this vector space?In what
sense are there more analog signals than digital ones?
2.5.8 Show that the set of all analog periodic signals with the same period is a
vector space. Is it denumerably or nondenumerably infinite in dimension?
f(t) x f(to) + $Ito(t - to). If You want to know what the function does even
further away, find the second derivative evaluated at to, and then the third
derivative, etc. Higher and higher derivatives allow one to stray further and
further from the original point in time. Knowing all derivatives at any one
point in time is equivalent to knowing the functions values at all times. This
law is called Taylors Theorem and is the very fabric of the classical analysis
way of looking at functions. It approximates functions using polynomials as
the basis for the vector space of functions.
The fundamental law of signal processing proclaims a different way of
representing signals. Real-world signals have finite energy and occupy some
finite bandwidth. Hence polynomials are not a natural basis for describing
them. The signal processing approximation is global rather that local, i.e.,
for any finite order is about as good (or bad) simultaneously for all times
- 00 < t < +oc . Rather than using derivatives and polynomials, the signal
processing way of looking at the world emphasizes spectrum and its basic sig-
nals are sinusoids. The signal processing law (the Fourier transform) states
that all signals can be approximated by summing together basic sinusoids.
Because of this unique way of representing signals, signal processing
tends to be quite schizophrenic. One has to continuously jump back and
forth between the time domain representation, which gives the value of the
signal for all times, and the frequency domain representation, where the
harmonic content of the signal at every frequency is given.
Spectrum is simply a shorter way of saying frequency domain represen-
tation, and the idea is probably not new to you. You surely realize that
the operation of a prism on white light consists of its decomposition into
different frequencies (colors). You certainly have tuned in a station on the
radio by changing the center frequency being demodulated. You may even
have an audio system with a graphic equalizer enables amplifying certain
component acoustic frequencies more than others.
The spectrum of a signal that consists of a pure sine wave has a single
line at the frequency of this sine. The sum of two sines corresponds to two
lines in the frequency domain. If the sum is weighted the relative heights of
these lines will reflect this. In general, any signal that can be constructed
by weighted combination of a finite number of sines will have a discrete
spectrum with lines corresponding to all the frequencies and weights.
Not all signals have spectra comprised of discrete lines. For example,
the analog unit width impulse has a sine-shaped spectrum, where the sine
function
w&f >
sine(f) = f
46 SIGNALS
A
I
c
Figure 2.12: The unit width analog impulse and its spectrum. In (A) we depict the
unit width impulse in the time domain, and in (B) its (sine-function) frequency domain
representation. The latter is the raw spectrum including negative frequencies.
(see Figure 2.12). The meaning of negative spectral values and negative fre-
quencies will become clear later on. The spectrum has a strong DC compo-
nent because the impulse is nonnegative. In order to make the infinitesimally
sharp corners of the impulse, an infinite range of frequencies is required. So
although this spectrum decreases with increasing frequency, it never be-
comes zero. Its bandwidth, defined as the spectral width wherein mast of
the energy is contained, is finite.
Signal processing stresses the dual nature of signals-signals have time
domain and frequency domain (spectral) characteristics. Although the signal
(time domain) and its Fourier transform (frequency domain) contain exactly
the same information, and indeed either can be constructed from the other,
some signal processing algorithms are more natural in one domain than in
the other. This dual way of looking at signals is what makes signal processing
different from mathematical analysis.
EXERCISES
2.6.1 Experiment with plotting signals composed of several sinusoids with various
frequencies and amplitudes. Can you recognize the original frequencies in
the resulting waveform? What do you observe when one sinusoid is much
stronger than the others? When all the frequencies are multiples of a common
frequency? When the frequencies are very close together? When they are well
separated? When does the signal seem unpredictable?
2.6.2 Taylor expand a sine wave (you can do this by hand since you only need to
know the derivatives of sinusoids). Fourier expand a parabola (it will probably
be easiest to use numeric Fourier transform software). What can you say
about the compactness of these descriptions?
2.7. ANALOG AND DIGITAL DOMAINS 47
2.6.3 The Taylor expansion can be interpreted as the expansion of arbitrary contin-
uous functions in a basis of polynomials. Are the functions fo(s) = 1, fl(z) =
z, f&c) = x2, f&T) = 53,. . . a basis? Are they an orthonormal basis?
2.6.4 Lets examine a more complex signal with a discrete line spectrum. The V.34
probe signal is composed of 21 sinusoids sin(2nft + 4) with frequencies f
that are multiples of 150 Hz, and phases 4 given in the following table.
Plot a representative portion of the final signal. What is special about the
phases in the table? (Hint: Try altering a few phases and replotting. Observe
the maximum absolute value of the signal.)
Figure 2.13: Conversion of an analog signal into a corresponding digital one involves
quantizing both axes, sampling time and digitizing signal value. In the figure we see the
original analog signal overlaid with the sampled time and digitized signal value grid. The
resulting digital signal is depicted by the dots.
are now reflected in the varying heights of the pulses. The process of varying
some aspect of a signal in order to carry information is called modulation.
In this case we have modulated the amplitudes of the pulse stream, and
so have produced Pulse Amplitude Modulation (PAM). Other aspects of
the pulse stream could have been varied as well, resulting in Pulse Width
Modulation (PWM), and Pulse Position Modulation (PPM). We now wish
to digitally record the amplitude of each pulse, which we do by giving each
a code, e.g. the binary representation of the closest quantization level. From
this code we can accurately (but not necessarily precisely) reconstruct the
amplitude of the pulse, and ultimately of the original signal. The resulting
sequence of numbers is called a Pulse Code Modulation (PCM) stream.
EXERCISES
2.7.1 It would seem that sampling always gives rise to some loss of information,
since it always produces gaps between the sampled time instants; but some-
times we can accurately guess how to fill in these gaps. Plot a few cycles of
a sinusoid by connecting a finite number of points by straight lines (linear
interpolation). How many samples per cycle are required for the plot to look
natural, i.e., for linear interpolation to accurately predict the missing data?
How many samples per cycles are required for the maximum error to be less
than 5%? Less than l%?
2.7.2 Drastically reduce the number of samples per cycle in the previous exercise,
but generate intermediate samples using quadratic interpolation. How many
true samples per cycle are required for the predictions to be reasonably ac-
curate?
2.7.3 The sampling theorem gives a more accurate method of interpolation than
the linear or quadratic interpolation of the previous exercises. However, even
this method breaks down at some point. At what number of samples per
cycle can no method of interpolation work?
2.8 Sampling
We will generally sample the analog signal at a uniform rate, corresponding
to a sampling frequency fS. This means that we select a signal value every
t, = i seconds. How does t, influence the resulting digital signal? The main
effects can be observed in Figures 2.14-2.17.
50 SIGNALS
Figure 2.14: Conversion of an analog signal into the corresponding digital one with a
lower sampling rate. As in the previous figure, the original analog signal has been overlaid
with the sampled time and digitized signal value grid. However, the time interval between
samples t, is longer.
Figure 2.15: Conversion of an analog signal into the corresponding digital one with yet
a lower sampling rate. Once again the original analog signal has been overlaid with the
sampled time and digitized signal value grid. Although there are only four samples per
cycle, the original signal is still somewhat recognizable.
2.8. SAMPLING 51
Figure 2.16: Conversion of an analog signal into a digital one at the minimal sampling
rate. Once again the original analog signal has been overlaid with the sampled time and
digitized signal value grid. Although there are only two samples per cycle, the frequency
of the original sine wave is still retrievable.
Figure 2.17: Conversion of an analog signal into a digital one at too low a sampling
rate. Once again the original analog signal has been overlaid with the sampled time and
digitized signal value grid. With only one sample per cycle, all information is lost.
52 SIGNALS
In Figures 2.14 and 2.15 the sampling rate is eight and four samples per
cycle respectively, which is high enough for the detailed shape of the signal to
be clearly seen (it is a simple sinusoid). At these sampling rates even simple
linear interpolation (connecting the sample points with straight lines) is not
a bad approximation, although peaks will usually be somewhat truncated.
In Figure 2.16, with only two samples per cycle, we can no longer make out
the detailed form of the signal, but the basic frequency is discernible. With
only a single sample per cycle, as in Figure 2.17, even this basic frequency
is lost and the signal masquerades as DC.
Have you ever watched the wagon wheels in an old western? When the
wagon starts to move the wheels start turning as they should; but then at
some speed they anomalously seem to stand still and then start to spin
backwards! Then when the coach is going faster yet they straighten out for
a while. What is happening? Each second of the moving picture is composed
of some number (say 25) still pictures, called frames, played in rapid succes-
sion. When the wheel is rotating slowly we can follow one spoke advancing
smoothly around the axle, from frame to frame. But when the wheel is ro-
tating somewhat faster the spoke advances so far between one frame and the
next that it seems to be the next spoke, only somewhat behind. This gives
the impression of retrograde rotation. When the wheel rotates exactly the
speed for one spoke to move to the next spokes position, the wheel appears
to stand still.
This phenomenon, whereby sampling causes one frequency to look like
a different one, is called aliasing. The sampled pictures are consistent with
different interpretations of the continuous world, the real one now going
under the alias of the apparent one. Hence in this case the sampling caused
a loss of information, irreversibly distorting the signal. This is a general
phenomenon. Sampling causes many analog signals to be mapped into the
same digital signal. This is because the digitized signal only records the
values of the continuous signal at particular times t = nt,; all analog signals
that agree at these points in time, but differ in between them, are aliased
together to the same digital signal.
Since sampling always maps many analog signals into the same digital
signal, the question arises-are there conditions under which A/D does not
cause irreparable damage? That is, is there any way to guarantee that we
will be able to recover the value of the analog signal at all times based
on the sampled signal alone? We expect the answer to be negative. Surely
the analog signal can take on arbitrary values at times not corresponding
to sample periods, and therefore many different analog signals correspond
to the same digital one. An affirmative answer would imply a one-to-one
2.8. SAMPLING 53
Figure 2.18: Aliasing of high-frequency analog signal into lower-frequency one. The high-
frequency signal has only a sample every one and a half cycles, i.e., it corresponds to a
digital frequency of $. The lower-frequency sinusoid is sampled at four samples per cycle,
i.e., cp = f.
2.8. SAMPLING 55
and the sampling theorem tells us that we must have cp < i. Consistently
using this digital frequency frees us from having to think about real (analog)
frequencies and aliasing. All the DSP will be exactly the same if a 2 Hz signal
is sampled at 10 Hz or a 2 MHz signal is sampled at 10 MHz.
Before continuing we should mention that the sampling theorem we have
been discussing is not the final word on this subject. Technically it is only
the low-pass sampling theorem for uniform time intervals. If the signals of
interest have small bandwidth but are centered on some high frequency, it is
certainly sufficient to sample at over twice the highest frequency component,
but only necessary to sample at about twice the bandwidth. This is the con-
tent of the band-pass sampling theorem. It is also feasible in some instances
to sample nonuniformly in time, for example, at times 0, $, 2,2$, 4, . . . . For
such cases there are nonuniform sampling theorems.
Now that we understand the first half of the sampling theorem, we are
ready to study the reconstruction formula in the second half. We can rewrite
equation (2.27) as
ccl
s(t) = c s,h(t - nt,) (2.28)
7X=--o;)
Figure 2.19: The reconstruction formula depicted graphically. In (A) we see an analog
signal and the samples digitized slightly higher than twice the highest frequency compo-
nent. (B) shows the sine kernels weighted by the sample value placed at each sample time;
note that at sample times all other sines contribute zero. In (C) we sum the contributions
from all kernels in the area and reconstruct the original analog signal.
EXERCISES
2.8.1 The wagon wheel introduced in the text demonstrates the principle of aliasing
in a popular context. What is the observedfrequency as a function of intended
frequency.
2.8.2 Redraw Figures 2.13-2.17 with sample times at different phasesof the sinu-
soid. Is a sine wave sampled at exactly twice per cycle (as in Figure 2.16)
always recoverable?
2.8.3 Redraw Figures 2.13-2.17 with a noninteger number of samples per cycle.
What new effects are observed?Are there any advantagesto such sampling?
Doesnt this contradict the sampling theorem?
2.8.4 Plot an analog signal composed of several sinusoids at ten times the Nyquist
frequency (vastly oversampled). Overlay this plot with the plots obtained for
slightly above and slightly below Nyquist. What do you observe?
2.8.5 Write a program for sampling rate conversion based on the algorithm for
reconstruction of the analog signal at arbitrary times.
2.9 Digitization
Now we return to the issue of signal value quantization. For this problem,
unfortunately, there is no panacea; there is no critical number of bits above
which no information is lost. The more bits we allocate per sample the less
noise we add to the signal. Decreasing the number of bits monotonically
reduces the SNR.
Even more critical is the matching of the spacing of the quantization
levels to the signals dynamic range. Were the spacing set such that the signal
resided entirely between two levels, the signal would effectively disappear
upon digitizing. Assuming there are only a finite number of quantization
levels, were the signal to vary over a much larger range than that occupied
by the quantization levels, once again the digital representation would be
close to meaningless. For the time being we will assume that the digitizer
range is set to match the dynamic range of the signal (in practice the signal
is usually amplified to match the range of the digitizer).
For the sake of our discussion we further assume that the analog signal
is linearly digitized, corresponding to b bits. This means that we select the
signal level 1 = - (2b-1 - 1) . . . + 2-l that is closest to s(tn). How does b
influence the resulting digital signal? The main effects can be observed in
Figures 2.20-2.24.
58 SIGNALS
Figure 2.20: Conversion of an analog signal into a corresponding digital one involves
quantizing both axes, sampling time and digitizing signal value. In the figure we see the
original analog signal overlaid with the sampled time and digitized signal value grid. The
resulting digital signal is depicted by the dots.
I IYI I I I I I/I I
I I I IKI I I I I l/l I
I I I I I I I lugI l l l I l l l l l l&p l l l l l l I
Figure 2.21: Conversion of an analog signal into the corresponding digital one with fewer
digitizing levels. As in the previous figure the original analog signal has been overlaid with
the sampled time and digitized signal value grid. However, here only 17 levels (about four
bits) are used to represent the signal.
2.9. DIGITIZATION 59
Figure 2.22: Conversion of an analog signal into the corresponding digital one with fewer
digitizing levels. Once again the original analog signal has been overlaid with the sampled
time and digitized signal value grid. Here only nine levels (a little more than three bits)
are used to represent the signal.
Figure 2.23: Conversion of an analog signal into the corresponding digital one with fewer
digitizing levels. Once again the original analog signal has been overlaid with the sampled
time and digitized signal value grid. Here only five levels (about two bits) are used to
represent the signal.
60 SIGNALS
i i
L c
Figure 2.24: Cc313rersion of an anal0 sig al int ;o the corresponding digital one with
. . _
the minimum number of digitizing levels. Once again the original analog signal has been
overlaid with the sampled time and digitized signal value grid. Here only three levels (one
and a half bits) are used to represent the signal.
Reflect upon the discrete time signal before signal value quantization
(the pulses before coding). This sequence of real numbers can be viewed as
the sum of two parts
an = & + vn where dn z Round(a,)
and so d, are integers and Ivn 1< 12. Assuming an to be within the range of
our digitizer the result of coding is to replace a, with dn, thus introducing
an error u, (see Figure 2.25). Were we to immediately reconvert the digital
signal to an analog one with a D/A converter, we would obtain a signal
similar to the original one, but with this noise added to the signal.
The proper way of quantifying the amount of quantization noise is to
compare the signal energy with the noise energy and compute the SNR from
equation (2.13). For a given analog signal strength, as the quantization levels
become closer together, the relative amount of noise decreases. Alternatively,
from a digital point of view, the quantization noise is always a constant
414 levels, while increasing the number of bits in the digital representation
increases the digital signal value. Since each new bit doubles the number of
levels and hence the digital signal value
SNR(dB) M 10 ( logIo(2b)2 - loglo 12) = 20blogro 2 M 6b (2.29)
that is, each bit contributes about 6 dB to the SNR. The exact relation will
be derived in exercise 2.9.2.
2.9. DIGITIZATION 61
Figure 2.25: Noise created by digitizing an analog signal. In (A) we see the output of a
digitizer as a function of its input. In (B) the noise is the rounding error, i.e., the output
minus the input.
quantized values, essentially rounding each input to the closest output level.
Hence the output as a function of the input will produce a graph that looks
like a staircase, as in Figure 2.25.A. Accordingly the rounding error, the
output minus the input, will look like a sawtooth, as in Figure 2.25.B. Thus
the quantization noise is predictable and strongly correlated with the sig-
nal, not random and uncorrelated as we tacitly assumed. This result seems
contradictory-if the noise signal is predictable, then it isnt noise at all.
Were the error to be truly predictable, then one could always compensate
for it, and digitizing would not harm the signal at all. The resolution of this
paradox is simple. The noise signal is indeed correlated to the analog signal,
but independent of the digitized signal. After digitizing the analog signal is
unavailable, and the noise becomes, in general, unpredictable.
EXERCISES
2.9.1 Dither noise is an analog noise signal that can be added to the analog signal
before digitizing in order to lessen perceived artifacts of round-off error. The
dither must be strong enough to effectively eliminate spurious square wave
signals, but weak enough not to overly damage the SNR. How much dither
should be used? When is dither needed?
2.9.2 Refine equation (2.29) and derive SNR = (2 logre 2b+1.8)dB by exploiting the
statistical uniformity of the error, and the definition of standard deviation.
2.9.3 Plot the round-off error as a function of time for sinusoids of amplitude 15,
and frequencies 1000, 2000, 3000, 1100, 1300, 2225, and 3141.5 Hz, when
sampled at 8000 samples per second and digitized to integer levels (-15, -14,
7 0 9 .7 14, 15). Does the error look noise-like?
and recreate the proper behavior at all times. In this section we will briefly
discuss these filters.
Assume that the highest frequency of importance in the signal to be
sampled is fmaZ. Strictly speaking the sampling theorem allows us to sample
at any frequency above the Nyquist frequency fiv = 2f,,,, but in practice
we can only sample this way if there is absolutely nothing above fmaz .
If there are components of the signal (albeit unimportant ones) or other
signals, or even just background noise, these will fold back onto the desired
signal after sampling unless removed by the antialiasing filter. Only an ideal
antialiasing filter, one that passes perfectly all signals of frequency less than
f max and blocks completely all frequencies greater than fmaz, would be able
to completely remove the undesired signals; and unfortunately, as we shall
learn in Section 7.1, such an ideal filter cannot be built in practice.
Realizable antialiasing filters pass low frequencies, start attenuating at
some frequency fr, and attenuate more and more strongly for higher and
higher frequencies, until they effectively block all frequencies above some
f2. We must be sure that the spectral areas of interest are below fr since
above that they will become attenuated and distorted; however, we cant
use 2fr as our sampling frequency since aliasing will occur. Thus in order to
utilize realizable filters we must sample at a frequency 2f2, higher than the
sampling theorem strictly requires. Typically sampling frequencies between
20% and 100% higher (1.2fN 5 fs 2 2fN) are used. The extra spectral
real-estate included in the range below $ is called a guard band.
The D/A reconstruction filters purpose is slightly less obvious than that
of the antialiasing filter. The output of the D/A must jump to the required
digital value at the sampling time, but what should it do until the next
sampling time? Since we have no information about what the analog signal
does, the easiest thing to do is to stay constant until the next sampling time.
Doing this we obtain a piecewise constant or boxcar signal that doesnt ap-
proximate the original analog signal very well. Alternatively, we might wish
to linearly interpolate between sampling points, but there are two difficul-
ties with this tactic. First, the linear interpolation, although perhaps better
looking than the boxcar, is not the proper type of interpolation from the
signal processing point of view. Second, and more importantly, interpolation
of any kind is noncausal, that is, requires us to know the next sample value
before its time. This is impossible to implement in real-time hardware. What
we can do is create the boxcar signal, and then filter it with an analog filter
to smooth the sharp transitions and eliminate unwanted frequencies.
The antialiasing and reconstruction filters may be external circuits that
the designer must supply, or may be integral to the A/D and D/A devices
64 SIGNALS
EXERCISES
2.10.1 Simulate aliasing by adding sinusoidswith frequenciesabove Nyquist to prop-
erly sampled sinusoidal signals. (You can perform this experiment using ana-
log signals or entirely on the computer.) Make the aliases much weaker than
the desired signals. Plot the resulting signals.
2.10.2 If you have a DSP board with A/D and D/A determine how the filters are
implemented. Are there filters at all or are you supposed to supply them
externally? Perhaps you have a sigma-delta converter that effectively has
the filter built into the A/D. Is there a single compromise filter, several filters,
or a programmable filter? Can you control the filters using software? Measure
the antialiasing filters response by injecting a series of sine waves of equal
amplitude and increasing frequency.
2.10.3 What does speech sound like when the antialiasing filter is turned off? What
about music?
tu
camp
cf
THERMO-
METER
-F--i TO
BINARY
first passes the sample and hold voltage. Stepping through 2b levels can be
a complex and time-intensive job, and can be replaced by a continuously
increasing ramp. The counter is replaced by a mechanism that measures the
time until the comparator triggers. A sawtooth waveform is usually utilized
in order to quickly return to the starting point. This class of A/D converters
is called a counting converter or a slope converter.
High-precision counting converters are by their very nature extremely
slow. Successive-upproxirnation converters are faster for the same reason
that half-flash are faster than flash converters. The principle is to start with
a steep slope, thus quickly determining a rough approximation to the input
voltage. Once the reference passes the input it is reduced one level and
further increased at a slower rate. This process continues until the desired
number of bits has been obtained.
Now that we understand some of the principles behind the operation
of real-world A/D devices, we can discuss their performance specifications.
Obviously the device chosen must be able to operate at the required sam-
pling rate, with as many bits of accuracy as further processing requires.
However, bits are not always bits. Imagine a less-than-ethical hardware en-
gineer, whose design fails to implement the require number of bits. This
engineer could simply add a few more pins to his A/D chip, not connecting
them to anything in particular, and claim that they are the least significant
bits of the converter. Of course they turn out to be totally uncorrelated to
the input signal, but that may be claimed to be a sign of noise. Conversely, if
a noisy input amplifier reduces the SNR below that given by equation (2.29)
we can eliminate LSBs without losing any signal-related information. A/D
specifications often talk about the number of efective bits as distinct from
the number of output bits. Effective bits are bits that one can trust, the num-
ber of bits that are truly input-signal correlated. We can find this number
by reversing the use of equation (2.29).
The number of effective bits will usually decrease with the frequency
of the input signal. Lets understand why this is the case. Recall that the
A/D must actually observe the signal over some finite interval, known as the
aperture time, in order to determine its value. For a low-frequency signal this
is not problematic since the signal is essentially constant during this entire
time. However, the higher the frequency the more the signal will change
during this interval, giving rise to aperture uncertainty. Consider a pure sine
wave near where it crosses the axis. The sine wave is approximately linear
in this vicinity, and its slope (derivative) is proportional to the frequency.
68 SIGNALS
fTqxrture 5 rb (2.30)
EXERCISES
2.11.1 Design a thermometer to binary converter circuit for an eight level digitizer
(one with eight inputs and three outputs). You may only use logical gates,
devices that perform the logical NOT, AND, OR, and XOR of their inputs.
2.11.2 A useful diagnostic tool for testing A/D circuits is the level histogram. One
inputs a known signal that optimally occupies the input range and counts
the number of times each level is attained. What level histogram is expected
for a white noise signal? What about a sinusoid? Write a program and find
the histograms for various sounds (e.g., speech, musical instruments).
2.11.3 An A/D is said to have bad transitions when certain levels hog more of the
input range than they should. An A/D is said to have a stuck bit when
an output bit is constant, not dependent on the input signal. Discuss using
sawtooth and sinusoidal inputs to test for these malfunctions.
70 BIBLIOGRAPHICAL NOTES
2.11.4 A signal that is too weak to be digitized can sometimes be captured using
a technique known as dithering whereby a small amount of random noise is
added before digitizing. Explain and demonstrate how dithering works.
2.11.5 Delta encoding is often allocated fewer bits than actually needed. In this
cases we must round the differences to the nearest available level. Assuming
uniform spacing of quantization levels, how much noise is introduced as a
function of the number of bits. Write a program to simulate this case and try
it on a speech signal. It is often the case that smaller differences are more
probable than larger ones. How can we exploit this to reduce the quantization
error?
2.11.6 Fixed step size delta modulation encodes only the sign of the difference
between successive signal values, d, = sgn(s, - s+r), but can afford to
oversample by b, the number of bits in the original digitized signal. Recon-
struction of the signal involves adding or subtracting a fixed 6, according to
n
sn = s^,-1 + d,6. What problems arise when S is too small or too large?
Invent a method for fixes these problems and implement it.
2.11.7 Prove equation (2.30).
Bibliographical Notes
The material in this chapter is treated to some extent in all of the elementary books
on DSP. Probably the first book devoted to DSP was the 1969 text by Gold and
Rader [79]. The author, like many others, first learned DSP from the classical text
by Oppenheim and Schafer [185] that has been updated and reissued as (1861. A
more introductory text co-authored by Oppenheim is [187]. Another comprehensive
textbook with a similar engineering approach is by Proakis and Manolakis [200].
A very comprehensive but condensed source for almost everything related to DSP
is the handbook edited by Mitra and Kaiser (2411.
More accessible to non-engineers, but at a much more elementary level and
covering much less ground is the book by Marven and Ewers [159]. Steiglitz has
written a short but informative introductory book [252]. Finally, Mclellan, Schafer
and Yoder have compiled a course for first year engineering students that includes
demos and labs on a CD [167].
3
Signals dwell both in the time and frequency domains; we can equally ac-
curately think of them as values changing in time (time domain), or as
blendings of fundamental frequencies (spectral domain). The method for de-
termining these fundamental frequencies from the time variations is called
Fourier or spectral analysis. Similar techniques allow returning to the time
domain representation from the frequency domain description.
It is hard to believe that 300 years ago the very idea of spectrum didnt
even exist, that less than 200 years ago the basic mechanism for its cal-
culation was still controversial, and that as recently as 1965 the algorithm
that made its digital computation practical almost went unpublished due to
lack of interest. Fourier analysis is used so widely today that even passing
mention of its most important applications is a lengthy endeavor. Fourier
analysis is used in quantum physics to uncover the structure of matter on
the smallest of scales, and in cosmology to study the universe as a whole.
Spectroscopy and X-ray crystallography rely on Fourier analysis to analyze
the chemical composition and physical structure from minute quantities of
materials, and spectral analysis of light from stars tells us of the composition
and temperature of bodies separated from us by light years. Engineers rou-
tinely compute Fourier transforms in the analysis of mechanical vibrations,
in the acoustical design of concert halls, and in the building of aircraft and
bridges. In medicine Fourier techniques are called upon to reconstruct body
organs from CAT scans and MRI, to detect heart malfunctions and sleep dis-
orders. Watson and Crick discovered the double-helix nature of DNA from
data obtained using Fourier analysis. Fourier techniques can help us differ-
entiate musical instruments made by masters from inferior copies, can assist
in bringing back to life deteriorated audio recordings of great vocalists, and
can help in verifying a speakers true identity.
In this chapter we focus on the concepts of spectrum and frequency,
but only for periodic signals where they are easiest to grasp. We feel that
several brief historical accounts will assist in placing the basic ideas in proper
71
72 THE SPECTRUM OF PERIODIC SIGNALS
context. We derive the Fourier series (FS) of a periodic signal, find the FS for
various signals, and see how it can be utilized in radar signal processing. We
briefly discuss its convergence and properties, as well as its major drawback,
the Gibbs phenomenon. We also introduce a new notation that uses complex
numbers and negative frequencies, in order to set the stage for the use of
Fourier techniques in the analysis of nonperiodic signals in the next chapter.
EXERCISES
3.1.1 Each of the colors of the rainbow is characterized by a single frequency,
while artists and computer screens combine three basic colors. Reconcile the
one-dimensional physical concept of frequency with the three-dimensional
psychological concept of color.
3.1.2 Wavepackets are particle-like waves, that is, waves that are localized in space.
For example, you can create a wavepacket by multiplying a sine wave by a
Gaussian
m =e * 2~ sin(&)
where p is the approximate location. Plot the signal in space for a given time,
and in time for a given location. What is the uncertainty in the location of
the particle? If one wishes the particle to travel at a speed V, one can
substitute 1-1= vt. What happens to the space plot now? How accurately can
the velocity be measured?
74 THE SPECTRUM OF PERIODIC SIGNALS
From all of these facts together we can conclude that a signal that results
from weighted summing of sinusoidal signals with frequencies n f, and possi-
bly addition of a constant signal, is itself periodic with period P = -& Such a
trigonometric series is no longer sinusoidal, indeed it can look like just about
anything, but it is periodic. You can think of the spectrum as a recipe for
preparing an arbitrary signal; the frequencies needed are the ingredients,
and the weights indicate how much of each ingredient is required.
The wealth of waveforms that can be created in this fashion can be
demonstrated with a few examples. In Figure 3.1 we start with a simple sine,
and progressively add harmonics, each with decreased amplitude (the sine
of frequency rCf having amplitude i). On the left side we see the harmonics
themselves, while the partial sums of all harmonics up to that point appear
on the right. It would seem that the sum tends to a periodic sawtooth signal,
cK sin@&)
k=O
k
K-*~
+ -I(t) (3.1)
Figure 3.1: Building up a periodic sawtooth signal -T(t) from a sine and its harmonics.
In (A) are the component sinusoids, and in (B) the composite signal.
Figure 3.2: Building up a periodic square wave signal from a sine and its odd harmonics.
In (A) are the component sinusoids, and in (B) the composite signal.
76 THE SPECTRUM OF PERIODIC SIGNALS
and this feeling is strengthened when the summation is carried out to higher
harmonics. Surprisingly, when we repeat this feat with odd harmonics only
we get a square wave
K 1
---sin
ck (p+l ((2k + 1)wt) - q (t> (3.2)
=
as can be seen in Figure 3.2.
The signal f(t) = sin(wt) is an odd function of t, that is f(-t) = -f(t).
Since the sum of odd functions is odd, all signals generated by summing only
harmonically related sines will be odd as well. If our problem requires an
even function, one for which f(4) = f(t), we could sum cosines in a similar
way. In order to produce a signal that is neither odd nor even, we need to
sum harmonically related sines and cosines, which from here on we shall call
Harmonically Related Sinusoids (HRSs). In this way we can produce a huge
array of general periodic signals, since any combination of sines and cosines
with frequencies all multiples of some basic frequency will be periodic with
that frequency.
In fact, just about anything, as long as it is periodic, can be represented
as a trigon .ometric series involving harmonically related sinusoids. Just about
anything, as long as it is periodic, can be broken down into the weighted
sum of sinusoidal signals with frequencies nf, and possibly a constant sig-
nal. When first discovered, this statement surprised even the greatest of
mathematicians.
EXERCISES
3.2.1 In the text we considered summing all harmonics and all odd harmonics with
amplitude decreasing as i. Why didnt we consider all even harmonics?
3.2.2 When two sinusoids with close frequencies are added beats with two observ-
able frequencies result. Explain this in terms of the arguments of this section.
3.2.3 To what waveforms do the following converge?
1. ;- ip$E.l+~+~~~+...)
2. p-p?g.l+~+~+...)
. . .
3. .$ + ipin - ii(E$E.l + Ei$l+ SCZ.#l +. . .)
4. i- $(?y-~~+?~.+...,
3.3. FOURIERS DISCOVERY 77
Figure 3.3: The instantaneous deflection of a vibrating string may be sinusoidal, and
the boundary conditions restrict the possible frequencies of these sines. The top string
contains only half of its wavelength between the strings supports; the next contains a full
wavelength, the third three-quarters, etc.
78 THE SPECTRUM OF PERIODIC SIGNALS
question is whether this is the most general pattern of deflection. In the eigh-
teenth and nineteenth century there were good reasons for suspecting the
answer to be negative. Not having the benefit of the computer-generated
plots of sums of HRSs presented in the previous section, even such great
mathematicians as Lagrange believed that all such sums would yield smooth
curves. However, it was easy to deform the string such that its shape would
be noncontinuous (e.g., by pulling it up at its middle point forcing a trian-
gular shape). What would happen the moment such a plucked string was
released? Since the initial state was supposedly not representable in terms
of the basic sinusoidal modes, there must be other, nonsinusoidal, solutions.
This was considered to be a fatal blow to the utility of the theory of trigono-
metric series. It caused all of the mathematicians of the day to lose interest
in them; all except Jean Baptiste Joseph Fourier. In his honor we are more
apt today to say Fourier series than trigonometric series.
Although mathematics was Fouriers true interest, his training was for
the military and clergy. He was sorely vexed upon reaching his twenty-first
birthday without attaining the stature of Newton, but his aspirations had to
wait for some time due to his involvement in the French revolution. Fourier
(foolishly) openly criticized corrupt practices of officials of Robespierres gov-
ernment, an act that led to his arrest and incarceration. He would have gone
to the guillotine were it not for Robespierre himself having met that fate
first. Fourier returned to mathematics for a time, studying at the Ecole Nor-
mal in Paris under the greatest mathematicians of the era, Lagrange and
Laplace. After that school closed, he began teaching mathematics at the
Ecole Polytechnique, and later succeeded Lagrange to the chair of mathe-
matical analysis. He was considered a gifted lecturer, but as yet had made
no outstanding contributions to science or mathematics.
Fourier then once again left his dreams of mathematics in order to join
Napoleons army in its invasion of Egypt. After Napoleons loss to Nel-
son in the Battle of the Nile, the French troops were trapped in Egypt,
and Fouriers responsibilities in the French administration in Cairo included
founding of the Institut dEgypte (of which he was secretary and member of
the mat hemat its division), the overseeing of archaeological explorations, and
the cataloging of their finds. When he finally returned to France, he resumed
his post as Professor of Analysis at the Ecole Polytechnique, but Napoleon,
recalling his administrative abilities, snatched him once again from the uni-
versity, sending him to Grenoble as Prefect. Although Fourier was a most
active Prefect, directing a number of major public works, he neglected nei-
ther his Egyptological writing nor his scientific research. His contributions
to Egyptology won him election to the French Academy and to the Royal
3.3. FOURIERS DISCOVERY 79
Society in London. His most significant mathematical work is also from this
period. This scientific research eventually led to his being named perpetual
secretary of the Paris Academy of Sciences.
Fourier was very interested in the problem of heat propagation in solids,
and in his studies derived the partial differential equation
dV
-- -
d2V
K
at dX2
now commonly known as the difjhion equation. The solution to such an
equation is, in general, difficult, but Fourier noticed that there were solutions
of the form f(t)g(x), where f(t) were decreasing exponentials and g(x) were
either sin(nx) or cos(nx). Fourier claimed that the most general g(x) would
therefore be a linear combination of such sinusoids
the expansion known today as the Fourier series. This expansion is more
general than that of Euler, allowing both sines and cosines to appear simul-
taneously. Basically Fourier was claiming that arbitrary functions could be
written as weighted sums of the sinusoids sin(nx) and cos(nx), a result we
now call Fouriers theorem.
Fourier presented his theorem to the Paris Institute in 1807, but his
old mentors Lagrange and Laplace criticized it and blocked its publication.
Lagrange once again brought up his old arguments based on the inability
of producing nonsmooth curves by trigonometric series. Fourier eventually
had to write an entire book to answer the criticisms, and only this work
was ever published. However, even this book fell short of complete rigorous
refutation of Lagranges claims. The full proof of validity of Fouriers ideas
was only established later by the works of mathematicians such as Dirichlet,
Riemann, and Lebesgue. Today we know that all functions that obey certain
conditions (known as the Dirichlet conditions), even if they have discontin-
uous derivatives or even if they are themselves discontinuous, have Fourier
expansions.
EXERCISES
3.3.1 Consider functions f(t) defined on the interval -1 5 t 5 1 that are defined
by finite weighted sums of the form XI, fit cos(dt), where k is an integer.
What do all these functions have in common? What about weighted sums of
sin@t)?
80 THE SPECTRUM OF PERIODIC SIGNALS
3.3.2 Show that any function f(t) defined on the interval -1 5 t < 1 can be written
as the sum of an even function fe(t) (fe(--t) = fe(-t)) and an odd function
(fo(-t) = -fo(-t)).
3.3.3 Assume that all even functions can be representedas weighted sums of cosines
as in the first exercise, and that all odd functions can be similarly rep-
resented as weighted sums of sines. Explain how Fourier came to propose
equation (3.3).
3.3.4 How significant is the difference between a parabola and half a period of
a sinusoid? To find out, approximate z(t) = cos(t) for -4 5 t 5 4 by
y(t) = at2 + bt + c. Find the coefficients by requiring y(-t) = y(t), y(0) = 1
and y(f ;) = 0. Plot the cosine and its approximation. What is the maximal
error? The cosine has slope 1 at the ends of the interval; what is the slope
of the approximation? In order to match the slope at t = &4 as well, we
need more degreesof freedom, so we can try y(t) = at4 + bt2 + c. Find the
coefficients and the maximum error.
The desired proof of a similar claim for HRSs can rest on our showing
that any shifted analog impulse in the required time interval can be built
up from such sinusoids. Due to the HRSs periodicity in T, the shifted im-
pulse will automatically be replicated in time to become a periodic impulse
train. Consequently the following algorithm finds the HRS expansion of any
function of period T.
We are almost done. We have just shown that the HRSs span the vector
space of periodic analog signals. In order for this set to be a basis the ex-
pansions must be unique. The usual method of proving uniqueness involves
showing that there are no extraneous signals in the set, i.e., by showing that
the HRSs are linearly independent. Here, however, there is a short-cut; we
can show that the HRSs comprise an orthonormal set, and we know from
Appendix A.14 that all orthonormal sets are linearly independent.
In Section 2.5 the dot product was shown to be a valid scalar multipli-
cation operation for the vector space of analog signals. For periodic analog
signals we neednt integrate over all times, rather the product given by
T
r =x-y means r = x(t) I@> dt
s0
(where the integration can actually be performed over any whole period)
should be as good. Actually it is strictly better since the product over all
times of finite-valued periodic signals may be infinite, while the present
product always finite. Now it will be useful to try out the dot product on
sinusoids.
We will need to know only a few definite integrals, all of which are
derivable from equation A.34. First, the integral of any sinusoid over any
number of whole periods gives zero
iTsin ($t) dt = g W)
which can be derived by realizing that symmetry dictates
and so
by identity (A.20). Somewhat harder to guess is the fact that the integral
of the product of different harmonics is always zero, i.e.
forms an orthonormal set of signals. Since we have proven that any signal
of period T can be expanded in these signals, they are an orthonormal set
of signals that span the space of periodic signals, and so an orthonormal
basis. The {vk} are precisely the HRSs to within unimportant multiplica-
tive constants, and hence the HRSs are an orthogonal basis of the periodic
signals. The Fourier series takes on a new meaning. It is the expansion of
an arbitrary periodic signal in terms of the orthogonal basis of HRSs.
We now return to our second task-given a periodic signal s(t), we now
know there is an expansion:
How do we find the expansion coefficients ck? This task is simple due to
the basis {2/k) being orthonormal. From equation A.85 we know that for an
orthonormal basis we need only to project the given signal onto each basis
signal (using the dot product we defined above).
i&=sv=
s
0
T
s(t) vk(t) dt
84 THE SPECTRUM OF PERIODIC SIGNALS
This will give us the coefficients for the normalized basis. To return to the
usual HRSs
(3 .9)
is not difficult.
ak = $iTs(t)sin(Ft) dt
1 T
b(-J = - s(t) dt (3.10)
J
bl, = i j&COS(~t) dt
This result is most fortunate; were the sinusoids not orthogonal, finding
the appropriate coefficients would require solving normal equations (see
Appendix A.14). When there are a finite number N of basis functions, this
is a set of N equations in N variables; if the basis is infinite we are not even
able to write down the equations!
These expressions for the FS coefficients might seem a bit abstract, so
lets see how they really work. First lets start with a simple sinusoid s(t) =
A sin(&) + B. The basic period is T = c and so the expansion can contain
only sines and cosines with periods that divide this T. The DC term is, using
equations (3.6) and (3.7),
1 T
bo = -
T o J s(t) dt = iiT (Asin +B) dt= $BT=l3
as expected, while from equations (3.8) all other terms are zero except for
one.
al = GiTs(t)sin (Ft) dt
= $iT(Asin($t)+B)sin($t)dt=$Ag=A
This result doesnt surprise us since the expansion of one of basis signals
must be exactly that signal!
3.4. REPRESENTATION BY FOURIER SERIES 85
Slightly more interesting is the case of the square wave q (t/T). There
will be no DC term nor any cosine terms, as can be seen by direct symmetry,
To show this mathematically we can exploit a fact we have previously men-
tioned, that the domain of integration can be over any whole period. In this
case it is advantageous to use the interval from -T/2 to T/2. Since q (t/T)
is an odd function, i.e., 0(-t/T) = - q (t/T), the contribution from the left
half interval exactly cancels out the contribution of the right half interval.
This is a manifestation of a general principle; odd functions have only sine
terms, while even functions have only DC and cosine term contributions.
The main contribution for q (t/T) will be from the sine of period T, with
coefficient
al = $- LTs(t)sin ($t) dt
cannot contribute because of the odd problem once again. Therefore only
odd harmonic sinusoids can appear, and for them
uk = $ /)(t)sin (Ft) dt
= ~~~sin(~~)~~-~~~sin(~~)~~
= 2$S,Tsin($5+~= -$
EXERCISES
3.4.1 Our proof that the HRSs span the space of periodic signals required the HRSs
to be able to reproduce all SUIs, while Figure 3.4 reproduced only an impulse
centered at zero. Show how to generate arbitrary SUIs (use a trigonometric
sum formula).
86 THE SPECTRUM OF PERIODIC SIGNALS
3.4.2 Observe the sidelobes in Figure 3.4. What should the constant term bc be
for the sidelobes to oscillate around zero? In the figure each increase in the
number of cosines seems to add another half cycle of oscillation. Research
numerically the number and amplitude of these oscillations by plotting the
sums of larger numbers of cosines. Do they ever disappear?
3.4.3 Reproduce a graph similar to Figure 3.4 but using sines instead of cosines.
Explain the results (remember that sine is an odd function). Why isnt the
result simply a shifted version of cosine case?
3.4.4 Find the Fourier series coefficients for the following periodic signals. In order
to check your results plot the original signal and the partial sums.
1. Sum of two sines al sin(&) + iz2 sin(2Lctt)
2. Triangular wave
3. Fully rectified sine 1sin(z)1
4. Half wave rectified sine sin(z)u(sin(z))
3.4.5 We can consider the signal s(t) = Asin + B to be periodic with period
T = kW What is the expansion now? Is there really a difference?
3.4.6 For the two-dimensional plane consider the basis made up of unit vectors
along the x axis Al = (1,O) and along the 45 diagonal A = (-&, -$). The
unit vector of slope $ is Y = (5, -&). Find the coefficients of the expansion
Y = cqAl + cx2A2 by projecting Y on both Al and A2 and solving the
resulting equations.
3.4.7 Find explicitly the normal equations for a set of basis signals Al,(t) and
estimate the computational complexity of solving these equations
n(t) didnt converge very well. In fact there was significant ringing, both-
ersome oscillations that wouldnt go away with increasing number of terms.
Unsure whether he had discovered a new mathematical phenomenon or sim-
ply a bug in his analyzer he turned to the eminent American theoretical
physicist of the time, Josiah Willard Gibbs. Gibbs realized that the problem
was caused by discontinuities. Dirichlet had shown that the Fourier series
converged to the midpoint at discontinuities, and that as long as there were
a finite number of such discontinuities the series would globally converge;
but no one had previously asked what happened near a discontinuity for a
finite number of terms. In 1899 Gibbs published in Nature his explanation
of what has become known as the Gibbs phenomenon.
In Section 3.3 we mentioned the Dirichlet conditions for convergence of
the Fourier series.
It can then be shown that for any signal s(t) the partial sums equal
Figure 3.5: Partial sums of the Fourier series of a periodic square wave signal O(t) for
K = 0, 1,2,3,5 and 7. Note that although far from the discontinuity the series converges
to the square wave, near it the overshoot remains.
(the integration being over one period of duration T) from which Dirichlets
convergence results emerge.
Now you may believe, as everyone did before Gibbs, that Dirichlets
theorem implies that amplitude of the oscillations around the true values
decreases as we increase the number of terms in the series. This is the case
except for the vicinity of a discontinuity, as can be seen in Figure 3.5. We
see that close to a discontinuity the partial sums always overshoot their
target, and that while the time from the discontinuity to the maximum
overshoot decreases with increasing K, the overshoot amplitude does not
decrease very much. This behavior does not contradict Dirichlets theorem
since although points close to jump discontinuities may initially be affected
by the overshoot, after enough terms have been summed the overshoot will
pass them and the error will decay.
For concreteness think of the square wave 0 (t). For positive times close
to the discontinuity at t = 0 equation (3.13) can be approximated by
Figure 3.6: Gibbs phenomenon for the discontinuity of the square wave at t = 0. Plotted
are the square wave, the partial sum with K = 3 terms, and the approximation using the
sine integral.
EXERCISES
3.5.1 Numerically integrate sine(t) and plot Sine(t). Show that it approaches &$
for large absolute values. Find the maximum amplitude. Where does it occur?
Verify that the asymptotic behavior of the amplitude is i.
3.5.2 The following exercises are for the mathematically inclined. Prove equa-
tion (3.12) by term-by-term multiplication of the sum in the definition of
OK by sin (3) and using trigonometric identity (A.32).
3.5.3 Prove equation (3.13) and show Dirichlets convergence results.
3.5.4 Prove the approximation (3.14).
3.5.5 Lanczos proposed suppressing the Gibbs phenomenon in the partial sum SK
by multiplying the kth Fourier coefficient (except the DC) by sine ($) . Try
this for the square wave. How much does it help? Why does it help?
3.5.6 We concentrated on the Gibbs phenomenon for the square wave. How do
we know that other periodic signals with discontinuities act similarly? (Hint:
Consider the Fourier series for s(t) + au(t) w h ere s(t) is a continuous signal
and a a constant.)
Signals which are neither odd nor even. truly require combinations of both
Since the FS in equation (3.9) includes for every frequency both a sine and
cosine function (which differ by 90 or a quarter cycle), it is said to be in
quadrature form.
The first signal space basis we studied, the SUI basis, required only one
functional form. Is there a single set of sinusoidal signals, all of the same type,
that forms a basis for the space of periodic signals? Well, for each frequency
component w the FS consists of the sum of two terms a cos(wt) + b sin@!).
Such a sum produces a pure sinusoid of the same frequency, but with some
phase offset d sin(wt + cp). In fact, it is easy to show that
as long as
dk = J&Gj @= taI+(bk, Q) (3.16)
where the arctangent is the full four-quadrant function, and
1
cos(wt) = f (&t + f+) sin(&) = 5 (eiwt - eeiwt) (3.19)
We can think of the exponents with positive eiwt and negative e-jut expo-
nents as a single type of exponential eiwt with positive and negative frequen-
cies w. Using only such complex exponentials, although of both positive and
negative frequencies, we can produce both the sine and cosine signals of the
quadrature representation, and accordingly represent any periodic signal.
s(t) = F ekej&&t
T
k=-co1
We could once again derive the expression for the coefficients ck from those
for the quadrature representation, but it is simple enough to derive them
from scratch. We need to know only a single integral.
1 jalrnt
eT e -ivt ,jt = 6,,,T (3.21)
J0
This shows that the complex exponentials are orthogonal with respect to
the dot product for complex signals
Sl S2 =J 0
T
Q(t) s;(q c&t (3.22)
and that
(3.23)
J s(t)
esiFt
&
1 T
Ck(t) = T (3.24)
0
with a minus sign appearing in the exponent. Thus Fouriers theorem can be
stated in a new form: All periodic functions (which obey certain conditions)
can be written as weighted sums of complex exponentials.
The complex exponential form of the FS is mathematically the simplest
possible. There is only one type of function, one kind of coefficient, and
there is strong symmetry between equations (3.20) and (3.24) that makes
them easier to remember. The price to pay has been the introduction of
3.6. COMPLEX FS AND NEGATIVE FREQUENCIES 93
which after using the orthogonality relation (3.21) leaves two terms.
EXERCISES
3.6.1 Plot sin(z) + sin(2a: + p) with cp = 0, 5, T, %. What can you say about the
effect of phase? Change the phases in the Fourier series for a square wave.
What signals can you make?
3.6.2 Derive all the relations between coefficients of the quadrature, amplitude and
phase, and complex exponential representations. In other words, show how
to obtain cLk and bl, from ck and vice versa; c&k and bk from dk and vice versa;
ck from dk and vice versa. In your proofs use only trigonometric identities
and equation (A.7).
94 THE SPECTRUM OF PERIODIC SIGNALS
l 2rk
is e-l- T ck. The coefficients magnitudes are unchanged, but the phases have
been linearly shifted. As we know from exercise 3.6.6 such phase shifts do
not change the power spectrum but still may be significant. We see here that
phase shifts that are linear in frequency correspond to time shifts.
When a transformation leaves a signal unchanged or changes it in some
simple way we call that transformation a symmetry. Time shift is one inter-
esting symmetry, and another is time reversal Rev s. Although the import
of the latter is less compelling than the former many physical operations are
unchanged by time reversal. It is not difficult to show that the effect of time
reversal is to reverse the FS to c-k.
The next property of importance was discovered by Parseval and tells
us how the energy can be recovered from the FS coefficients.
- T1 oT ls@)12dt = 2
E -- lck12 (3.25)
s k=-co
What does Parsevals relation mean? The left hand side is the power com-
puted over a single period of the periodic signal. The power of the sum of two
signals equals the sum of the powers if and only if the signals are orthogonal.
1
-IT lx(t) + y(t)12dt = 1T LT (x(t) + !#I)* (x(t) + y(t)) dt
T 0
- 1
T SOT bW12 + lYW12 + m (x*~t)Yw) dt
Since any two different sinusoids are uncorrelated, their powers add, and
this can be generalized to the sum of any number of sinusoids. So Parsevals
relation is another consequence of the fact that sinusoids are orthogonal.
For complex valued signals s(t) there is a relation between the FS of
the signal and that of its complex conjugate s*(t). The FS of the complex
conjugate is c:k. For real signals this implies a symmetry of ck (i.e., c-k =
cz), which means Ic-k I = lckl and %(c-k) = %(ck) but $(c-k) = -g(ck).
There are many more symmetries and relations that can be derived for
the FS, e.g., the relationship between the FS of a signal and those of its
derivative and integral. There is also an important rule for the FS of the
product of two signals, which the reader is not yet ready to digest.
EXERCISES
3.7.1 Show that adding to the argument of a sinusoid a phase that varies linearly
with time shifts its frequency by a constant. Relate this to the time shift
property of the FS.
96 THE SPECTRUM OF PERIODIC SIGNALS
3.7.2 Plot the sum of several sinusoids with various phases. Demonstrate that a
linear phase shift causes a time shift. Can you tell that all these signals have
the same power spectrum?
3.7.3 How does change of time scale s(CYt) affect ck? Prove that the effect of time
reversal is to reverse the FS.
3.7.4 Derive Parsevals relation for the FS.
3.7.5 Show that if a signal is symmetric(antisymmetric), i.e., if s(t + 5) = &s(t),
then its FS contains only even (odd) harmonics.
3.7.6 The FS of s is ck; what is the FS of its derivative? Its integral?
-? A
4 II II I
I
I
I.
I
1 b
-
2T - T 0 T 2T
Figure 3.7: The rectangular signal with amplitude A, period T, and duty cycle S = $.
a rectangular wave spends in the higher of its levels is called its duty cycle
6 = $!, and a rectangular wave with 6 = $ duty cycle is a square wave.
We also wish to make the amplitude and period explicit, and to have the
signal more symmetric in the time domain; we accordingly introduce A, T,
and d = ST, and require the signal to be high from -$ to $. Unlike the
square wave, a non-50% duty cycle rectangular signal will always have a DC
component. There is consequently no reason for keeping the levels symmetric
around zero, and we will use 0 and A rather than &A.
Thus we will study
1
Ifrac( +)I < $
s(t) = A 0 $ < Ifrac($)l < T - 4 (3.28)
1 1 T- 4 < Ifrac($)l < T
(where frac(z) is the fractional part of x) as depicted in Figure 3.7.
The period is T and therefore the angular frequencies in the Fourier series
will all be of the form wk = a/c
T . We can choose the interval of integration
in equation (3.24) as we desire, as long as it encompasses a complete period.
The most symmetric choice here is from -5 to 5, since the signal then
becomes simply
4)
- A 1 I<+>1
<$ (3.29)
i 0 else
and as a consequence
T
0 1 T
r3jc = - T s(t) Ciyt dt
T s -- 2
--
A d2 +)kt &
T
-1 -- d
2
98 THE SPECTRUM OF PERIODIC SIGNALS
sin( %$)
sk=A d = Asinc (y) = Asinc (T) = Asinc(nlc6) (3.30)
2
where we have recognized our old friend sine. The FS is dependent only on
the duty cycle, not directly on T. Of course this does not mean that the
Fourier series is not dependent on T! The coefficient Sk multiplies the term
containing wk = q, and consequently the distribution on the frequency
axis indeed changes. Taking into account this meaning of Sk we see that the
spectral envelope is influenced by the pulse width but not the period.
The main lobe of the sine function is between -7r and 7r, which here
means between Sic = -1 and Sk = 1. Hence most of the energy is between
WI, = y = fj+, or otherwise stated, the frequency spread is Aw = 3.
The minimum spacing between two points in time that represent the same
point on the periodic signal is obviously At = T. The relationship between
the time and frequency spreads can therefore be expressed as
4n
AwAt = T (3.31)
A
,,I 1.1.lllllil/1ll1 II ll.,ll
B
I .1A..,,..,, ) I,llll11l ...I, , ,, , * 1.I
C
,.Tr I .I l., 1lill 1.. ,.1.I..I..,, ,
lf I I
D
-1 I., ,_*1,,,1Ii1 I ,I I I.. , , ,.I
i - - - - - - _
Figure 3.8: The effect of changing the duty cycle at constant period. In these figures
we see on the left a periodic rectangular signal, and on the right the absolute squares of
its FS amplitudes represented as vertical bars placed at the appropriate frequencies. (A)
represents a duty cycle of 20%, (B) 40%, (C) 60? o and (D) 80%. Note that when the duty
cycle vanishes all amplitudes become equal, while when the signal becomes a constant,
only the DC term remains.
the time offset by two and multiplying by the speed of radar waves (the
speed of light c) we obtain the distance from radar to target. The round-trip
time should be kept lower than the PRI; and echo returning after precisely
the PRI is not received since the radar receiver is blanked during trans-
mission; if the round-trip time exceeds the PRI we get aliasing, just as
in sampling analog signals. Hence we generally strive to use long PRIs so
that the distance to even remote targets can be unambiguously determined.
More sophisticated radars vary the PRI from pulse to pulse in order to dis-
ambiguate the range while keeping the echo from returning precisely when
the next pulse is to be transmitted.
Due to the Doppler effect, the PRF of the reflection from target moving
at velocity w is shifted from its nominal value.
-l - - -
B
....... . . .....I..I....,..,,..,.,...
l..l..L~.,.~.~.I..I.
.,..
I..lllllllll IL1
C
I.....I...,... ....................I...........1....1....I.....I
..........I.....IllI
I
D
I......,............,...........~..~........~...........
1...........f...........1
Figure 3.9: The effect of changing the period at constant duty cycle. In these figures we
see on the left a periodic rectangular signal, and on the right the absolute squares of its
FS amplitudes represented as vertical bars placed at the appropriate frequencies. As we
progress from (A) through (D) the period is halved each time. Note that as the period is
decreased with constant pulse width the frequency resolution decreases but the underlying
sine is unchanged.
EXERCISES
3.8.1 Show how to regain the Fourier series of the square wave (equation (3.2)),
from (3.30) by taking a 50% duty cycle.
3.8.2 We assumed that A in equation (3.28) was constant, independent of T and
d. Alternative choices are also of interest. One could demand that the basic
rectangle be of unit area A = $, or of unit energy A = -&, or that the power
(energy per time) be unity A = X-
&. Explain the effect of the different choices
on the signal and its FS when 6 and 2 are varied.
3.8.3 Show that the FS of a train of impulses s(t) = c J(t - IcT) is a train of
impulses in the frequency domain. How does this relate to the calculations
of this section? To which choice of A does this correspond?
3.8.4 One technique that radar designers use to disambiguate longer ranges is PRI
staggering. Staggering involves alternating between several PRIs. How does
staggering help disambiguate? How should the PRIs be chosen to maximize
the range? (Hint: Use the Chinese remainder theorem.)
3.8.5 What is the FS of a rectangular wave with stagger two (i.e., alternation
between two periods Tl and Zz)?
Bibliographical Notes
For historical background to the development of the concept of frequency consult
[223]. Newtons account of the breaking up of white light into a spectrum of colors
can be read in his book Opticks [179]. For more information on the colorful life
of Fourier consult [83]. Incidentally, Marc Antoine Parseval was a royalist, who
had to flee France for a while to avoid arrest by Napoleon. Lord Rayleigh, in his
influential 1877 book on the theory of sound (started interestingly enough on a
vacation to Egypt where Fourier lived eighty years earlier), was perhaps the first
to call the trigonometric series by the name Fourier series. Gibbs presentation of
his phenomenon is [74].
There are many books devoted entirely to Fourier series and transforms. To get
more practice in the mechanics of Fourier analysis try [104]. In-depth discussion
of the Dirichlet conditions can be found in the mathematical literature on Fourier
analysis.
The Frequency Domain
The concept of frequency is clearest for simple sinusoids, but we saw in the
previous chapter that it can be useful for nonsinusoidal periodic signals as
well. The Fourier series is a useful tool for description of arbitrary periodic
signals, describing them in terms of a spectrum of sinusoids, the frequencies
of which are multiples of a basic frequency.
It is not immediately obvious that the concepts of spectrum and fre-
quency can be generalized to nonperiodic signals. After all, frequency is only
meaningful if something is periodic! Surprisingly, the concept of spectrum
turns out to be quite robust; for nonperiodic signals we simply need a con-
tinuum of frequencies rather than harmonically related ones. Thus analog
signals can be viewed either as continuous functions of time or as continuous
functions of frequency. This leads to a pleasingly symmetric view, whereby
the signal can be described in the time domain or the frequency domain.
The mathematical tool for transforming an analog signal from its time
domain representation to the frequency domain, or vice versa, is called the
Fourier transform (FT). The name hints at the fact that it is closely related
to the Fourier series that we have already discussed. For digital signals we
have close relatives, namely the discrete Fourier transform (DFT) and the z
transform (zT). In this chapter we introduce all of these, review their prop-
erties, and compute them for a few example signals. We also introduce a
non-Fourier concept of frequency, the instantaneous frequency. The FS, FT,
DFT, zT, and instantaneous frequency, each in its own domain of applica-
bility, is in some sense the proper definition of frequency.
103
104 THE FREQUENCY DOMAIN
Looking back at the steps in proving the existence of the Fourier series we
see that the periodicity of the signals was not really crucial; in fact the whole
periodicity constraint was quite a nuisance! The SUIs form a basis for all
signals, whether periodic or not. It was only when we introduced the HRSs,
sums of which are necessarily periodic, that we had to restrict ourselves to
representing periodic signals. It would seem that had we allowed arbitrary
frequency sinusoids we would have been able to represent any signal, and
indeed this is the case. In fact it would have been just as easy for us to have
directly derived the Fourier transform without the annoyance of the Fourier
series; however this would have involved a grave break with mathematical
tradition that mandates deriving the Fourier transform from the Fourier
series.
The basic idea behind this latter derivation is inherent in the FS derived
in Section 3.8. There we saw how increasing the period of the signal to be
analyzed required decreasing the fundamental frequency of the HRSs. It is
a general result that the longer the time duration that we must accurately
reproduce, the more frequency resolution is required to do so. Now let us
imagine the period going to infinity, so that the signal effectively is no longer
periodic. If you find this infinity troublesome just imagine a period longer
than the time during which you are willing to wait for the signal to repeat.
The required frequency resolution will then become infinitesimal, and at ev-
ery step of the way the corresponding HRSs form a basis for the signals with
this large period. In the limit of aperiodic signals and continuous spectrum
we discover that the set of all sinusoids forms a basis for the entire vector
space of signals. Of course, for our basis signals we can choose to use si-
nusoids in quadrature sin(&) and cos(wt), sinusoids with arbitrary phases
sin(wt + cp), or complex exponentials eid with both positive and negative
frequencies.
We have neglected an essential technical detail-as long as the funda-
mental frequency is small, but still finite, there are a denumerably infinite
number of basis signals, and so the dimension of the space is No and ex-
pansions of arbitrary signals are infinite sums. Once the spectrum becomes
continuous, there are a nondenumerable infinity of basis functions, and we
must replace the infinite sums with integrals. The set of coefficients Sk
becomes a single continuous function of frequency S(w).
The result is an expression for a signal as an integral over all time of a
function of frequency times a complex exponential.
-ii.& &
(4.1)
4.1. FROM FOURIER SERIES TO FOURIER TRANSFORM 105
an identity sometimes called the Fourier Integral Theorem. The two rep-
resentations related by the FT and FT-l operators are called a Fourier
transform pair. They are both functions of a single continuous variable and
contain exactly the same information about the signal, but in different forms.
The function s(t) is the time domain representation of the signal, while S(w)
is its frequency domain representation.
Lets prove equation (4.4).
We now see why the exponents have different signs-its required to get the
needed delta function. Incidentally, we see that instead of the normalization
constant &
-.. in the iFT we could have used any constants in both FT and
iFT whose product is $. For instance, physicists usually define a more
symmetric pair
but any other combination could be used as well. The DSP convention of
putting the constant only in the definition of the inverse transform becomes
more symmetric when using the frequency f in Hz (cycles per second) rather
than the angular frequency w in radians per second.
S(f)= I s(t)e-2iftdt
s(t)
=s fT-
-cxI
S(f)e2iftdf (4.6)
4.1. FROM FOURIER SERIESTO FOURIERTRANSFORM 107
S(w) = FT (A cos(wt)) = Jm
t=-ocl
A cos(wt)eviwtdt
the FT has discrete lines of precisely the expected relative weights. Next we
would have to consider the continuous spectra of nonperiodic signals and
show that the FT captures the meaning we anticipate. Finally, we would
need to show that the FT is zero for unwanted frequencies. This could con-
ceivably involve forcibly notching out frequencies from an arbitrary signal,
and observing the FT at these frequencies to be zero.
This prescription is perhaps overly ambitious for us at this point, and in
any case there is a shrewd way out. All we really need do is to show that the
FT is the proper generalization of the FS for possibly nonperiodic signals.
This will ensure that all well-known properties of FS spectra will survive in
the FT, and all new properties of the FT will be taken to be the definition
of what the true spectrum should be.
108 THE FREQUENCY DOMAIN
@> 5 =i(w)eiwtAw
w=-00
In the limit T + 00 and Aw + 0 several things happen. The integral
t
over now runs from -oo to +oo. The finite difference Aw becomes the
infinitesimal dw. The sum over the discrete w index in the formula for s(t)
will of course become an integral over the continuous w variable. Substitution
of these brings us to
1 00 -iwtdt
s( w > =- 27r me
J -00
s(t) - J-0000S(w)eiwtdw
4.1. FROM FOURIERSERIESTO FOURIERTRANSFORM 109
EXERCISES
4.1.1 Prove the opposite direction of (4.4), namely
S(w) = FTFT-1 (S(w))
FT (s(t)) = lw 6(t)eeiwtdt = e = 1
=- 00
using property (A.69) of the delta function. So it works the other way as
well-the transform of a delta is a constant. With only minimal additional
effort we can find the transform of an impulse at any nonzero time r. In this
case we pick out the exponential at some other time
-&t& = ,-&T
FT (6(t - Qt - de (4.81
4.2. FOURIER TRANSFORM EXAMPLES 111
and actually the same transform holds for complex A, as long as the real
part of X is positive.
Up to now we have treated rather smooth signals and impossibly singular
ones (the delta). We will also need to investigate archetypical jump discon-
tinuities, the sgn and step functions. Since sgn is odd, sgn(-t) = -sgn(t),
we can immediately deduce that the zero frequency component of sgns FT
must be zero. The zero frequency component of u(t) is obviously infinite
and so we know that u(w) must have a M(w) component. The value of AJ
can be determined from the fact that u(-t) + u(t) = 1 and from linearity
FT(u(-t)) + FT(u(t)) = FT(l) = 2nS(w); so the DC component is simply
7rS(w).
Trying to find the nonzero frequency components of either sgn or u(t)
we stumble upon one of those impossible integrals, like (4.7). For large w it
112 THE FREQUENCYDOMAIN
should go to zero since the integral over an even number of cycles of sinusoids
is zero; but for smaller w there is the issue of the end effects. We will be able
to prove later that the spectrum decays as $, i.e., every time we double the
frequency the amplitude drops to half its previous value. When displaying
the spectrum on a logarithmic scale this translates to a linear drop of 6 dB
per octave. Since any signal with a single discontinuity can be considered to
be continuous signal plus a step or sgn, all signals with step discontinuities
have this 6 dB per octave drop in their spectra.
EXERCISES
4.2.1 Calculate the FT of a complex exponential from those of sin and cos using
linearity and equation (A.8).
4.2.2 What is the difference between the FT of sin and cos? Explain the effect of
A and cpon the FT of Asin(wt + cp).
4.2.3 Find the FT of the single rectangle (equation (3.29)).
4.2.4 Show that ~~=-~ eeiwnT = 0 when w is not a multiple of y.
4.2.5 Formally prove that the FT of the impulse train s(t) = C s(t - IcT) is an im-
pulse train in the frequency domain by finding its Fourier series and relating
the transform to the series.
4.2.6 Our proof of the universality of the Fourier series in Section 3.4 rested on the
expansion of shifted delta functions in the basic period in terms of harmoni-
cally related sinusoids. Show how this can be simplified using our results for
impulse trains.
4.2.7 Prove that the following are FT pairs:
14 -3
I
emuIt 2a
3x7 1
4.3. FT PROPERTIES 113
4.3 FT Properties
As we saw in the examples, the Fourier transform of a signal may look like
just about anything. It is customary to differentiate between continuous
and discrete line spectra. When the FT is a continuous smooth function of
frequency a nondenumerable number of frequency components are required
to reproduce the signal. A FT composed of some number of sharp discrete
lines results from a signal that is the sum of that number of sinusoids.
In general, spectra may have both continuous and discrete parts. In fact all
signals encountered in practice are noisy and so cannot be precisely periodic,
and hence some continuous spectrum contribution is always present.
The question of the mathematical existence of the FT is an important
one for mathematicians, but one we will not cover extensively. The Dirichlet
conditions for the FT require that the integral over all time of the absolute
value of the signal be finite, as well as there being only a finite number of
extrema and discontinuities in any finite interval. The FT obviously does
not exist in the technical sense for periodic signals such as sinusoids, but by
allowing delta functions we bypass this problem.
Although we will not dwell on existence, there are many other character-
istics of the FT that we will need. Many times we can find the FT of signals
without actually integrating, by exploiting known transforms and some of
the following characteristics. These characteristics are often closely related
to characteristics of the FS, and so we need not derive them in detail.
First, it is important to restate the Fourier Integral Theorem that the
inverse FT given by equation (4.3) is indeed the inverse operation.
There are two properties that deal with changing the clock, namely the
time shifting property
Conversely, there is a property that deals with shifting the frequency axis
Ox(t)y(t)e+dt =
J-03
1 O ib-n)t & dfl
= i% JQ=-00 xm / t_m_mY(W
What we did was simply to replace z(t) by its iFT, change the order of
integration, and recognize the FT of y. So we have found the following:
Now that we have the answer, what does it mean? The FT of the product
of two signals in the time domain is the integral of a strange-looking product
in the frequency domain. We hide this strangeness by using the symbol *,
implying a product of some sort. Its a truly unusual product since the
integration variable in Y runs in the opposite direction to that of the X
variable. If that is not bad enough, repeating the above computation for
iFT of a product in the frequency domain, we find
X(fl)Y(-0) dS2
and by taking x(t) = s(t), y(t) = s*(t) and changing the name of the
integration variable, we get Parsevals relation for the FT.
Parsevals relation tells us that the signals energy is the same whether we
look at it in the time domain or the frequency domain. This is an important
physical consistency check.
To demonstrate the usefulness of some of these properties, we will now
use the integration rule to derive a result regarding signals with discontinu-
ous derivatives. We know that the FT of the impulse is constant, and that
its integral is the unit step u(t). Thus we would expect from (4.17) for the
FT of the step to be simply 6, which is not what we previously found!
The reason is that (4.17) breaks down at w = 0, and so we always have to
allow for the possible inclusion of a delta function. Integrating once more we
get f(t) = Wt), which is continuous but has a discontinuous first deriva-
tive. The integration rule tells us that the FT of this f is -wB2 (except at
W = 0). Integrating yet another time gives us a signal with continuous first
derivative but discontinuous second derivative and iwS3 behavior. Continu-
ing this way we see that if all derivatives up to order k are continuous but
the (k + l)th is not, then the (nonzero frequency) transform is proportional
116 THE FREQUENCY DOMAIN
Figure 4.1: The effect of derivative discontinuity on the FT. In (A) the signal itself (the
zeroth derivative) is discontinuous and the spectrum is constant. In (B) the first derivative
is discontinuous and the spectrum decays as wB2. In (C) the second derivative jumps and
the spectrum decays as we4.
EXERCISES
4.3.1 Explain why s-, &dt = 2nS(w) using a graphical argument.
4.3.2 Show that time reversal causes frequency reversal FT (s(A)) = S(-w).
4.3.3 Show how differentiation and integration of the spectrum are reflected back
to the time domain.
4.3.4 The derivative of cos(wt) is -w sin(wt). State this fact from the frequency
domain point of view.
4.3.5 Show that we can interchange X and Y in the convolution integral.
4.3.6 Redraw the right-hand side of Figure 4.1 using dB. How does the slope relate
to the order of the discontinuity?
4.3.7 Generalize the relationship between spectral slope and discontinuity order to
signals with arbitrary size discontinuities not necessarily at the origin. What
if there are manv discontinuities?
4.4. THE UNCERTAINTY THEOREM 117
- 1 ItI -< T
4) (4.21)
0 else
which is like a single cycle of the rectangular wave. The term window is
meant to evoke the picture of the opening a window for a short time. Its FT
FT (s(t)) = ST eiWtdt
-T
e+iwT - e-iwT
--
iW
- 2sin(wT)
= 2 T sinc(wT)
W
turns out to be a sine. Now the interesting thing about this sine is that its
bandwidth is inversely proportional to T, as can be seen in Figure 4.2.
The wider the signal is in the time domain, the narrower it is in frequency,
and vice versa. In fact if we define the bandwidth to be precisely between
the first zeros of the sine, Aw = Sj?, and relate this to the time duration
At = 2T, we find that the uncertainty product
Aw At = 4~
ALtA-+LL-f
Figure 4.2: Rectangular windows of various widths with their Fourier transforms. Note
that the signal energy is not normalized.
ii8 THE FREQUENCY DOMAIN
Figure 4.3: The effect of observation window duration on frequency uncertainty. In (A)
we only observe the sinusoid for an extremely short time, and hence we can not accurately
gauge its frequency. In (B) we observe about half a cycle and can now estimate the
frequency, but with relatively large uncertainty. In (C) two full cycles are observed and
consequently the uncertainty is much reduced.
4.4. THE UNCERTAINTY THEOREM 119
(4.23)
which doesnt look hopeful. The mathematical trick to use here is complet-
ing the square. The exponent is -(/3t2 + id). We can add and subtract $
so that
AtAw = f
Now it turns out that no signal has a smaller uncertainty product than
this. This theorem is called the uncertainty theorem, and it is of importance
both in DSP and in quantum physics (where it was first enunciated by
Heisenberg). Quantum physics teaches us that the momentum of a particle
is the Fourier transform of its position, and hence the uncertainty theorem
limits how accurately one can simultaneously measure its position and ve-
locity. Energy and time are similarly related and hence extremely accurate
energy measurements necessarily take a long time.
120 THE FREQUENCY DOMAIN
E= Os2(t)dt
J-ccl
time center-of-gravity
w2 -= .f.T&<t
- (t>>2s2(tPt
E
frequency center-of-gravity
(Au)2 = s-,(w
- tw>
>2s2(w)dw
E
then the uncertainty product
depicted in Figure 4.4. This signal is truly fundamental to all of DSP, since
it is the link between analog signals and their digital representations. We
can think of sampling as multiplication of the analog signal by just such a
train of impulses,
so /W = ,__,
-
c
n--m
qt - nT)+%t
4.4, THE UNCERTAINTY THEOREM 121
Figure 4.4: Trains of Dirac delta functions in time and frequency domains. Note that
the spacing in the time domain is the inverse of that in the frequency domain.
and once again we are stuck. Looking carefully at the sum we become con-
vinced that for most w the infinite sum should contain just as many negative
contributions as positive ones. These then cancel out leaving zero. At w = 0,
however, we have an infinite sum of ones, which is infinite. Does this mean
that the FT of a train of deltas is a single Dirac delta? No, because the same
thing happens for all w of the form 3 as well! So similarly to the Gaussian,
a train of impulses has an FT of the same form as itself, a train of impulses
in the frequency domain; and when the deltas are close together in the time
domain, they are far apart in the frequency domain, and vice versa. The
product of the spacings obeys
At Aw = 2n
EXERCISES
4.4.1 Prove the Schwartz inequality for signals.
4.4.2 Using Parsevals relation and the FT of a derivative prove the following re-
lation involving the uncertainties and the energy E.
4.4.3 Using the Schwartz inequality, the above relation, and integration by parts,
prove the uncertainty theorem.
f
Figure 4.5: Power spectral density for the DTMF digit 8. The horizontal axis is the
frequency in KHz and the vertical axis is a linear measure of the energy density. The eight
possible frequencies are marked for convenience.
In the next section we will see that many signals have spectral distribu-
tions that vary as time progresses. For such signals we wish to know how
much energy is in the frequency range around f at times around t. Since
the energy density in the time domain is the power, the desired quantity
is called the Power Spectral Density (PSD). PSDs that change in time are
so common that we almost always use the term power spectrum instead of
energy spectrum.
Writing the full FT as a magnitude times an angle S(f) = A(f)e@(f),
we see that the PSD contains only the magnitude information, all the angle
information having been discarded. At this stage of our studies it may not
yet be entirely clear why we need the full frequency domain representation,
but it is easy to grasp why we would want to know how a signals energy is
divided among the component frequencies. For example, push-button dialing
of a phone uses DTMF signals where two tones are transmitted at a time
(see Figure 4.5). The lower tone of the two is selected from four candidate
frequencies Ll, L2, L3, L4, and the higher is one of HI, Hz, H3, H4. In order
to know that an eight was pressed we need only ascertain that there is energy
in the vicinities of L3 and Hz. The phases are completely irrelevant.
As a more complex application, consider a phone line on which several
signals coexist. In order for these signals not to interfere with each other
they are restricted by masks, i.e., specifications of the maximal amount of
power they may contain at any given frequency. The masks in Figure 4.6 are
specified in dBm/Hz, where dBm is the power in dB relative to a 1 milli-
watt signal (see equation (A.16)). The horizontal scale has also been drawn
logarithmically in order to accommodate the large range of frequencies from
100 Hz to over 10 MHz. Although the higher frequency signals seem to be
124 THE FREQUENCY DOMAIN
Oh POTS
-10 - I
-20 -
-30 - US ADSL DS ADSL
-40 -
-50 -
-60 -
-70 -
-80 -
-90 - I L I
1OOHz 1KHz 1OKHz 1OOKHz 1MHZ 1OMHz
Figure 4.6: PSD masks for several signals on a phone line. The horizontal axis is the
frequency in KHz on a logarithmic scale and the vertical axis is the maximum allowed
PSD in dBm per Hz. The leftmost signal is the POTS (Plain Old Telephone System)
mask, including voice and voicegrade modems. The middle mask is for ADSL, with the
lower portion for the 512 Kb/s upstream signal and the upper for the 6 Mb/s downstream
signal. At the far right is the mask for the 1 Mb/s Home Phone Network signal.
lower in power, this is only an illusion; the PSD is lower but the bandwidths
are much greater.
The mask containing the lowest frequencies is for regular telephone con-
versations, affectionately called Plain Old Telephone Service (POTS). This
mask, extending from 200 Hz to about 3.8 KHz, holds for voice signals,
signals from fax machines, and voicegrade modems up to 33.6 Kb/s.
The need for high-speed digital communications has led to innovative
uses of standard phone lines. The Asymmetric Digital Subscriber Line
(ADSL) modem is one such invention. It can deliver a high-speed down-
stream (from the service provider to the customer) connection of up to 8
Mb/s, and a medium-speed upstream (from the customer to the provider)
connection of 640 Kb/s. ADSL was designed in order not to interfere with
the POTS signal, so that the standard use of the telephone could continued
unaffected. By placing the ADSL signal at higher frequencies, and restricting
the amount of power emitted at POTS frequencies, interference is avoided.
This restriction may be verified using the power spectrum; the signal phases
are irrelevant.
In the same way, after the definition of ADSL the need arose for net-
working computers and peripherals inside a residence. Of course this can be
done by running cables for this purpose, but this may be avoided by using
the internal phone wiring but requiring the new home phone network signal
to lie strictly above the POTS and ADSL signals.
We see that based on the power spectrum alone we may deduce whether
signals may coexist without mutual interference. The principle behind this
4.5. POWER SPECTRUM 125
Figure 4.7: Power spectral density for speech, more specifically the sound eh pronounced
by a male speaker. The horizontal axis is the frequency in KHz and the vertical axis is
the energy density in dBm per Hz. The spectrum is obviously made up of discrete lines,
and we note that three main resonances at 500, 1820, and 2510 Hz and a weak fourth at
a higher frequency.
EXERCISES
4.5.1 Write a program that finds the PSD by numerical integration (equation (4.1))
and squaring. Use this program to find the PSD of a rectangular window
(equation (4.21)) for several different widths. Repeat the exercise for a sinc-
shaped pulse for several different pulse widths.
4.5.2 Build 1024 samples of sine waves of 1, 2, 3, 4, 5, 6, 7, and 8 KHz sampled
at 8 KHz. Observe the sines in the time domain; can you seethe aliasing for
f > 4 KHz? Extract the PSD (if you didnt write your own program in the
first exercise many programs are readily available for this purpose). Can you
read off the frequency? What do you see now for f > 4 KHz?
4.5.3 Build 1024 sample points of sine waveswith frequencies 1.1, 2.2, and 3.3 KHz
sampled at 8 KHz. What happened to the spectral line? Try multiplying the
signal by a triangular window function that linearly increases from zero at
n = 0 to one at the center of the interval, and then linearly decreasesback
to zero).
4.5.4 In exercise 2.6.4 we introduced the V.34 probe signal. Extract its power
spectrum. Can you read off the component frequencies?What do you think
the probe signal is for?
4.5.5 Find the PSD of the sum of two sinusoids separated by 500 Hz (use 2 KHz &
500 Hz) sampled at 8 KHz. Can you distinguish the two peaks? Now reduce
the separation to 200 Hz. When do the two peaks merge? Does the triangular
window function help?
4.5.6 In the text it was stated that isolation in the frequency domain is a sujficient
but not a necessary condition for signals to be separable. Explain how can
signals can be separated when their PSDs overlap.
In Section 4.2 we found the FT for various infinite time signals. Could we
have approximated these mathematical results by numerically integrating
over a finite amount of time? Other than the restrictions placed by the
uncertainty theorem it would seem that this is possible. One neednt observe
a simple sinusoid for years and years to be able to guess its spectrum. Of
course the longer we observe it the narrower the line becomes, but we will
probably catch on after a while. The problem is that we cant be completely
sure that the signal doesnt radically change the moment after we give up
observing it. Hence we can only give our opinion about what the signals
FT looked like over the time we observed it. Unfortunately, the FT isnt
defined that way, so we have to define a new entity-the Short Time Fourier
Transform (STFT).
Consider the signal
sin(27rflt) t < 0
s1(t) =
{
sin(27rfzt) t 2 0
which is a pure sine of frequency fl from the beginning of time until at time
t = 0 when, for whatever reason, its frequency abruptly changes to f2. What
is the FT of this signal?
As we have seen, the FT is basically a tool for describing a signal si-
multaneously at all times. Each frequency component is the sum total of all
contributions to this frequency from time t = -oo to t = +oo. Consequently
we expect the power spectrum calculated from the FT to have two equal
components, one corresponding to fr and the other to f~.
Now consider the signal
sin(27rfzt) t < 0
sz(t) =
{
sin(2nflt) t 2 0
It is clear that the power spectrum will continue to be composed of two equal
components as before since time reversal does not change the frequency
composition. Assume now that fl and fz correspond to a whole number of
cycles per second. Then the signal sa(t)
4000
3000
2000
1000
0
DI GI TA L SI GNAL PROCESSING
Figure 4.8: Sonogram of author saying digital signal processing. The vertical axis is the
frequency from 0 to 4000 Hz, while the horizontal axis is time (approximately 2 seconds).
The sounds being uttered at each time are indicated by the writing below.
4.6. SHORT TIME FOURIER TRANSFORM (STFT) 129
200 Hz is clearly visible at the bottom, and the difference between vowels
and consonants is readily identifiable, You can probably also discern which
syllables are accented, and may be able to see similarities among the various
i sounds. The two s sounds in the last word seem to be invisible; this is due
to their indeed having low energy, and most of that energy being spread
out and at high frequencies, above the bandwidth displayed here. The ing
is also very weak, due to being unaccented.
Rotating the sonogram by 90 we obtain the fulling raster spectrogram
popular in radar signal processing. Here the horizontal axis represents fre-
quencies in the region of interest, time advances from top to bottom, and
gray-scale intensity once again represents the square amplitude of the STFT
component. Once the desired range of frequencies is selected, falling raster
spectral displays provide intuitive real-time pictures; the display scrolling
upwards as text does on a computer terminal.
The transition from FT to STFT requires forcing arbitrary signals to
become finite time signals. To accomplish this we multiply the signal by a
window function, that is, a function w(t) that is strictly zero outside the time
of interest. The window function itself should not introduce any artifacts
to the spectrum of this product, and will be discussed in more detail in
Section 13.4. For now you can think of the simplest window, the rectangular
window of equation (4.21). Also commonly used are window functions that
rise smoothly and continuously from zero to unity and then symmetrically
drop back down to zero.
Of course, the uncertainty theorem puts a fundamental limitation on
the precision of the STFT. The longer the time during which we observe a
signal, the more precise will be our frequency. distribution predictions; but
the longer the window duration the more we blur the frequency changes
that may be taking place in the signal. The uncertainty inequality does not
allow us to simultaneously measure to arbitrary accuracy both the spectral
composition and the times at which this composition changes.
The sonogram and similar graphic displays are tools to view the signal
simultaneously in the time and frequency domains, yet they do not treat
time and frequency on equal footing. What we may really want is to find a
function f(t,w) such that f(t, w) dtdw is the energy in the time-frequency
cell. This brings us to define joint time-frequency distributions.
These are derived by considering time and frequency to be two character-
istics of signals, just as height and weight are two characteristics of humans.
In the latter case we can define a joint probability density p(h, w) such that
p(h, w) dh dw is the percentage of people with both height between h and
h + dh and weight between w and w + dw (see Appendix A.13). For such
130 THE FREQUENCYDOMAIN
where the integrations are over the entire range of possible heights and
weights, p(h)dh is the percentage of people with height between h and h+dh
regardless of weight, and p(w)dw is the percentage of people with weight
between w and w + dw regardless of height.
Similarly, a joint time-frequency distribution is a function of both time
and frequency p(t, w). We require that the following marginals hold
s(t) 00
=s-co
I44
4d&J S(w) = /* p(t,w) dt
-00
and the integration over both time and frequency must give the total energy,
which we normalize to E = 1. We may then expect p(t, w) dt CL to represent
the amount of energy the signal has in the range between w and w + dw
during the times between t and t + dt.
Gabor was the first to express the STFT as a time-frequency distribution
PW = WI2 lS(412
while more complex is the Wigner-Ville distribution
Pk 4 = &/s*(t-;) ciwTs(t+;) dr
The double square requires computing lS(c~)l~ by the FTs integral over all
time, and then simply multiplies this by the signal in the time domain. It
is obviously zero for times or frequencies for which the signal is zero, but
doesnt attempt any more refined time-frequency localization. The Wigner-
Ville formula looks similar to equation (4.27) for finding the power spectrum
via the autocorrelation, and is only one of an entire family of such bilinear
distributions.
4.6. SHORT TIME FOURIERTRANSFORM (STFT) 131
but none are perfect. Although they all satisfy the marginals, unexpected
behaviors turn up. For example, when two frequencies exist simultaneously,
some distributions display a third in between. When one frequency compo-
nent ceases and another commences a short time later, some distributions
exhibit nonzero components in the gap. These strange phenomena derive
from the bilinear nature of the Cohen distributions. Even more bizarre is
the fact that while the short-time PSD and the double-square are always
positive, most of the others can take on nonintuitive negative values.
EXERCISES
4.6.1 There is another case for which we can compute the FT after only a finite
observation time, namely when someone guarantees the signal to be periodic.
Do we need the STFT for periodic signals?
4.6.2 In the text, examples were presented of signals with identical power spec-
tra. Doesnt this contradict the very nature of a transform as a reversible
transformation to another domain? Resolve this paradox by demonstrating
explicitly the difference between the three cases.
4.6.3 Compute the FT by numerical integration and plot the empirical PSD of a
sinusoid of time duration T. How does the line width change with T?
4.6.4 A FSK signal at any given time is either one of two sinusoids,,one of frequency
wr, and the other of frequency LJ~. Generate a FSK signal that alternates
between wr and w2 every T seconds, but whose phase is continuous. Using
a sampling frequency of 8000 Hz, frequencies 1000 and 2000 Hz, and an
alternation rate of 100 per second, numerically compute the power spectrum
for various window durations. You may overlap the windows if you so desire.
Plot the result as a falling raster spectrogram. What do you get when a
transition occurs inside a window? Does the overall picture match what you
expect? Can you accurately measure both the frequencies and the times that
the frequency changed?
4.6.5 Repeat the previous exercise with the double-square distribution.
4.6.6 Show that the uncertainty theorem does not put any restrictions on joint
time-frequency distributions, by proving that any distribution that satisfies
the marginals satisfies the uncertainty theorem.
132 THE FREQUENCYDOMAIN
which is the DFT. The power spectrum for the digital case is ISkI and each
Ic represents the energy that the signal has in the corresponding frequency
bin.
For a given N, it is useful and customary to define the Nth root of unity
WN. This is a number, in general complex, that yields unity when raised to
the Nth power. For example, one square root of unity is -1 since ( -1)2 = 1;
but l2 = 1 so 1 is a square root of itself as well. Also i is a fourth root of
unity since i2 = (-1O)2 = 1, but so are -i, -1, and 1. There is a unique best
choice for WN, namely the trigonometric constant
(4.30)
This is the best choice since its powers Wh for k = 0.. . N - 1 embrace all
the N roots. Thinking of the complex numbers as points in the plane, WN is
clearly on the unit circle (since its absolute value is one) and its phase angle
is $ of the way around the circle. Each successive power moves a further h
around the circle until for N = 1 we return to WE = 1. This is illustrated
in Figure 4.9 for N = 8.
Figure 4.9: The N complex roots of unity displayed graphically. (Here N = 8.)
(just note that (WN) = (e-is)nk = e-v). The powers (WN)~ are also
on the unit circle, but at integer multiples of the basic angle. Consequently
the set of all the powers of WN divides the unit circle into N equal pieces.
134 THE FREQUENCY DOMAIN
so = -&snw;= so+s1
n=O
Sl = 5 SnWF = SO - Sl (4.33)
n=O
The (short time) DFT takes in a finite number of digital values and
returns a finite number of digital values. We thus have a true transform
designed for digital computation. However, this transform is still a math-
ematical concept, not a practical tool. In Chapter 14 we will see that the
DFT is eminently practical due to the existence of an efficient algorithm for
its computation.
EXERCISES
4.7.1 Derive the LTDFT directly from the FT.
4.7.2 Express W, and kVhN-)-k in terms of IV&. Express IV; + I%; and
IV; - IVGk in terms of sine and cosine. How much is kVk+m)k? Derive
the trigonometric sum formulas (A.23) using these relations.
4.7.3 What is the graphical interpretation of raising a complex number to a positive
integer power ? What is special about numbers on the unit circle? Give a
graphical interpretation of the fact that all powers of VVN are N roots of
unity. Write a program that draws the unit circle and ail the IV&. Connect
consecutive powers of each root with straight lines. Describe the pictures you
obtain for odd and even N.
4.7.4 What are the equations for 4-point DFT, and what is their interpretation?
4.7.5 Write a straightforward routine for the computation of the DFT, and find
the digital estimate of the PSD of various sinusoids. Under what conditions
is the estimate good?
for integer k, which are precisely the signals with only one nonzero DFT com-
ponent . For all integer k in the range 1 5 k 5 s the signal with frequency
f = +h and the corresponding signal with negative frequency f = -$
are different. The real part of the complex exponential is a cosine and so is
unchanged by sign reversal, but the imaginary term is a sine and so changes
sign. Hence the two signals with the same IfI are complex conjugates, When
k = -% the frequency is f = -3 and the imaginary part is identically zero.
Since this signal is real, the corresponding f = +$ signal is indistinguish-
able. Were we (despite the redundancy) to include both f = ZJZ$signals in a
basis, the corresponding expansion coefficients of an arbitrary signal would
be identical; exactly that which is needed for periodicity to hold.
. . . s-4, s-3, s-2, s-1, SO, sl, s2, s3, s-4, s-3, s-2, s-1, SO, sl, s2, s3, . . .
obtained by swapping the first half (S-d,S-3, S-2, S-1) with the second
(sO,%s2,s3)~
Lets observe a digital signal sn from time n = 0 until time n = N - 1
and convert it to the frequency domain Sk. Now using the iDFT we can
compute the signal in the time domain for all times n, and as we saw in
the previous section the resulting sn will be periodic. No finite observation
duration can completely capture the behavior of nonperiodic signals, and
assuming periodicity is as good a guess as any. It is convenient to visualize
digital signals as circular buffers, with the periodicity automatically imposed
by the buffer mechanics.
Now for some new properties. The DFT of a real signal is Hermitian
even,
s-k = s; for real sn (4.38)
and that of an imaginary signal is Hermitian odd. Evenness (or oddness) for
finite duration discrete time signals or spectra is to be interpreted accord-
ing to the indexation scheme of the previous paragraph. For example, the
spectrum So, Sl, S2, S3, S-4, S-3, S-2,S-1 =
A-1. lb-l.
7, -l+@+l)i, -l+i, -l+~i, -1, -1-41, -l-i, -l-(&+i)i
will be real.
The properties that deal with transforming the discrete time and fre-
quency axes are the time shifting property
Of course, for finite-duration DFTs, time shifts can move us to times where
we havent observed the signal, and frequency shifts to frequencies where we
havent computed the DFT. When this happens simply use the periodicity
properties. When we use the word shift for digital signals we always mean
circular shift (i.e., shift in a circular buffer).
Parsevals relation for the DFT is easy to guess
(4.42)
n=O k=O
and for infinite duration signals the sum on the left is over a denumerably
infinite number of terms and the right-hand side becomes an integral.
The simplest application of Parsevals relation for the DFT involves a signal
of length two. The DFT 1s
so = so + Sl Sl = so - Sl
When the signals are of finite time duration the periodicity forces us to define
a new kind of convolution sum, known as circular (or cyclic) convolution.
(4.46)
x2Y3 + x3Y2
X3Y3
Ill t
Figure 4.10: The sampling theorem. On the left we see the signals of interest in the time
domain, and on the right in the frequency domain. The graphs in (A) depict the original
analog signal, those in (B) the sampling impulses, and in (C) the sampled signal.
EXERCISES
4.8.1 Prove all of the DFTs properties stated above.
4.8.2 DFT routines usually return the same number of outputs as inputs, but
sometimes we need higher frequency resolution. Assuming that we only have
access to N samples, how can we generate 2N DFT components? Conversely,
assume we have N DFT components and require 2N signal values. How can
we retrieve them? These tricks seem to create new information that didnt
previously exist. How can this be?
4.8.3 Prove that an even time signal has an even DFT, and an odd time signal has
an odd DFT. What can you say about real even signals?
4.8.4 Explain why we didnt give the counterparts of several of the properties
discussed for the FT (e.g., time scaling and differentiation).
4.8.5 Why does the circular convolution depend on N? (Some people even use the
notation x Q y to emphasize this fact.)
4.8.6 In Section 2.8 we mentioned the band-pass sampling theorem that holds for
a signal with components from frequency fo > 0 to fi > fo. Using a figure
similar to Figure 4.10 find the precise minimal sampling rate.
4.8.7 What can be said about the FT of a signal that is zero outside the time
interval -T < t < +T? (Hint: This is the converse of the sampling theorem.)
4.9. FURTHER INSIGHTS INTO THE DFT 141
-s=ws
--
-
(4.48)
For example, the simple two-point DFT of equation (4.33) can be written
more compactly as
(z)=(: l)(Z)
as can be easily seen. More generally, the WN matrix is
w=
=
(4.49)
1 1 1 *
wN-1 \
1 WN
w2rN-l)
1 w;t,
= w?(N-l)
1 w; w; N
.. .. .
. ..
; wj- 1 w2(N-l) . . . w(/-)(N-)
N
and since WN is the Nth root of unity, the exponents can be reduced modulo
N. Thus
w2=(% z)=(: !l) (4.50)
C
142 THE FREQUENCY DOMAIN
w4 = pg!i#j=[;~$gj(4.51
=
t 1 -1
-i 1i -11 -;-i1 1
and
ws = (4.52)
-s = =-
w-s (4.53)
and
=- ;W* (4.54)
E- =
where the Hermitian conjugate of the WN matrix has elements
(w*)$c = ,+iy = W-$C
indices and write W E IV;. Then the DFT takes the form of a polynomial
in W with coefficients sn
N-l
Sk = c &wn (4.55)
n=O
which is a viewpoint that is useful for two reasons. First, the connection with
polynomials will allow use of efficient algorithms for computation of polyno-
mials to be used here as well. The FFT, although first introduced in signal
processing, can be considered to be an algorithm for efficient multiplica-
tion of polynomials. Also, use of Horners rule leads to an efficient recursive
computation for the DFT known as Goertzels algorithm. Second, a more
modern approach considers the DFT as the polynomial approximation to the
real spectrum. When the real spectrum has sharp peaks such a polynomial
approximation may not be sufficient and rational function approximation
can be more effective.
EXERCISES
4.9.1 Write explicitly the matrices for DFT of sizes 3, 5, 6, 7, and 8.
4.9.2 Invert the DFT matrices for sizes2, 3, and 4. Can you write the iDFT matrix
in terms of the DFT matrix?
4.9.3 Prove that any two rows of the DFT matrix are orthogonal and that the
squared length of any row is N. Show that -&b is a unitary matrix.
In the special world of DSP there is, however, an alternative. This alter-
native is entirely meaningless in the analog world, in some ways less mean-
ingful than the Fourier spectrum even in the digital world, and on occasion
seems to be a mere artificial, purely mathematical device. It does sometimes
enhance our understanding of signals, often greatly simplifies calculations,
and always includes Fouriers spectrum as a special case.
This alternative is called the x trunsfonn, which we shall denote zT. This
nomenclature is admittedly bizarre since the use of the letter x is completely
arbitrary (there was no section in the previous chapter named Z Discovers
Spectrum), and it is not really a transform at all. Recall that the FS, which
maps periodic analog signals to discrete spectra, is not called a transform.
The FT, which maps analog signals to continuous spectra, and the DFT,
which makes digital signals into discrete spectra, are. The zT takes an arbi-
trary digital signal and returns a continuous function. This change of form
from sequence to function should disqualify it from being called a transform,
but for some reason doesnt. Even more curious is the fact that outside the
DSP sphere of influence the term z transform is entirely unknown; but a
closely related entity is universally called the generating function.
As we have done in the past, we shall abide by DSP tradition. After all,
every field has its own terminology that has developed side by side with its
advances and applications, even if these terms seem ridiculous to outsiders.
Computer hardware engineers use flip-flops without falling. Programmers
use operating systems without upsetting surgeons. Mathematicians use ir-
rational numbers and nonanalytic functions, and no one expects either to
act illogically. High-energy physicists hypothesize subatomic particles called
quads that have strangeness, flavor, and even charm. When lawyers garnish
they leave people without appetite, while according to their definitions the
victim of battery can be left quite powerless. So saying DC when there is no
electric current, spectral when we are not scared, and x transform pales in
comparison with the accepted terminologies of other fields!
The basic idea behind the classic generating function is easy to explain;
it is a trick to turn an infinite sequence into a function. Classic mathematics
simply knows a lot more about functions than it does about infinite se-
quences. Sometimes sequences can be bounded from above or below and in
this way proven to converge or not. A few sequences even have known limits.
However, so much more can be accomplished when we know how to change
arbitrary sequences into functions; specifically, recursions involving sequence
elements become algebraic equations when using generating functions.
4.10. THE Z TRANSFORM 145
and this is what we wish to evaluate. To proceed, take the recursion that
defines the Fibonacci sequence, multiply both sides by P and sum from
n = 2 to infinity.
cm co co
Cf nxn = Cf n-lxn+ Cf
n=2
n-2X
n
n=2 n=2
= x c fn-lxn-l + x2 c fne2p-2
n=2 n=2
= xgfnXn+X2gfnXn
n=l n=O
f (4 - fox0- flJ: l = x (f (2) - foxO)+ x2f (x)
f (2) - 1 - x = f (x)x - x + f (x)x2
Solving the algebraic equation we easily find an explicit expression for the
generating function A
f(x) = l-x-221
which is plotted in Figure 4.11.
Figure 4.11: The generating function for the Fibonacci sequence. Note the divergences
at -y x -1.618 and -7 M 0.618.
4.10. THE Z TRANSFORM 147
Ox) = (x + $x + 7) >
where a + b = -ab = 1. Utilizing the formula for the sum of a geometric
1
progression l-az = Cr!-o(all:)n and comparing term by term, we find
and you surely discern two modifications but there is also a third. First,
we needed to make the sum run from minus infinity rather than from zero;
second, the DSP convention is to use 2-l rather than x; and third, we
will allow .Z to be a complex variable rather than merely a real one. The
second change is not really significant because of the first; using .Z instead
of x-l is equivalent to interchanging sn with s+. The really consequential
148 THE FREQUENCY DOMAIN
2 = r&W
(4.60)
n=-co n=--00
which is precisely the DFT. The zT reduces to the DFT if evaluated on the
unit circle.
For other nonunity magnitudes we can always write r = ex so that
2 = ex+iw and
which is a digital version of the Laplace Transform (LT). The Laplace trans-
form, which will not be discussed in detail here, expands functions in terms
of exponentially increasing or damped sinusoids, of the type described in
equation (2.11). Its expression is
S=1+2+4+8+...
S = 1 + 2(1+ 2 + 4 + 8 + . . .)
and we see that the expression in the parentheses is exactly S. This implies
that S = 1 + 2S, which can be solved to give S = -1. The problem here
is that the infinite sum in the parentheses is meaningless, and in particular
one cannot rely on normal arithmetical laws (such as 2(a + b) = 2a + 2b) to
be meaningful for it. Its not just that I is infinite; I is truly meaningless
and by various regroupings, factorings, and the like, it can seem to be equal
to anything you want.
The only truly well-defined infinite series are those that are absolutely
convergent. The series
S=C an
n=O
is absolutely convergent when
A= y, ~%-tl
n=O
Since the zT terms are a, = s&, our first guess might be that 1x1
must be very small for the sum to converge absolutely. Note, however, that
the sum in the zT is from negative infinity to positive infinity; for absolute
convergence we require
where we defined C G z-r. If lzl is small then ICI is large, and consequently
small values of Iz I can be equally dangerous. In general, the Region Of
Convergence (ROC) of the z transform will be a ring in the z-plane with
the origin at its center (see Figure 4.12). This ring may have T = 0 as its
lower radius (and so be disk-shaped), or have r = co as its upper limit, or
even be the entire z-plane. When the signal decays to zero for both n + --00
and n + 00 the ring will include the unit circle.
Figure 4.12: In general, the region of convergence (ROC) of the z transform is a ring in
the z-plane with the origin at its center.
The x-plane where the zT lives, with its ROCs, poles, and zeros, is a
more complex environment than the frequency axis of the FT. We will learn
a lot more about it in the coming chapters.
EXERCISES
4.10.1 The zT is an expansion in basis functions zn = reiwn. Show that this basis
is orthogonal.
4.10.2 Derive the generating function for a Fibonacci sequencewith initial condi-
tions fo = 1, fl = 2. What is the explicit formula for fn?
4.11. MORE ON THE Z TRANSFORM 151
4.10.3 The integer recursions for the two families of sequences g+ and g-
which is analogous to the FT result. The series converges for all z in the
z-plane. Were the impulse to appear at time m $ 0, it is easy to see that
we would get S(z) = z-~, which has a zero at the origin for negative times
and a pole there for positive ones. The ROC is the entire plane for m 5 0,
and the entire plane except the origin for m > 0.
152 THE FREQUENCY DOMAIN
Using the equation (A.47) for the sum of an infinite geometric series, we find
z
S(x) = l =- (4.63)
1 - cw-1 z-a
which has a pole at z = a. The ROC is thus 1x1> 1~11,the exterior of disk
of radius a. When does the FT exist? As a general rule, poles in the z-plane
outside the unit circle indicate explosive signal growth. If Ial < 1 the ROC
includes the unit circle, and so the FT converges. For the special case of the
unit step Sn = un, we have QI = 1, SO the zT is 5 with ROC 1~1> 1; the
FT does not exist.
We can shift the signal step to occur at time m here as well. In this case
The ROC is now jz[ < Ial, th e interior of the disk. Shifting the ending time
to n = m we get
The next most important property of the zT is the effect of a time shift.
For the FS and FT, shifting on the time axis led to phase shifts, here there is
something new to be learned. In the cases we saw above, the effect of shifting
the time by m digital units was to multiply the zT by zmrn. In particular the
entire effect of delaying the digital signal by one digital unit of time was to
multiply the zT by a factor of z -. This is a general result, as can be easily
derived.
-(n+l)
zT(xn-1) = C~?-ooXn-lX-n = C XnZ
n=-co
co
n=-00
i+$+i+&+...=l
all of the material has been lost (actually converted into another material).
The mass left as a function of time measured in half-lives is
over any closed counterclockwise contour within the ROC. This type of
integral is often calculated using the residue theorem, but we will not need
to use this complex mechanism in this book.
4.12. THE OTHER MEANING OF FREQUENCY 155
Many more special zTs and properties can be derived but this is enough
for now. We will return to the zT when we study signal processing systems.
Systems are often defined by complex recursions, and the zT will enable us
to convert these into simple algebraic equations.
EXERCISES
4.11.1 Write a graphical program that allows one to designate a point in the z-plane
and then draws the corresponding signal.
4.11.2 Plot the z transform of 6, 9m for various m.
4.11.3 Prove the linearity of the zT.
4.11.4 Express zT(~n2,) in terms of z(z) = zT(z,).
4.11.5 What is the z transform of the following digital signals? What is the ROC?
1. 60
2. %2+2
3. anu(n)
4. ant+n - 1)
5. 2jnUn + inU-n
for some A(t) and G(t). This is related to what is known a~ the analytic
representation of a signal, but we will call it simply the instantaneous rep-
resentation. The function A(t) is known a~ the instantaneous amplitude of
the signal, and the Q(t) is the instantaneous angle. Often we separate the
angle into a linear part and the deviation from linearity
where the frequency w is called the carrier frequency, and the residual 4(t)
the instantaneous phase.
The instantaneous frequency is the derivative of the instantaneous angle
which for a pure sinusoid is exactly the frequency. This frequency, unlike
the frequencies in the spectrum, is a single function of time, in other words,
a signal. This suggests a new world view regarding frequency; rather than
understanding signals in a time interval as being made up of many frequen-
cies, we claim that signals are fundamentally sinusoids with well-defined
instantaneous amplitude and frequency. One would expect the distribution
of different frequencies in the spectrum to be obtained by integration over
the time interval of the instantaneous frequency. This is sometimes the case.
4.12. THE OTHER MEANING OF FREQUENCY 157
Figure 4.13: The beat signal depicted here is the sum of two sinusoids of relatively
close frequencies. The frequencies we see (and hear) are the average and half-difference
frequencies, not the Fourier components.
158 THE FREQUENCYDOMAIN
x(t) = Acos(wt)
y(t) = Aces w t
( (-&))=x(t-&)
we have
y(t) = %x(t) = A 1sin(wt) + A2 sir@)
of a signal, which we now call z(t). Since the Hilbert transform instanta-
neously shifts all A cos(wt) to A sin(wt), we can explicitly express y(t).
w(t)=---
d4@)
dt
(4.72)
EXERCISES
4.12.1 We applied the Hilbert transform to z(t) = cos(wt + 4(t)) and claimed that
one obtains y(t) = sin@ + 4(t)). Using trigonometric identities prove that
this is true for a signal with two frequency components.
4.12.2 Even a slowly varying phase may exceed 2n or drop below zero causing
nonphysical singularities in its derivative. What should be done to phases
derived from equation (4.71) in such a case?
4.12.3 What is the connection between the instantaneous frequency and the spec-
trum of the signal? Compare the short time power spectrum calculated over
a time interval to the histogram of the instantaneous frequency taken over
this interval.
4.12.4 Show that given a signal s(t) and any amplitude signal A(t) an appropriate
phase (a(t) can be found so that equation (4.65) holds. Similarly, show that
given any phase an amplitude signal may be found. The amplitude and phase
are not unique; the z(t) and y(t) that are related by the Hilbert Transform
are the canonical (simplest) representation.
4.12.5 Find an explicit direct formula for the instantaneous frequency as a function
of z(t) and y(t). What are the advantages and disadvantages of these two
methods of finding the instantaneous frequency?
4.12.6 We can rewrite the analytic form of equation (4.68) in quadrature form.
x(t) = a(t) cos(wt) + b(t) sin(wt)
What is the connection between a(t), b(t) and A(t), 4(t)? We can also write
it in sideband form.
x(t) = (u(t) + l(t)) cos(wt) + (u(t) - l(t)) sin(wt)
What are the relationships now?
Bibliographical Notes
The DFT and zT are covered well in many introductory texts, e.g., [187, 252, 1671,
while the Hilbert transform and analytic representation are confined to the more
advanced ones [186, 2001. An early book devoted entirely to the zT is [125], while
tables were published even earlier [97].
The uncertainty theorem was introduced in quantum mechanics by Heisenberg
[99]. Another physicist, Wigner [282], derived the first example of what we would
call a time-frequency distribution in 1932, but this mathematical achievement had
to be translated into signal processing terms. The article by Leon Cohen [39] is the
best introduction.
5
Noise
161
162 NOISE
EXERCISES
5.1,l The game of guessit is played by two or more people. First the players
agree upon a lengthy list of functions of one variable t, each of which is also
dependent on one or two parameters. The inventor picks function from the
list and supplies parameters. Each analyst in turn can request a single value
of the function and attempt to guesswhich function has been selected. What
strategy should the inventor use to make the analysts task more difficult?
What tactics can the analysts use? Try playing guessit with some friends.
5.1.2 Generate a signal x with values in the interval [0 . . . l] by starting at an
arbitrary value in the interval and iterating xn+l = Xx,(1-xzn) for 0 5 X 5 4.
For what values of X does this signal look random?
5.1.3 To which of the four types of unpredictable signal does each of the following
most closely belong?
1. Static noise on shortwave radio
2. Sequenceof heads (s=l) and tails (s=O) obtained by throwing a coin
3. World population as a function of time
4. Value of stock portfolio as a function of time
5. Sequence produced by your compilers random number generator
6. Distance from earth to a given comet
7. Position of a certain drop of water going down a waterfall
8. Maximum daily temperature at your location
9. The sequenceof successivedigits of 7r
164 NOISE
Figure 5.1: A few realizations of a noise signal. The set of all such realizations is called
the ensemble. Note that each realization is erratic, but although the different realizations
are quite varied in detail, there is something similar about them.
Figure 5.2: A few realizations of another noise signal. Note the differences between this
noise signal and the previous one. Although both have zero average and roughly the
same standard deviation, the first is uniformly distributed while this signal is Gaussian
distributed. A few values are off-scale and thus do not appear.
___ _..._.........
.... .-....... -. ..
KG.
__............
.....__.
._._.._
......-........
...--..
.._...
Figure 5.3: A few realizations of a third noise signal. Note the differences between this
noise signal and the previous two. Although the signal is also zero average and of the
same standard deviation, the first two signals were vrhite while this signal has been low-
pass filtered and contains less high-frequency energy,
166 NOISE
Note that the probabilities sum to one since each signal value must be either
-1, 0, or +l. One signal with such a distribution may be
. . . - l,O, +l, +1,0, -l,O, -l,O, +I, +I, -1, +I, 0, -1, * * *
CP(Xi) = 1 (5.2)
where the sum is over all possible signal values.
We said before that the probability distribution contains the most de-
tailed information available as to individual signal values. This implies that
all single signal value statistics can be derived from it. For a digital signal
we can express the mean as a sum over time,
1 N
I-L= ( Sn > =- N c Sn (5 .3)
n=l
or we can sort the terms such that smaller sn appear before larger ones.
5.2. A NAIVE VIEW OF NOISE 167
P= 2 P(S>S (5 .4
S=--00
o2 =
(
(Sn - P)2) = $ &sn - P>2 (5.5)
n=l
ccl
some real number, the entire distribution must be normalized.
s --00
p(x) dx = 1 (5.7)
In analogy with the digital case, the mean and variance are given by the
P=s00
following.
-00
P(X) x dx o2 = I p(x)(x - p)2dx (5.8)
168 NOISE
lt 0
Figure 5.4: Four different probability distributions. (A) representsthe uniform distribu-
tion. (B) depicts an exponential distribution. (C) is the bell-shapedGaussian(or normal)
distribution. (D) is a representative bimodal distribution, actually the mixture of two
Gaussianswith different means.
(5.9)
is the famous Gaussian function, depicted in Figure 5.4.C. It is well known
that when many students take an exam, their grades tend to be distributed
in just this way. The rather strange constant before the exponent ensures
that the Gaussian is normalized.
The frequency domain characteristics of random signals are completely
distinct from the single-time signal value characteristics we have discussed so
far. This may seem remarkable at first, since in DSP we become accustomed
to time and frequency being two ways of looking at one reality. However,
the dissimilarity is quite simple to comprehend. Consider a digital signal
with some signal value distribution and a new signal obtained by arbitrarily
replicating each signal value
so that each value appears twice in a row. The new signal obviously has the
same single-sample statistics as the original one, but its frequencies have
been halved! Alternatively, consider permuting the order of signal values;
this once again obviously results in an identical probability distribution,
but quite different frequency characteristics! A signals frequency statistics
are determined by the relationship between signal values at various rela-
tive positions, and thus contains information different from the signal value
statistics.
We will often talk of white noise. White noise is similar to white light
in that its spectrum is flat (constant, independent of frequency). Having all
possible frequencies allows the signal to change very rapidly, indeed even
knowing the entire past history of a white noise signal does not contribute
anything to prediction of its future. We thus call a discrete time signal sn
white noise if observation of {sn}!&, does not allow us to say anything
useful about the value of sk other than what the single-signal value statistics
tell us.
Of course not all noise is white; when the noise signals spectrum is
concentrated in part of the frequency axis we call it colored noise. Colored
noise can be made by passing white noise through a band-pass filter, a
device that selectively enhances Fourier components in a certain range and
rejects others. As we decrease the bandwidth of the filter, the signal more
and more resembles a sine wave at the filters central frequency, and thus
becomes more and more predictable.
Since they are independent, time and frequency domain characteristics
can be combined in arbitrary ways. For example, white noise may happen
to be normally distributed, in which case we speak of Gaussian white noise.
However, white noise may be distributed in many other ways, for instance,
uniformly, or even limited to a finite number of values. This is possible
because the time domain characteristics emanate from the individual signal
values, while the frequency domain attributes take into account the relation
between values at specific times.
Our naive description of noise is now complete. Noise is just like any other
signal-it has well defined time domain and frequency domain properties.
Although we have not previously seen a flat spectrum like that of white
noise, nothing prevents a deterministic signal from having that spectrum;
and colored noise has narrower spectra, more similar to those with which we
are familiar. The time domain characterization of noise is different from that
of regular signals -rather than specifying how to create the signal, we must
content ourselves with giving the signals statistics. From our naive point of
view we can think of all noise signals as being pseudorandom or incompletely
170 NOISE
EXERCISES
5.2.1 Write a program to generate digital noise signals with probability distribution
(5.1). Estimate the probability distribution using 10, 100, 1000, and 10,000
samples. What is the error of the estimation?
5.2.2 Equation (5.6) for the variance require two passes through the signal values;
the first for computation of ~1 and the second for 0. Find a single-pass
algorithm.
5.2.3 Using the random number generator supplied with your compiler write a
zero-mean and unity variance noise generator. Make a histogram of the val-
ues it produces. Is it uniform? Calculate the empirical mean and standard
deviation. How close to the desired values are they?
5.2.4 Using the noise generator of the previous exercise, generate pairs of random
numbers and plot them as Z, y points in the plane. Do you see any patterns?
Try skipping L values between the 2 and y.
5.2.5 The noise generator you built above depends mainly on the most significant
bits of the standard random number generator. Write a noise generator that
depends on the least significant bits. Is this better or worse?
5.2.6 You are required to build the sample value histogram of a signal that only
takes on values in a limited range, based on N samples. If you use too few
bins you might miss relevant features, while too many bins will lead to a
noisy histogram. What is the right number of bins, assuming the probability
distribution is approximately flat? What is the error for 10,000 samples in
100 bins?
5.2.7 What are the average, variance, and standard deviation of a Gaussian signal?
What are the sufficient statistics? In what way is a Gaussian noise signal the
simplest type of noise?
5.3. NOISE REDUCTION BY AVERAGING 171
and our task is to recover 2, to the best of our abilities. The periodic-
ity (with period equal to the PRI) of the desired signal derives from the
supposition that the target is stationary or moving sufficiently slowly, and
it enables us to observe the same echo signal many times. For sufficiently
strong echoes we can simply isolate the echoes and measure the TOA for
each pulse transmitted. Then we need only subtract successive TOAs to find
the lag. However, this approach is not optimal, and doesnt work at all when
the echoes are hidden deep in the noise. We are thus led to seek a stronger
technique, one that exploits more knowledge regarding the noise.
The only quantitative statement made about the additive noise Un was
that it had zero mean. From one PRI to the next the desired signal xn re-
mains unchanged, but the received signal yn is seems completely different
from Xn, as depicted in Figure 5.5. Sometimes yn is greater than xn, but
(due to the noise having zero mean) just as frequently it will be less. Math-
ematically, using the linearity of the expectation operator, we can derive
(Yn) = (X7-h + V,) = (Xn) + (%) = Xn-
Figure 5.5: A pulsed radar signal contaminated by additive zero-mean noise. Note that
from pulse to pulse the noise is different, but the pulse shape stays the same. Thus the
uncontaminated signal can be reconstructed by pulse-to-pulse integration.
5.3. NOISE REDUCTION BY AVERAGING 173
Hence, although in general the observed yn is not the desired xn, its
average is. We can thus average the observed signals and obtain a much
cleaner estimate of xn. Such averaging over successive pulses is called radar
return integration. With each new pulse transmitted, the true echo signal
becomes stronger and stronger, while the noise cancels out and grows weaker
and weaker. Even if the echo was initially completely buried in the noise,
after sufficient averaging it will stand out clearly. Once detected, the lag
measurement can be made directly on the average signal.
A similar operation can be performed for all periodic phenomena. When
the desired underlying signal is periodic, each period observed supplies in-
dependent observations, and averaging increases the SNR. Another special
case is slowly varying signals. Assuming the additive noise to be white, or at
least containing significant spectral components at frequencies above those of
xn, we can average over adjacent values. The time domain interpretation of
this operation is clear-since xn varies more slowly than the noise, adjacent
values are close together and tend to reinforce, while the higher-frequency
noise tends to average out. The frequency domain interpretation is based
on recognizing the averaging as being equivalent to a low-pass filter, which
attenuates the high-frequency noise energy, while only minimally distorting
the low-frequency signal. So once again just the zero mean assumption is
sufficient to enable us to increase the SNR.
These averaging techniques can be understood using our naive theory,
but take on deeper meaning in the more sophisticated treatment of noise.
For example, we assumed that we could perform the averaging either in time
or over separate experiments. This seemingly innocent assumption is known
as the ergodic hypothesis and turns out to be completely nontrivial. We will
return to these issues in Section 5.6.
EXERCISES
5.3.1 Generate M random fl values and sum them up. The average answer will
obviously be zero, but what is the standard deviation? Repeat for several
different M and find the dependence on M.
5.3.2 In this exercise we will try to recover a constant signal corrupted by strong
additive noise. Choose a number x between -1 and +l. Generate M random
numbers uniformly distributed between -1 and +l and add them to the
chosen number, sn = x + u,. $70~ try to recover the chosen number by
averaging over M values 2 = Cn=i sn and observe the error of this procedure
z - 2. Perform this many times to determine the average error. How does
the average error depend on M?
174 NOISE
How well does this work compared to the previous exercise? Try Q = 0.001
and St = 0.1. What can you say about time averaging?
Given integers m, a, b, x
Precompute:
Q t mla
r + m-a*q
h-m-b
Loop :
x + a*(x-q*k)-wk-1
if x < 0 then x + x + m
By the way, if what you want is random bits then its not a good idea to
generate random integers and extract the LSB. This is because a sequence
of integers can appear quite random, even when its LSB is considerably
less so. Luckily there are good methods for directly generating random bits.
The most popular is the Linear Feedback Shift Register (LFSR), which
is somewhat similar to linear recursion. A shift register is a collection of
bits that can be shifted one bit to the right, thus outputting and discarding
the LSB and making room for a new MSB. Linear feedback means that
the new bit to be input is built by xoring together some of the bits in the
shift register. Starting off with some bits in the shift register, we generate
a sequence of bits by shifting to the right one bit at a time. Since the state
of the shift register uniquely determines the future of the sequence, the
504. PSEUDORANDOM SIGNALS 177
sequence eventually become periodic. If the shift register ever has all zeros
it becomes stuck in this state, and so this must be avoided at all costs.
One of the first random number generators was suggested by John von
Neumann back in 1946. His method starts with some D digit integer. Squar-
ing this integer produces an integer with 20 digits from which the next
integer in the sequence is obtained by extracting the middle D digits. This
recursion produces a periodic sequence of D digit integers, but this sequence
will be considerably less random than one generated by a properly selected
linear recursion generator.
Another random number generator does not require a multiplication, but
does need more memory
distributed noise. The first relies on the law of large numbers (see Ap-
pendix A.13) that states that the sum of a large number of independent
random numbers, whatever their original distribution, will tend to be Gaus-
sianly distributed. To exploit this law requires generating and adding N
(even 12 is often considered large enough) uniform random numbers. Of
course the maximum value that can be obtained is N times the maximum
value of the uniform generator, so in reality the Gaussian is somewhat trun-
cated, but the true distribution is extremely small there anyway. Often of
more concern is the computational burden of computing N uniform random
numbers per Gaussian random required.
The second method commonly used to generate Gaussianly distributed
numbers, sometimes called the Box-Muller algorithm after its inventors, is
best understood in steps. First pick at random a point inside the unit circle,
x:+iy=re ie . If we selected the point such that x and y are independent
(other than the constraint that the point be inside the circle) then r and
0 will be as well. Now 8 is uniformly distributed between 0 and 27r; how
is r distributed? It is obvious that larger radii are more probable since the
circumference increases with radius; in fact it is quite obvious that the prob-
ability of having a radius between zero and r increases as r2. We now create
a new point in the plane u + iv, whose angle is 6 but with radius p that
obeys r2 = e -P2j2 . The probability of such a point having radius less than R
is the same as the probability that the original squared radius r2 is greater
than e -R2/2. From this it follows that u and v are Gaussianly distributed.
How do we select a point inside a circle with all points being equally
probable? The easiest way is to randomly pick a point inside the square that
circumscribes the unit circle, and to discard points outside the circle. Picking
a point inside a square involves independently generating two uniformly
distributed random numbers x and y. Since u and v are also independent,
for every two uniform random numbers that correspond to a point inside
the circle we can compute two Gaussianly distributed ones.
Thus we arrive at the following efficient algorithm:
EXERCISES
5.4.1 Not only isnt there such a thing as a random number, there really is no such
thing as a random sequence of finite length. For example, all sequences of ten
digits are equally probable, namely one chance in lOlo. Yet we feel viscerally
that sequences such as { 1, 1, 1, 1, 1, 1, 1, 1, 1) or { 1,2,3,4,5,6,7,&g} are less
random than say { 1,9,3,6,3,4,5,8,2}. Can you explain this feeling?
5.4.2 You can test a random function using the following graphical test. Gener-
ate successive values ~1, r2, . . . and make a scatter plot consisting of points
(rk, rk- 1). If the resulting picture has structure (e.g., noticeable lines) the
random sequence has short-term correlations. If the plot looks reasonably
homogeneous repeat the procedure but plot (rk, ?-k-m) instead. Test the in-
teger recursions (equation (5.11)) defined by a=lO, b=5, m=50; a=15625,
b=O, m=65536; and the generator supplied with your programming environ-
ment .
5.4.3 Take inferior random generators from the previous exercise and whiten them
using the algorithm given in the text. Perform the graphical test once again.
5.4.4 Code a Gaussian noise generator based on the law of large numbers and check
its distribution.
5.4.5 Some people use this algorithm to generate Gaussianly distributed numbers:
generate two uniform random numbers, z and y, between 0 and +l
a=&BiZi,q6=2~y
u + asin and v +- aces(#)
Is this algorithm correct? What are the advantages and disadvantages relative
to the algorithm given in the text?
5.4.6 Other people use the following algorithm:
generate two uniform random numbers, x and y, between 0 and +l
u = j/Xsin(2ny)
v = ~xiiEcos(2Ty)
Show that this method is mathematically equivalent to the method given
in the text. In addition to requiring calls to sine and cosine functions, this
method is numerically inferior to the one given in the text. Why?
5.4.7 Complete the proof of the second algorithm for generating Gaussianly dis-
tributed random numbers.
5.4.8 How can we generate random numbers with an arbitrary distribution given
a uniform generator?
5.4.9 Show that after an initial transient LFSR sequences are always periodic.
What is the maximal period of the sequence from a shift register of length
K? Find a maximal length LFSR sequence of length 15.
180 NOISE
which shifts the signal values decimal point to the right, and then removes
the integer part. The first few values starting with ~0 = n - 3 are
which seem to oscillate wildly over the unit interval. Had we chosen ~0
slightly different from x - 3, the deviation of the resulting xx;nfrom the
above values would exponentially increase; for example, with a difference of
10B5 all similarity is lost after only five iterations.
The weather prediction example is similar. It turns out that the equations
relating air pressure, temperature, wind velocity, etc. are highly nonlinear,
even for rather simplistic models of atmospheric conditions. Weather pre-
diction relies on running such models, with appropriate initial weather con-
ditions, on large computers and observing the resulting weather conditions.
The initial specification is rather coarsely defined, since only gross features
such as average air temperature and pressure are known. This specification
leads to specific predictions of the weather as a function of time. However,
slight changes in the specification of the initial weather conditions lead to
rather different predictions, the differences becoming more and more signif-
icant as time goes on. This is the reason that the weather can be predicted
well for the short term, but not weeks in advance. Lorenz, who discovered
the instability of weather prediction models in the early 196Os, called this
the butterfly effect; a butterfly flapping its wings in Peking will affect the
weather in New York a month later!
How can we hope to study such nonlinear equations? Isnt chaos by
definition incomprehensible and thus unresearchable? The trick is to study
routes to chaos; we start at values of parameters for which the nonlinear
equations are not chaotic, and then to vary the parameters in order to ap-
proach the chaotic region. Before entering the chaotic region, the output
signal, although increasingly bizarre, can be profitably investigated. In this
section we will study Feigenbaums route to chaos. This route is easy to
study since it occurs in a simple one-parameter setting, arguably the sim-
plest nonlinear equation possible. It also seems to model well many interest-
ing physical situations, including some of the examples mentioned above.
Well introduce Feigenbaums route with a simple example, that of fish in
a closed pond. Let us denote by x the present fish population divided by the
maximum possible population (thus 0 < x < 1). We observe the population
every day at the same hour, thus obtaining a digital signal xn. How does xn
vary with time? Assuming a constant food supply and a small initial number
182 NOISE
xn+1
Figure 5.6: The logistics recursion relates the new signal value zn+l to the old one xn
by a inverted parabola. As such it is the simplest nonlinear recursion relation. It can also
be used to approximate any recursion with a single smooth maximum.
but once the number of fish becomes large, we anticipate an opposite ten-
dency due to overpopulation causing insufficient food and space, and possi-
bly spread of disease. It makes sense to model this latter tendency by a 1 -xn
term, since this leads to pressure for population decrease that is negligible
for small populations, and increasingly significant as population increases.
Thus we predict
Xn+l = r G-t (1 - Xn) (5.13)
which is often called the logistics equation. This equation is quadratic (see
Figure 5.6) and thus nonlinear. It has a single free parameter r (which is
related to the amount we feed the fish daily), which obey 0 5 r 5 4 in order
for the signal x to remain in the required range 0 < x 5 1. Although a
nonlinear equation with one free parameter is not guaranteed to be chaotic,
we will see that there are values of r for which small changes in x0 will lead
to dramatic changes in xn for large n. This means that when we overfeed
there will be large unpredictable fluctuations in fish population from day to
day.
You may object to studying in depth an equation derived from such a
fishy example. In that case consider a socialistic economy wherein the state
wishes to close the socioeconomic gap between the poor and the wealthy. It
5.5. CHAOTIC SIGNALS 183
a-&+1= (l+,,(l-k))&
xn+1
--X n
Figure 5.7: Graphical method of finding fixed points of the logistics equation. From
bottom to top the inverted parabolas correspond to 7 = 0, 1,2,3,4. We see that for T < 1
the parabola intersects the identity line only at z = 0, while for larger T there is an
additional point of intersection.
45 line (Figure 5.7). It is easy to see that for the parameter region 0 < r 5 1
the only possible fixed point is ~0 = 0, but for r > 1 the new fixed point
p, appears. For r 2 1 the new fixed point p, is close to the old one (zero),
gradually moving away with increasing r.
So we have found that the steady state behavior of the recursion is
really very simple. For r < 1 we are underfeeding our fish, or the interest is
negative, and so our fish or money disappear. An example of this behavior
is displayed in Figure 5.8.A. For 1 < r < 3 the number or fish or amount of
money approaches a constant value as can be seen in Figure 5.8.B. However,
we are in for quite a shock when we plot the behavior of our fish or money
for r > 3 (Figures 5.8.C and 5&D)! In the first case the signal oscillates
and in the second it seems to fluctuate chaotically, with no possibility of
prediction. In the chaotic case starting at a slightly different initial point
produces a completely different signal after enough time has elapsed! We
dont yet understand these phenomena since we progressed along the route
to chaos too quickly, so lets backtrack and increase r more slowly.
The most important feature of the behavior of the signal for small r is
the existence of the fixed point. Fixed points, although perhaps interesting,
are not truly significant unless they are attractive. An attractive fixed point
is one that not only replicates itself under the recursion, but draws in neigh-
boring values as well. For r 5 1 the zero fixed point is attractive-no matter
where we start we rapidly approach x = 0; but for 1 < r < 3 the new fixed
5.5. CHAOTIC SIGNALS 185
Figure 5.8: Signal produced by recursionof the logisticsequation for different values of
r. In (A) we have T = 0.9 and the signal decaysto zero. In (B) we have T = 1.9 and the
signal approachesa constant value. (C) depicts r = 3.1 and the signal oscillatesbetween
two values. In (D) we have r = 4 with two slightly different initial states; the signal is
noise-likeand irreproducible.
Figure 5.9: Use of return maps to depict the dynamics of a simple recursion. Each
iteration starts on the 45 line, proceeds vertically until intersecting the recursion curve,
and then returns to the diagonal line. Here we see that after enough iterations we converge
on a fixed point, which is the intersection of the recursion curve with the diagonal line. In
(A) we have T = 0.5 and the only fixed point is zero, while in (B) we see convergence to
the nonzero fixed point p,-.
point). We will see shortly that for r > 3 even this fixed point ceases being
an attractor; if one starts exactly at it, one stays there, but if one strays
even slightly the recursion drives the signal away.
How can we mathematically determine if a fixed point p is an attractor?
The condition is that the absolute value of the derivative of the recursive
relation f must be less than unity at the fixed point.
This ensures that the distance from close points to the fixed point decreases
with each successive recursion. It is now easy to show that for r > r2 = 3 the
fixed point p, becomes unattractive; but what happens then? No new fixed
point can appear this time, since the reasoning that led to the discovery of p,
as the sole nonzero fixed point remains valid for all r! To see what happens
we return to the return map. In Figure 5.10.A we see that starting from some
initial point we approach a square, which translates to alternation between
two points. Once the signal reaches its steady state it simply oscillates back
and forth between these two values, as can be seen in Figures 5.10.B and
5.11. This dual-valued signal is the new attractor; unless we start with 20 =
O,l,l - + or f-l(1 - $) we eventually oscillate back and forth between two
values. As r increases the distance between the two values that make up this
attractor also increases.
So attractors can be more complex than simple fixed points. What hap-
pens when we increase r still further? You may have already guessed that
this two-valued attractor also eventually becomes unattractive (although if
5.5. CHAOTIC SIGNALS 187
Figure 5.10: Return map representation of the logistics equation for r > 3. In (A)
r = 3.2 and we seethat from an arbitrary initial state 20 we converge on a non fixed
point attractor closeto pr. The attractor contains two points, one on either side of pr. In
(B) T = 3.4 and we display only the long time behavior (steady state behavior after the
transient hasdied down).
A 0.5
OL t
B* 0 0.5
Figure 5.11: The signal resulting from recursion of the logisticsequation for T = 3.2. In
(A) we seethe steady state signal in the time domain. It oscillatesbetweenthe two values
that make up the attractor, which meansthat zn+l = f(zn) and xn+2 = f(zn+l) = zn.
In (B) we seethe samesignal in the frequency domain. The DC component represents
the nonzero averageof the two points. Sincethe signal oscillatesat the maximum possible
frequency, we have a spectral line at digital frequency i.
one starts at ezactly one of its points one stays trapped in it) and a new
more complex attractor is born. In this case, this happens at rg = 1 + fi
and the new attractor is composed of a cycle between four signal values, as
depicted in Figure 5.12. If we call these points al, a2, as, and ~4, the require-
ment is u2 = f(ar), a3 = f(c4, a4 = f(a3), and al = f(a4). Note that the
2-cycles al split up into our present al and ~3, while its u2 became our new
a2 and ~4. So the attractors components obey al < u3 < u2 < ~4, which
188 NOISE
Figure 5.12: The return map, signal, and spectrum for the steady state behavior when
T = 3.5. The attractor is a $-cycle.
0.
I.. . . . . . . . . . .
O-
0 ,,a 1,. 31. ,/a
Figure 5.13: The return map, signal, and spectrum for the steady state behavior when
T = 3.55. The attractor is a &cycle.
Figure 5.14: The return map, signal, and spectrum for the steady state behavior when
r = 3.5675. The attractor is a 16-cycle.
5.5. CHAOTIC SIGNALS 189
means that the closest together in time are the farthest in space and vice
versa. This induces a spectrum wherein an additional spectral line appears
at twice the period, or half the frequency of the previous line.
We saw that when r increased above rg each component of the 2-cycle
splits into two, just as the fixed point had earlier split. The same thing hap-
pens in turn for the 4cycle when r goes above r4 and an &cycle is born. The
critical feature is that at each stage all components of the present attrac-
tor become unattractive simultaneously, a phenomenon known as pitchfork
bifurcation. Due to the bifurcation, with increasing r we find 16-cycles, 32-
cycles, and all possible 2n-cycles. Examples of such cycles are depicted in
Figures 5.12 through 5.14. The rule of closest in time are farthest in space
continues to be obeyed, so that new spectral lines continue to appear at
harmonics of half the previous basic frequency. Eventually the lines are so
close together that the spectrum becomes white, and we have chaotic noise.
The transition from periodicity to chaos can best be envisioned by plot-
ting the attractors as a function of T, as in Figure 5.15. The transition points
rn as a function of n approach a limit
7?,-+00
so that the regions where these cycles exist become smaller and smaller. By
the time we reach roe we have finished all the 2n-cycles.
Figure 5.15: The attractors of the recursion as a function of T. Observe the zero attractor
for 0 < r < 1, the fixed point p,. for 1 < P < 3, the 2-cycle for 3 < T < 1 + 6, and the
2-cycles for 3 < f < roe. Certain odd cycle regions can also be clearly seen for r > Too.
At r = 4 chaos reigns.
190 NOISE
Figure 5.16: Non 2n-cycle attractors for r > roe. We present return maps for a 3-cycle
(T = 3,83), a 6-cycle that results from bifurcation of that 3-cycle (r = 3.847), a quite
different 6-cycle (T = 3.63), and a 5-cycle (T = 3.74).
What happens between here and r = 4? It turns out that every length
attractor is possible. For example, in Figure 5.16 we see 3-cycles, 5-cycles
and 6-cycles. There is a theorem due to Sarkovskii that states that the order
of first appearance of any given length is
so that once a 3-cycle has been found we can be certain that all cycle lengths
have already appeared.
For roe < r < 4 there are other types of behavior as well. Let us start at
r = 4 where all possible x values seem to appear chaotically and decrease
r this time. At first x seems to occupy the entire region between i and
r (1 - $) , but below a certain ri this band divides into two broad subbands.
The signal always oscillates back and forth between the two subbands, but
where it falls in each subband is unpredictable. Decreasing r further leads
us past ri where each subband simultaneously splits into two somewhat
narrower subbands. The order of jumping between these four subbands is
closest in time are farthest in space, but the exact location inside each sub-
band is chaotic. Decreasing further leads us to a cascade of r6 in between
which there are 2n chaotic subbands, a phenomenon known as reverse bi-
furcation. Interspersed between the reverse bifurcations are regions of truly
periodic behavior (such as the 3-, 5-, and 6-cycles we saw before). The rk
converge precisely onto ra where the reverse bifurcations meet the previous
bifurcations.
We have seen that the simplest possible nonlinear recursion generates
an impressive variety of periodic and chaotic signals; but although complex,
these signals are still deterministic. In the next section we will see what a
truly random signal is.
5.5. CHAOTIC SIGNALS 191
EXERCISES
5.5.1 What is the change of variables that converts the socialistic economy equation
into the fish pond one?
5.5.2 Write a simulator that graphically depicts the behavior of the signal generated
by the logistics equation (5.13). Vary r and by trying various starting points
identify the attractors in the different regions.
5.5.3 Write a program to plot the attractors as a function of r. For each r go
through the possible xc systematically and identify when periodic behavior
has been reached and plot all points in this attractor. Can you identify the
various regions discussed in the text?
5.5.4 Extend the simulator written above to display the spectrum as well. Repro-
duce the results given in the text.
5.5.5 Prove that equation (5.15) is indeed the criterion for attractiveness. Prove
that for r 5 1 zero is indeed an attractor. Prove that for 1 < p 5 3 the fixed
point p, is an attractor.
5.5.6 At r = 4 a change of variable
2 = $ (1 - cos 27re)
brings us to a variable 8, which is homogeneously distributed. Show that x
is distributed according to - .
&kj
5.5.7 Plot the signal and spectrum of the 3-, 5-, and 6-cycles.
5.5.8 Henon invented a number of area preserving two-dimensional chaotic signals,
for example,
Here the superscript r specifies the specific realization; for different r the
deterministic component is identical, but the noise realization is different.
Each specific realization is a signal in the normal sense of the word; although
it might not be possible to find an explicit equation that describes such
a signal, it nonetheless satisfies the requirements of signalhood, with the
exception of those we usually ignore in any case. While taking a specific r
results in a signal, taking a specific time t or r~ furnishes a random variable,
which is a function of r over the real numbers that can be described via its
probability distribution function. Only when both r and t or 72 are given do
we get a numeric value; thus a stochastic signal can be better described as
a function of two variables.
When describing a stochastic signal we usually specify its ensemble statis-
tics. For example, we can average over the ensemble for each time, thus ob-
taining an average as a function of time. More generally, for each time we
5.6. STOCHASTIC SIGNALS 193
f(t, s)ds = MN =
AD (5.17)
Prob(s 5 s(t) 5 s + ds) Prob(s 5 sn 5 s + ds)
Here, even for the digital case we assumed that s was unquantized, fn (s)ds
representing the probability that the signal value at time n will be between s
and s+ds. Note that unlike the statistics of a regular random variable, statis-
tics of stochastic signals are functions of time rather than simple numbers.
Only for the special case of stationary signals are these statistics constants
rather than time-dependent.
As for regular random variables, in addition to the density function f(s)
we can define the Cumulative Distribution Function (CDF)
and it is obvious that for every time instant the density is the derivative
of the cumulative distribution. These distribution functions are in practice
cumbersome to use and we usually prefer to use statistics such as the mean
and variance. These can be derived from the density or cumulative distri-
bution. For example, for analog signals the mean for all times is calculated
from the density by integration over all x values
(5.19)
o-Z(t)
= Srn (s(t)- mso) f(t, 4 ds (5.20)
-03
There are also other statistics for stochastic signals, which have no coun-
terpart for simple random variables. The simplest is the correlation between
the signal at different times
which for stationary signals is a function only of the time difference r = tl-t2
or m = n1 - 722.
The autocorrelation tells us how much the value of the signal at time tl
influences its value at time t2, and is so important that we will devote all
of Chapter 9 to its use. The single time variance is simply the autocor-
relation when the two time variables coincide a:(t) = Cs (t, t), and so for
stationary signals ai = Cs(0). We also sometimes use the autocovariance
K$l, t2) = (u s t1 > - m&1)) (@a) - 77GzN) and it is easy to show that
v,(h, t2) = G@l, t2) - m,(h) m&2)*
More generally we can have statistics that depend on three or more time
instants. Unlike single time statistics, which can be calculated separately for
each time, these multitime statistics require that we simultaneously see the
entire stochastic signal (i.e., the entire ensemble for all times). We often use
only the mean as a function of time and the correlation as a function of two
times. These are adequate when the probability density function for all times
is Gaussian distributed, since Gaussians are completely defined by their
mean and variance. For more general cases higher-order signal processing
must be invoked (see Section 9.12), and we define an infinite number of
moment functions
ms =
( tSs(t) >
which are simply numbers. A stochastic signal is said to be stationary to
order Ic if its moments up to order Ic obey
C(t1,t2)
= e 7T
Thus there is essentially only correlation between signal values separated
by about 7 in time; for much larger time differences the signal values are
5.6. STOCHASTIC SIGNALS 195
This means that for any two distinct times, no matter how close these times
are, there is no correlation at all between signal values of white noise.
For discrete time signals we define Markov signals of different orders.
A first-order Markov signal is one for which sn depends on s,-1 but not
directly on any previous value. A second-order Markov signal has the signal
value depending on the two previous values.
There is an important connection between white noise and Markov sig-
nals; A Markov signal sn can be generated by filtering white noise v,. We
cannot fully explain this result as our study of filters will only begin in
the next chapter, but the main idea can be easily understood. Signal val-
ues of white noise at different times can be independent because of the
high-frequency components in the noise spectrum. Filtering out these high
frequencies thus implies forcing the signal value at time n to depend on
those at previous instants. A particular type of low-pass filtering produces
precisely Markovian behavior.
Sn = O!Sn-1 + vn
s, = Xn + C hnWn-rn
m=O
sn = Asin
would remain zero, while the ensemble average becomes the time-dependent
ij sin(wn).
What does ergodicity really mean ? Simply that rather than acquiring
an ensemble of N signal generators we can use only a single generator but
restart our experiment N times. If the signal with all the possible different
initial times reproduces the entire ensemble of the stochastic signal, then the
signal is ergodic. Not only must all possible realizations be reproduced, they
must be reproduced the same number of times. When we thinking about it
this way, ergodicity is rather too strong a statement; no signal can really be
so random that a single realization completely samples all the possibilities
of the ensemble! The number of realizations generated by restarting the
experiment at all possible times equals the number of points on the real line,
while there are many more different functions of time! However, ergodicity
makes life so simple that we most often assume it anyway.
For ergodic signals we can redefine the correlations in terms of time
averages. For example, the autocorrelation becomes
EXERCISES
5.6.1 Consider a stationary signal that can only take the values 0 and 1. What is
the probability that the signal is nonzero at two times r apart? What is the
meaning of moments for this type of signal?
5.6.2 Derive the relation between autocovariance and autocorrelation V,(tl, t2) =
G@lJ2> - 774074~2).
5.6.3 Show that for white signals (for which all times are independent) the auto-
covariance is zero except for when tl = t2.
5.6.4 In the text we discussed the filtering of white noise, although white noise is
not a signal and thus we have never properly defined what it means to filter
it. Can you give a plausible meaning to the filtering of a stochastic signal?
5.6.5 Prove that a first-order Markov signal can be obtained by low-pass filtering
white noise. Assuming that sn is created from a noise signal u, by equa-
tion 5.24 with 1~1< 1, what is the probability distribution of sn given that
we already observed s,+i?
5.6.6 Show that the signal sn generated by the recursion sn = ars+r +cQs,-.~+v,
(where V, is white) is a second-order Markov signal.
198 NOISE
5.6.7 Given the Markov signals of equation (5.24) and the previous exercise, can
you recover the white noise signal u, ?. What can you learn from the expression
for vn?
5.6.8 What is the power (si) of the Markov signal of equation (5,24)? Why did we
require Ial < l? The special case CI = 1 is called the random walk or Wiener
signal. What happens here?
5.6.9 Pink noise is a term often used for a noise whose power spectrum decreases 3
dB per octave (doubling of frequency). What is the spectral densitys depen-
dence on frequency? How does the power per octave depend on frequency?
5.6.10 Blue noise is the opposite of pink, with power spectrum increasing 3 dB per
octave; red noise has a 6 dB drop per octave. How do these spectral densities
depend on frequency?
m=- 1 cN 4-b
N n=l
single realization, the estimated power spectrum thus obtained will be biased
and will have a standard deviation of about the same size as the value being
estimated. Increasing the size of the window of the STFT does reduce the
bias but doesnt reduce the variance at all!
It is informative to understand the reasons for these enigmas. The bias
problem is the less severe of the two and the easier one to understand. Sim-
ply stated, the bias comes from comparing two different entities. When we
use the STFT to estimate the energy at a given frequency, we are actu-
ally dividing the frequency axis into bins, each of width determined by the
number of signal points in the transform. The STFT estimated spectrum
averages together the true spectrums values for all frequencies in the bin.
Thus the STFT power spectrums value at some frequency f should not be
expected to precisely replicate the true spectrums value there. However, as
the number of points in the STFT becomes larger, the bins become smaller
and the difference between the two decreases. Another way of looking at this
is to think of the STFT as the FT of the original signal multiplied by the
data window. This will of course equal the desired FT convolved with the
FT of the window function. For any given window duration use of good win-
dow functions can help (see Section 13.4), but the fundamental uncertainty
remains. As the duration of the window increases the FT of the window
function approaches a delta function and the bias disappears.
The true problem is the variance of our estimator. The spectral vari-
ance, unlike the variance of the mean, does not decrease with increasing the
number of data points used. At first this seems puzzling but the reason (as
first realized by Tukey in the late 1940s) is quite simple. When we double
the size of the STFT we automatically double the number of frequency bins.
All the information in the new data goes toward providing more frequency
resolution and not toward improving the accuracy of the existing estimates.
In order to decrease the variance we must find a way to exploit more of the
signal without increasing the frequency resolution. Two such methods come
to mind.
Assume that we increase the number of input signal values by a factor
of M. Bartlett proposed performing M separate power spectra and aver-
aging the results rather than performing a single (M times larger) STFT.
This averaging is similar to the mean estimator discussed above, and re-
duces the estimators variance by a factor of AL Welch further improved
this method by overlapping the data (with 50% overlap being about ideal).
Of course performing multiple transforms rather than a single large trans-
form is somewhat less efficient if the FFT is being used, but this is a small
price to pay for the variance reduction. The second way to reduce the vari-
5.7. SPECTRUM OF RANDOM SIGNALS 201
ante does perform a single STFT but then sums adjacent bins to reduce the
resolution. This effectively smooths the estimated power spectrum resulting
in a similar variance reduction. We will delve further into these techniques
in Section 13.3.
Earlier we stated that a unique (nonrandom) power spectrum can be
defined. This was first done by Wiener and Khintchine based on the following
theorem.
Theorem: Wiener-Khintchine
The autocorrelation and the power spectral density are an FT pair. n
In Chapter 9 we will prove this theorem for the deterministic case (after
properly defining the autocorrelation for deterministic signals). Here we take
this theorem as the definition for the stationary stochastic case. The basic
idea behind the theorem is clear. If we are only interested in the square of
the spectrum then we should only have to look at second-order entities in
the time domain; and the autocorrelation is the most basic of these.
Basing ourselves on Wiener-Khintchine we can now compute power spec-
tra of noisy signals in a new way, due to Blackman and Tukey. Rather than
directly computing the signals FT and squaring, we calculate the autocor-
relation and then take the FT. All that we have seen above about bias and
variance still holds, but averaging the computed spectra still helps. Since we
can use the FFT here as well, the Blackman-Tukey technique is similar in
computational complexity to the more direct Bartlett and Welch methods.
EXERCISES
5.7.1 Generate a finite-duration digital signal consisting of a small number of si-
nusoids and create K realizations by adding zero-mean Gaussian noise of
variance g2. Compute the power spectrum in the following three ways. Com-
pute the FT of each of the realizations, average, and then square. Compute
the FT, square, and then average. Compute the autocorrelation from the
realizations and find the power spectrum from Wiener-Khintchine. Compare
your results and explain.
5.7.2 Generate a single long (a power of two is best) realization of a signal as
above. Compare power spectrum estimates using windows without overlap,
overlapping windows, and smoothing of a single long FFT.
202 NOISE
Prob( i, Zk = x ) = 1
4
although the convergence may, in practice, be very slow.
EXERCISES
5.8.1 Is the division by k required for the deterministic case? Code the algorithm
and check for a few polynomials and a sinusoid.
5.8.2 Add noise to the signals you used in the previous exercise and run the full
algorithm. How does the error in the zero location depend on the noise level?
5.9. PROBABILISTIC ALGORITHMS 203
EXERCISES
5.9.1 Compute 7r by Monte-Carlo determination of the area of the unit-radius
circle. Monitor the error as a function of the number of points generated.
How does the computation required to obtain a given accuracy compare with
that of direct numerical integration?
5.9.2 Find the volume of a unit-radius sphere and the hypervolume of a unit-radius
hypersphere in four dimensions. Make the same computational complexity
comparisons as in the previous exercise.
5.9.3 In certain cases deterministic and probabilistic approaches to integration can
be combined to obtain a faster and more accurate method. Explain the idea
and apply to the previous exercise. (Hint: Inscribe the circle with a second
square.)
Bibliographical Notes
Our treatment of noise has been very different, and a good deal less pedantic, than
that found in engineering textbooks. For those who miss the formalistic treatment
there are several good books on stochastic processes. The classic text is that of
Papoulis [1901; only slightly less classic but much less friendly is van Trees [264];
but any text that has the words stochastic and process in its title will probably
do. There are also texts with a major emphasis on stochastic processes that mix
in a certain amount of straight signal processing, e.g., [250], and others with the
opposite stress, such as [188].
Those interested in more information regarding radar systems can try anything
by Skolnik [245, 243, 2441 or the book by Levanon [145].
The generation of pseudorandom signals is discussed at length in the second
volume of Knuth [136]. The transformation from uniform to Gaussian distributed
random numbers (also found in Knuth) was discovered by Box and Muller [22]. The
standard text on shift register sequences is by Golomb [81].
Deterministic chaos is quite a popular subject, with many books, each with its
own approach. A suitable text for physicists is [234], while there are other books
suitable for engineers or for mathematicians. The popular account by Gleick [75] is
accessible and interesting.
Perhaps the earliest mathematical account of noise is [219,220], which presented
a complete theory including power spectra, statistical properties, and the effect of
nonlinear systems on noise. Many would claim that the most important book on
stochastic processes is that of Papoulis mentioned above [190].
An accessible source from which one can gain insight regarding the true meaning
of noise is to be found in much of what is called modern music.
Part II
Systems
The study of signals, their properties in time and frequency domains, their
fundamental mathematical and physical limitations, the design of signals
for specific purposes, and how to uncover a signals capabilities through
observation belong to signal analysis. We now turn to signal processing,
which requires adding a new concept, that of the signal processing system.
A signal processing system is a device that processes input signals and/or
produces output signals. Signal processing systems were once purely analog
devices. Older household radio receivers input analog radio frequency signals
from an antenna, amplify, filter, and extract the desired audio from them
using analog circuits, and then output analog audio to speakers. The original
telephone system consisted of analog telephone sets connected via copper
wire lines, with just the switching (dialing and connecting to the desired
party) discrete. Even complex radar and electronic warfare systems were
once purely analog in nature.
Recent advances in microelectronics have made DSP an attractive al-
ternative to analog signal processing. Digital signal processing systems are
employed in a large variety of applications where analog processing once
reigned, and of course newer purely digital applications such as modems,
speech synthesis and recognition, and biomedical electronics abound. There
still remain applications where analog signal processing systems prevail,
mainly applications for which present-day DSP processors are not yet fast
enough; yet the number of such applications is diminishing rapidly.
In this chapter we introduce systems analogously to our introduction of
signals in Chapter 2. First we define analog and digital signal processing
systems. Then we introduce the simplest possible systems, and important
classes of systems. This will lead us to the definition of a filter that will
become a central theme in our studies. Once the concept of filter is under-
stood we can learn about MA, AR, and combined ARMA filters. Finally we
consider the problem of system identijication which leads us to the concepts
of frequency response, impulse response, and transfer function.
207
208 SYSTEMS
EXERCISES
6.1.1 Which of the following are signal processingsystems (we shall use 2 for inputs
and y for outputs)? Explain.
1. The identity y = x
2. The constant y = k irrespective of x
3. y=+/z
4. A device that inputs a pizza and outputs a list of its ingredients
5. y = sin(+)
6. Y(t) = S4,z(t)
7. The Fourier transform
8. A television
9. A D/A converter
6.1.2 Given any two signals x(t) and y(t), is there always a system that inputs x
and outputs y? Given a system that inputs x(t) and outputs y(t), is there
always a system that inputs y and outputs x?
Figure 6.1: The effect of clipping amplifiers with different gains on sinusoidal signals. In
(A) there is no clipping, in (B) intermediate gain and clipping, while (C) represents the
infinite gain case (hard limiter).
6.2. THE SIMPLEST SYSTEMS 211
Hard limiting changes sinusoids into square waves, and is frequently em-
ployed to obtain precisely this effect.
These clipping amplifiers deal symmetrically with positive and negative
signal values; another form of nonlinearity arises when the sign explicitly
affects the output. For example, the gain of an amplifier can depend on
whether the signal is above or below zero. Extreme cases are the half-wave
rectifier, whose output is nonzero only for positive signal values,
for small E > 0. More generally higher powers may contribute as well. Real
amplifiers always deviate from linearity to some degree, and power law dis-
tortion is a prevalent approximation to their behavior. Another name for
power law distortion is harmonic generation; for example, quadratic power
212 SYSTEMS
Figure 6.2: Half-wave and full-wave rectifiers. (A) depicts the output as a function of
input of a half-wave rectifier, as well as its effect on a sinusoid. (B) depicts the same for
a full-wave rectifier.
invariant (i.e., the behavior of the system does not change with time). Clas-
sical mathematical analysis and most non-DSP numerical computation deal
almost exclusively with memoryless systems, while DSP almost universally
requires combining values of the input signal at many different times. Time
invariance, the norm outside DSP, is common in many DSP systems as well.
However, certain important DSP systems do change as time goes on, and
may even change in response to the input. We will see such systems mainly
in Chapter 10.
EXERCISES
6.2.1 The digital amplifier is a linear system as long as no overflow or underflow
occur. What is the effect of each of these computational problems? Which is
worse? Can anything be done to prevent these problems?
6.2.2 Logarithmic companding laws are often used on speech signals to be quan-
tized in order to reduce the required dynamic range. In North America the
standard is called p-law and is given by
y _- sgn(x)wl + /44> (6.6)
logO+ CL)
where x is assumed to be between -1 and +l and p = 255. In Europe A-law
is used
sgn(~~~~ 0 < 1x1< a
Y=
sgn(x),w
!
+ < 1x1< 1
(6.7)
where A = 87.6. Why are logarithmic curves used? How much difference is
there between the two curves?
6.2.3 Time-independent point transformations can nontrivially modify a signals
spectrum. What does squaring signal values do to the spectrum of a pure
sinusoid? To the sum of two sinusoids? If point operations can modify a
signals spectrum why do you think systems with memory are needed?
The other line of thought, called the state-space description, considers the
output to be calculated based on the present input and the present internal
state of the system.
Yn = f CZ,, S) (6.9)
In the state-space description the effec; of tii input on the system is twofold,
it causes an output to be generated and it changes the state of the system.
These two ways of thinking are clearly compatible, since we could always
define the internal state to contain precisely the previous inputs and outputs.
This is even the best way of defining the systems state for systems that
explicitly remember these values. However, many systems do not actually
remember this history; rather this history influences their behavior.
The simplest system with memory is the simple delay
where the time r or m is called the lug. From the signal processing point
of view the simple delay is only slightly less trivial than the identity. The
delays output still depends on the input at only one time, that time just
happens not to be the present time, rather the present time minus the lag.
We have said that the use of delays is one of the criteria for contrasting
simple numeric processing from signal processing. Recall from Chapter 2
that what makes signal processing special is the schizophrenic jumping back
and forth between the time domain and the frequency domain. It is thus
natural to inquire what the simple delay does to the frequency domain rep-
resentation of signals upon which it operates. One way to specify what any
signal processing system does in the frequency domain is to input simple
sinusoids of all frequencies of interest and observe the systems output for
each. For the simple delay, when a sinusoid of amplitude A and frequency
w is input, a sinusoid of identical amplitude and frequency is output. We
will see later on that a system that does not change the frequency of sinu-
soids and does not create new frequencies is called a filter. A filter that does
not change the amplitude of arbitrary sinusoids, that is, one that passes all
frequencies without attenuation or amplification, is called an all-puss filter.
Thus the simple delay is an all-pass filter. Although an all-pass filter leaves
the power spectrum unchanged, this does not imply that the spectrum re-
mains unchanged. For the case of the delay it is obvious that the phase of
the output sinusoid will usually be different from that of the input. Only if
the lag is precisely a whole number of periods will the phase shift be zero;
otherwise the phase may be shifted either positively or negatively.
6.3. THE SIMPLEST SYSTEMS WITH MEMORY 215
B
Xn+ Xn-1 + Xn-2 -C- * l l -b- Xn-m-j-2 -b- Xn-m+l -b- Xn-m
Figure 6.3: Implementation of the simple delay. In (A) we see how an analog delay of
lag r can be obtained by allowing a voltage of current signal to travel at finite velocity
through a sufficiently long delay line. In (B) a digital delay of lag m is implemented using
a FIFO buffer.
stored in the FIFO to be output and promptly discarded, for all the other
values to move over, and for the present input to be placed in the buffer. Of
course long delays will require large amounts of memory, but memory tends
to drop in price with time, making DSP more and more attractive vis-a-vis
analog processing. DSP does tend to break down at high frequencies, which
is exactly where analog delay lines become practical.
Leaving the simple delay, we now introduce a somewhat more complex
system. Think back to the last time you were in a large empty room (or a
tunnel or cave) where there were strong echoes. Whenever you called out
you heard your voice again after a delay (that we will call 7) , which was
basically the time it took for your voice to reach the wall from which it
was reflected and return. If you tried singing or whistling a steady tone you
would notice that some tones resonate and seem very strong, while others
seem to be absorbed. We are going to model such a room by a system whose
output depends on the input at two different times, the present time and
some previous time t - r. Our simple echo system adds the signal values
at the two times
IH(
-6 -4 -2 0 2 4 6
Figure 6.4: Amplitude of the frequency response for the echo system with positive co-
efficients. The amplitude is plotted as a function of WT. The coefficients are h = 0 (the
straight line), 0.25,0.5,0.75, and 1.0 (the plot with zeros).
6.3. THE SIMPLEST SYSTEMS WITH MEMORY 219
l=O
This system goes under many different names, including Moving Average
(MA) filter, FIR filter, and alZ-xero filter, the reasoning behind all of which
will be elucidated in due course. The mathematical operation of summing
over products of indexed terms with one index advancing and one retreating
is called convolution.
Now this system may seem awesome at first, but its really quite simple.
It is of course linear (this you can check by multiplying x by a constant,
and by adding xr + x2). If the input signal is a pure sine then the output
is a pure sine of the same frequency! Using linearity we conclude that if
the input signal is the sum of sinusoids of certain frequencies, the output
contains only these same frequencies. Although certain frequencies may be
zeros of the frequency response, no new frequencies are ever created. In this
way this system is simpler than the nonlinear point transformations we saw
in the previous section. Although limited, the FIR filter will turn out to be
one of the most useful tools in DSP.
What should be our next step in our quest for ever-more-complex digital
signal processing systems? Consider what happens if echoes from the distant
past are still heard-we end up with a nonterminating convolution!
Yn = c hl X72-1 (6.14)
l=-00
In a real concert hall or cave the gain coefficients hl get smaller and smaller
for large enough 1, so that the signal becomes imperceptible after a while.
When an amplifier is involved the echoes can remain finite, and if they are
timed just right they can all add up and the signal can become extremely
strong. This is what happens when a microphone connected to an amplifier
is pointed in the direction of the loudspeaker. The squeal frequency de-
pends on the time it takes for the sound to travel from the speaker to the
microphone (through the air) and back (through the wires).
The FIR filter owes its strength to the idea of iteration, looping on all
input signal values from the present time back to some previous time.
Yn = 0
for i=O to L
Yn + Yn + hlxn-1
220 SYSTEMS
and the IIR filter exploits this by allowing yn to be a weighted sum of all
previous outputs as well as inputs.
We see here two convolution sums, one on the inputs and one on the (previ-
ous) outputs. Although even this system cannot create sinusoids of frequen-
cies that do not exist in the input at all, it can magnify out of all proportion
components that barely exist (see exercises). Of course even IIR filters are
simple in a sense since the coefficients al and bm do not vary with time.
More complex systems may have coefficients that depend on time, on other
signals, and even on the input signal itself. We will see examples of such
systems when we discuss adaptive filters.
Are these time- and signal-dependent systems the most complex systems
DSP has to offer? All I can say is I hope not.
EXERCISES
6.3.1 Prove the following characteristics of the convolution.
6.3.2 We saw that a generalized echo system y(t) = z(t) + hz(t -7) has no zeros in
its frequency response for lhl < 1; i.e., there are no sinusoids that are exactly
canceled out. Are there signals that are canceled out by this system?
6.3.3 Find the frequency response (both amplitude and phase) for the generalized
echo system. Use the trigonometric identity for the sine of a sum, and then
convert a sin(&) +b cos(wt) to A sin(wt+$). Check that you regain the known
result for h = 1. Show that the amplitude is indeed between 1 - 1hl and 1 + 1hi.
6.3.4 Plot the amplitude found in the previous exercise for positive coefficients and
check that Figure 6.4 is reproduced. Now plot for negative h. Explain. Plot
the phase found in the previous exercise. Is the system always linear-phase?
6.4. CHARACTERISTICSOF SYSTEMS 221
6.3.5 The digital generalized echo system yn = zn + hxnem can only implement
an echo whose delay is an integer number of sample intervals t,. How can a
fractional sample delay echo be accommodated?
6.3.6 Show that an IIR filter can blow up, that is, increase without limit even
with constant input.
6.3.7 Show that the IIR filter
when triggered with a unit impulse xn = Sn,fj can sustain a sinusoid. What
is its frequency?
6.3.8 The sound made by a plucked guitar string is almost periodic, but starts
loud and dies out with time. This is similar to what we would get at the
output of an IIR system with a delayed and attenuated echo of the output
yn = xn + gyn-m with 0 < g < 1. What is the frequency response of this
system? (Hint: It is easier to use xn = eiwn for n > 0 and zero for n < 0,
rather than a real sinusoid.)
6.3.9 All the systems with memory we have seen have been causc~l,that is, the
output at time T depends on the input at previous times t 5 T. What can
you say about the output of a causal system when the input is a unit impulse
at time zero? Why are causal systems sensible? One of the advantages of DSP
over analog signal processing is the possibility of implementing noncausal
systems. How (and when) can this be done?
Here H is a system that converts one signal into another, not merely a
function that changes numbers into numbers.
A memoryless system is called invertible if distinct input values lead to
distinct output values. The system yn = 2x, is thus invertible since every
222 SYSTEMS
finite value of xn leads to a unique yn. Such systems are called invertible
since one can produce an inverse system H-l such that xn = H-y,. For
the system just mentioned it is obvious that xn = iy,. Since
Yn=Xn-G-1
Xn = Yn + (yn-1 + Xns2)
and assuming that the input signal was zero for n = 0 we get an infinite
sum.
00
Xn = (6.19)
c Yi
i=o
Inverse systems are often needed when signals are distorted by a system
and we are called upon to counteract this distortion. Such an inverse system
is called an equalizer. An equalizer with which you may be familiar is the
adjustable or preset equalizer for high-fidelity music systems. In order to
reproduce the original music as accurately as possible, we need to cancel
out distortions introduced by the recording process as well as resonances
introduced by room acoustics. This is accomplished by dividing the audio
spectrum into a small number of bands, the amplification of which can be
individually adjusted. Another equalizer you may use a great deal, but with-
out realizing it, is the equalizer in a modem. Phone lines terribly distort data
signals and without equalization data transmission speeds would be around
6.4. CHARACTERISTICS OF SYSTEMS 223
is linear, but when presented with a constant input signal the output grows
(linearly) without limit.
We generally wish to avoid instability as much as possible, although the
above definition is somewhat constraining. Systems with sudden singularities
or exponentially increasing outputs should be avoided at all costs; but milder
divergences are not as damaging. In any case true analog systems are always
stable (since real power supplies can only generate voltages up to a certain
level), and digital systems can not support signal values larger than the
maximum representable number. The problem with this compelled stability
is that it comes at the expense of nonlinearity.
224 SYSTEMS
A time-variant system has some internal clock that influences its behavior.
For example,
Yn = (1 + sin(nt)) Xn
Definition: filter
A filter is a system H with a single input and single output signal that is
both linear (obeys (6.20)) and time-invariant (obeys equation (6.21)). n
As usual we often deviate from the precise definition and speak of nonlin-
ear jilters, time-variant filters, and multidimensional filters, but when used
without such qualifications the term filter will be taken to be equivalent to
LTI.
We already know about systems with memory and without. The output
value of a system without memory depends only on the input value at the
same time. Two weaker characteristics that restrict the time dependence of
the output are causality and streamability. A system is termed causal if the
output signal value at time T is only dependent on the input signal values for
that time and previous times t 5 T. It is obvious that a memoryless system
is always causal, and it is easy to show that a filter is causal if and only if
a unit impulse input produces zero output for all negative times. Noncausal
systems seem somewhat unreasonable, or at least necessitate time travel,
since they require the system to correctly guess what the input signal is
going to do at some future time. The philosophical aspects of this dubious
behavior are explored in an exercise below. Streamable systems are either
causal or can be made causal by adding an overall delay. For example, neither
yn = xmn nor gn = x,+1 are causal, but the latter is streamable while the
former is not.
6.4. CHARACTERISTICSOF SYSTEMS 225
When working off-line, for instance with an input signal that is available
as a file or known as an explicit function, one can easily implement noncausal
systems. One need only peek ahead or precompute the needed input values,
and then place the output value in the proper memory or file location. Ana-
log systems can realize only causal systems since they must output values
immediately without peeking forward in time, or going back in time to cor-
rect the output values. Since analog systems are also required to be stable,
stable causal systems are called realizable, meaning simply that they may be
built in analog electronics. Real-time digital systems can realize only stable
streamable systems; the amount of delay allowed is application dependent,
but the real-time constraint requires the required delay to be constant.
EXERCISES
6.4.1 Find the inverse system for the following systems. If this is in IIR form find
the FIR form as well (take xn = 0 for n 5 0).
1. yn = 5, +&p-1
2. yn = xn - ix,-1
3. yn = x, -x,-1 -x,-2
5. yn= Xn + Yn-1
6. yn =Xn - &a-l+ Yn-1
6.4.2 What can you say about the FIR and IIR characteristics of inverse systems?
6.4.3 Which of the following systems are filters? Explain. 1. yn = x,-r + k
2. Yn = Xn+lXn-1
3. Yn = Xln
4. yn=O
5. Yn = Yn-1
6.4.4 Show that a filter is causal if and only if its output, when the input is a unit
impulse centered on time zero, is nonzero only for positive times.
6.4.5 Show that if two signals are identical up to time t, then the output of a causal
system to which these are input will be the same up to time t.
6.4.6 Show that the smoothing operation yn = %(xn + xn-r) is causal while the
similar yn = $(xn+l + Xn) is not. The system yn = $(xn+i - xn-1) is an
approximation to the derivative. Show that it is not causal but is streamable.
Find a causal approximation for the derivative.
6.4.7 Consider the philosophical repercussions of noncausal systems by reflecting
on the following case. The system in question outputs -1 for two seconds if
its input will be positive one second from now, but fl for two secondsif its
input will be negative. Now feed the output of the system back to its input.
226 SYSTEMS
6.4.8 Explain why streamable systems can be realized in DSP but not in analog
electronics. What does the delay do to the phase response?
6.4.9 The systems yn = xn + a (which adds a DC term) and yn = xi (which
squares its input) do not commute. Show that any two filters do commute.
6.4.10 Systems do not have to be deterministic. The Modulated Noise Reference
Unit (MNRU) system, defined by yn = (1 + lo-%v~) x, (where v is wide-
band noise) models audio quality degradation under logarithmic companding
(exercise 6.2.2). Which of the characteristics defined in this section does the
MNRU have? Can you explain how the MNRU works?
6.5 Filters
In the previous section we mentioned that the combination of linearity and
time invariance is important enough to deserve a distinctive name, but did
not explain why this is so. The explanation is singularly DSP, linking char-
acteristics in the time domain with a simple frequency domain interpreta-
tion. We shall show shortly that the spectrum of a filters output signal is
the input signals spectrum multiplied by a frequency-dependent weighting
function. This means that some frequencies may be amplified, while oth-
ers may be attenuated or even removed; the amplification as a function of
frequency being determined by the particular filter being used. For exam-
ple, an ideal low-pass filter takes the input signal spectrum, multiplies all
frequency components below a cutoff frequency by unity, but multiplies all
frequency components over that frequency by zero. It thus passes low fre-
quencies while removing all high-frequency components. A band-pass filter
may zero out all frequency components of the input signal except those in
a range of frequencies that are passed unchanged.
Only filters (LTI systems) can be given such simple frequency domain
interpretations. Systems that are not linear and time-invariant can create
new frequency components where none existed in the input signal. For ex-
ample, we mentioned at the end of Section 6.2 and saw in exercise 6.2.3 that
the squaring operation generated harmonics when a sinusoidal signal was
input, and generated combination frequencies when presented with the sum
of two sinusoids. This is a general feature of non-LTI systems; the spectrum
of the output will have frequency components that arise from complex com-
binations of input frequency components. Just 51sthe light emerging from a
optical filter does not contain colors lacking in the light impinging upon it,
6.5. FILTERS 227
just as when pouring water into a coffee filter brandy never emerges, just so
you can be sure that the output of a signal processing filter does not contain
frequencies absent in the input.
Lets prove this important characteristic of filters. First, we expand the
input in the SUI basis (as the sum of unit impulses weighted by the signal
value at that time).
Xn = fJ Xmb,m
m=-oo
Next, using the linearity of the filter H, we can show that
but since the x, are simply constants multiplying the SUIs, linearity also
implies that we can move them outside the system operator.
Co
Yn = c xrn H&m
77-L=-M
Now the time has come to exploit the time invariance. The operation of the
system on the SUI HS,,, is precisely the same as its operation on the unit
impulse at time zero, only shifted m time units. The impulse response hn is
defined to be the response of a system at time n to the unit impulse.
hn = H&a,0 (6.22)
For causal systems hn = 0 for n < 0, and for practical systems hn must
become small for large enough n. So time invariance means HS,,, = h,-,
and we have found the following expression for a filters output.
00
Yn = c xmhn-m (6.23)
m=-00
which states that the output at frequency k is the input at that frequency
multiplied by a frequency-dependent factor Hk. This factor is the digital
version of what we previously called the frequency response H(w).
Using terminology borrowed from linear algebra, what we have proved
is that the sinusoids are eigenfunctions or, using more fitting terminology,
eigensignals of filters. If we allow complex signals we can prove the same for
the complex sinusoids xn = eiwn.
EXERCISES
6.5.1 In exercise 6.3.7 we saw that the system yn = xn - aiyn-1 - yn-2 could
sustain a sinusoidal oscillation even with no input. Yet this is a filter, and
thus should not be able to create frequencies not in the input! Explain.
6.5.2 Show that the system yn = xn+x+ is not a filter. Show that it indeed doesnt
act as a filter by considering the inputs xn = sin(wn+$) and xn = cos(wn+$).
6.5.3 Prove the general result that zn are eigenfunctions of filters.
6.5.4 Prove the filter property for analog signals and filters.
where vn is some noise signal. We know nothing about the noise save that
its average is zero, and that its variance is finite.
6.6. MOVING AVERAGES IN THE TIME DOMAIN 229
Since the noise averages to zero and the observed signal is the sum of the
desired constant signal and this noise, the observed signals average value
must be x. Our path is clear; we need to average the observed signal
-J, L-l
Xl = x (6.26)
r, c
1=0
with the sum approaching z more and more closely as we increase L. For
finite L our estimate of x will be not be exact, but for large enough L (the
required size depending on the noise variance) we will be close enough.
Now let us assume that xn is not a constant, but a slowly varying signal.
By slowly varying we mean that z, is essentially the same for a great many
consecutive samples. Once again we can only observe the noisy xn, and are
interested in recovering 2,. We still need to average somehow, but we can
no longer average as much as we please, since we will start blurring the
desired nonconstant signal. We thus must be content with averaging over
several xn values,
1 L-l
Xn+l M zn Yn =- (6.27)
L c1=0
and repeating this operation every j samples in order to track xn.
Yo Yj
P 9 7* * * XL-17 XL7 XL+17 . . . wXzj+l,+L, Xj+L+l, . **
Y2
Yl
230 SYSTEMS
Well, we can; this type of averaging is called a moving average, which is often
abbreviated MA. The moving average operation produces a new signal V.J~
which is an approximation to the original xn. Upon closer inspection we
discover that we have introduced a delay of $ in our estimates of 2,. We
could avoid this by using
1 L
Yn = 2~ + 1 lzwLxn+l
c c Xn (6.28)
The latter more strongly emphasizes the center term, de-emphasizing the
influence of inputs from different times. Similarly we can define longer mov-
ing averages with coefficients becoming smaller as we move away from the
middle (zero) terms.
The most general moving average (MA) filter is
L
Yn = c h XTl+l (6.29)
1=-L
although L here will need to be about twice as large, and the output yn will
be delayed with respect to zn.
6.7. MOVING AVERAGES IN THE FREQUENCY DOMAIN 231
EXERCISES
6.6.1 Experiment with the ideas presented in this section as practical techniques for
removing noise from a signal. Start with a signal that is constant 1 to which
a small amount of Gaussian white noise has been added xn = 1 + EU~. Try
to estimate the constant by adding iV consecutive signal values and dividing
by N. How does the estimation error depend on E and N?
6.6.2 Perform the same experiment again only this time take the clean signal to be
a sinusoid rather than a constant. Attempt to reconstruct the original signal
from the noisy copy by using a noncausal moving average with all coefficients
equal. What happens when the MA filter is too short or too long?
6.6.3 Now use an MA filter with different coefficients. Take the center coefficient
(that which multiplies the present signal value) to be maximal and the others
to decrease linearly. Thus for length-three use (a, f , a>, for length-five use
i(l, 2,3,2, l), etc. Does this perform better?
6.6.4 Find a noncausal MA differentiator filter, that is, one that approximates
the signals derivative rather than its value. How are this filters coefficients
different from those of the others we have discussed?
6.6.5 A parabola in digital time is defined by p(n) = on2 + bn + c. Given any three
signal values x _ 1, xc, x+1 there is a unique parabola that goes through these
points. Given five values x-2, x-1, x0, x+1, x+2 we can find coefficients a, b
and c of the best fitting parabola p(n), that parabola for which the squared
error c2 = (p(-2) -x-2) + (p(-1) - 2-1)~ + (p(0) -x~)~ + (p(+l) - x+~)~ +
(p(+2) - x+Z)~ is minimized. We can use this best fitting parabola as a
MA smoothing filter, for each n we find the best fitting parabola for the 5
signal values xn-2, x ,+.I, xn, xn+r, xn+2 and output the center value of this
parabola. Show that the five-point parabola smoothing filter is an MA filter.
What are its coefficients?
6.6.6 After finding the best fitting parabola we can output the value of its derivative
at the center. Find the coefficients of this five-point differentiator filter.
Yn = (Xn-1
3 + Xn + &+I) (6.31)
and since the moving average is a filter, we know that the output will be a
complex exponential of the same frequency.
Yn = H(w)eiwn
Substituting
eiw(n-l) + ,iwn + eiw(nfl) = i ,-iw + 1 + ,iw ,iwn
Yn = -i
( > ( >
we immediately identify
IH(
Figure 6.5: The (squared) frequency response of the simple three-point average filter.
The response is clearly that of a low-pass filter, but not an ideal one.
IH(
Figure 6.6: The (squared) frequency responses of two simple three-point average filters.
Both responses are clearly low-pass but not ideal. The average with coefficients goes to
zero at $ ff, but is wider than the simple average.
IH(
Figure 6.7: The (squared) frequency responses of simple averaging filters for L = 3,5,7
and 9. We see that as L increases the pass-band becomes narrower, but oscillations con-
tinue.
6.7. MOVING AVERAGESIN THE FREQUENCYDOMAIN 235
IH(o>12
Figure 6.8: The (squared) frequency responses of a 16-coefficient low-pass filter. With
these coefficients the lower frequency components are passed essentially unattenuated,
while the higher components are strongly attenuated.
the frequency response of which is depicted in Figure 6.8. While some os-
cillation exists in both the pass-band and the stop-band, these coefficients
perform the desired task relatively well.
Similarly we could find coefficients that attenuate low frequencies but
pass high ones, or pass only in a certain range, etc. For example, another
simple MA filter can be built up from the finite difference.
IH(ci$
Figure 6.9: The (squared) frequency response of a finite difference filter. With these
coefficients the lower frequency components are passed essentially unattenuated, while
the higher components are strongly attenuated.
It is easy to show that its frequency response (seeFigure 6.9) attenuates low
and amplifies high frequencies.
EXERCISES
6.7.1 Calculate the frequency response for the simple causal moving average.
L-l
1
Yn =-
c %x-k
L k=O
6.7.7 The secondfinite difference A2 is the finite difference of the finite difference,
i.e., A2xn = A(x, - x,+1) = xn - 2x,- 1 + x,+2. Give explicit formulas
for the third and fourth finite differences. Generalize your results to the kth
order finite difference. Prove that yn = Akxn is an MA filter with k + 1
coefficients.
A(z) = a0 + al x + a2 x2
B(x) = bo + bl x + b2x2
238 SYSTEMS
A(Z) = caixi
i=o
B(x) = Ebjxj
j=O
we have
Xk
N+M
= C (U * b)k Xk
k=O
and the fundamental reason for these indices to run in opposite directions
is obvious-the two exponents must sum to a constant!
Put this way the idea of indices running in opposite directions isnt so
new after all. In fact you probably first came across it in grade school.
Remember that an integer is represented in base- 10 as a polynomial in 10,
A = Cf/, uilOi (where dA is the number of digits). Thus multiplication of
two integers A and B is also really a convolution.
d/i+&
AB = c (a * b)klok
k=O
AN AN-~ a** Al Ao
* BN BN-1 **a B1 BO
BOAN BoAN-~ * ** BoAl BoAo
&AN &AN-I . . . &A0
2,-X(d)=x&n
n
Yo = hoxo + helxl
Yl = helxO + hoxl + hw1x2
Y2 = hmlxl + hOx2 + he1x3
Y3 = hm1x2 + h0x3 + hs1x4
Y4 = h-lx3 + hox4
240 SYSTEMS
The matrix has quite a distinctive form, all elements on each diagonal being
equal. Such a matrix is said to be Toeplitx in structure, and Toeplitz matrices
tend to appear quite a lot in DSP.
EXERCISES
6.8.1 N.G. Kneer, the chief DSP hardware engineer at NeverWorks Incorporated,
purchases pre-owned DSP processors on an as-is basis from two suppliers,
Alpha Numerics and Beta Million. From Alpha Numerics one gets a perfect
lot 30% of the time, but 10% of the time all five chips are bad. Prom Beta one
never gets a completely bad lot, but only gets a perfect lot 10% of the time.
In fact, N.G. has come up with the following data regarding the probability
of k defective chips out of a lot of five.
k Al, Bk
0 0.3 0.1
1 0.2 0.2
2 0.2 0.4
3 0.1 0.2
4 0.1 0.1
5 0.1 0.0
In order to reduce his risk, N.G. buys from both suppliers. Assuming he buys
ten chips, five chips from each, what should he expect to be the distribution
of defective chips? What is the connection between this and convolution?
6.8.2 Dee Espy has to purchase 100 DSPs, and decides that she should strive to
minimize the number of defective chips she purchases.How many should she
buy from each of the above suppliers?
6.8.3 Convolve the signal xn = . . . . 0,0,1,1,1,1,1,1,1,1,0,0... with the filter
h = (1, 1,1) and plot xn, h, and the output yn. Convolve vn with h, resulting
in y1211nand again this new signal to get yn[31, etc. Plot 273= y!?, Yn = y!l?
y1211,,y1311n,one under the other. What can be said about the effect of this
consecutive filtering?
6.9. PURELY RECURSIVESYSTEMS 241
and we have the relation a! + p = 1. Changing the index to our more usual
12we can now write
Figure 6.10: The (squared) frequency response of the simple AR low-pass filter for several
different values of p. From top to bottom /3 = 0.5,0.6,0.7,0.8,0.9,0.95.
and since the AR filter is a filter, we know that the output will be a complex
exponential of the same frequency.
yn = H(Ld)eiwn
= (1 - p> ~(~e-iw)keiwn
=
P-i, ,iwn
= (1 - p&w)
(1 - P>
H(w) = (1 _ p,-iw) (6.41)
I- 2p + P2
lW>12 = I - 2Pcos(w) + p2 (6.42)
as its square. We plot this squared frequency response, for several values of
,0, in Figure 6.10.
244 SYSTEMS
Yn = Xn + Yn-1 (6.43)
y = (1 + 2-l + C2 + * . .)x = Yx
Y Z C ZsmXn (6.44)
m=O
EXERCISES
6.9.1 What is the exact relation between ,0 in equation (6.39) and the amount of
past time r that is still influential? Define influential until the decrease is
by a factor of e-. Graph the result.
6.9.2 Quantify the bandwidth BW of the AR filter as a function of ,!!Iand compare
it with the influence time T.
6.9.3 Contrast the squared frequency response of the AR filter (as depicted in Fig-
ure 6.10) with that of the simple averaging MA filter (Figure 6.7). What can
you say about the amount of computation required for a given bandwidth?
6.10. DIFFERENCEEQUATIONS 245
6.9.4 Find an AR filter that passes high frequencies but attenuates low ones. Find
an AR filter that passes intermediate frequencies but attenuates highs and
lows.
6.9.5 Calculate the effect of the infinite accumulator operator Y on the following
signals, and then check by generating the first 10 values.
1. xn = un where un is the unit step of equation (2.4)
2. xn = (-l)Y& = 1, -l,l, -1,. . .
3. xn =nu,=0,1,2,3 ,...
4. xn = anun where Q!# 1
6.9.6 Apply the finite difference operator to the results obtained in the previous
exercise, and show that AT = 1.
6.9.7 Prove that YA = 1. (Hint: Prove that the telescoping series x1 - x0 + x2 -
Xl...Un -Un-l=X()+Xn.)
6.9.8 What is the condition for gn = axn + ,8yn-r to be stable? (Hint: Take a
constant input xn = 1 and compute yn when n --) XI.)
stock that day. The requisitions clerk orders new chips based on the average
usage over the past two days i (zn + xn-r), but these are only delivered the
next day. The number of chips in stock is thus given by an ARMA system.
where
w = al po c 1 pm = -bm for m = 1. . . M
although this way of expressing the relationship between y and x hides the
fact that yn can be simply derived from previous x and y values. This form
seems to be saying that the x and y signals are both equally independent,
but happen to obey a complex relationship involving present and past values
of both x and y. In fact the symmetric form equally well describes the inverse
system.
Xn = Yn - E bmyn-m - f&x,-l
m=l l=l
We can formally express equation (6.46) using the time delay operator
M L
and (as you will demonstrate in the exercises) in terms of finite differences.
Recalling from Section 2.4 that the finite difference operator bears some
resemblance to the derivative, this form bears some resemblance to a linear
differential equation
M
where xLkI is the kth derivative of x(t) with respect to t. For this reason
ARMA systems are often called &jJerence equations.
6.10. DIFFERENCEEQUATIONS 247
EXERCISES
6.10.1 Difference equations are not the only tool for describing ARMA systems; the
state-space description explicitly uses the systems memory (internal state).
Denoting the input xn, the output yn, and the vector of internal state vari-
ables at time n by sn, the state equation description relates the output to
the present input ana system state and furthermore describes how to update
the state given the input.
Yn = f 27-t-l + 9%
Sn+l = ASn + XnC
--
-
$1 = Jn-11
n
- $-;l
sin O(t - 2T)) = sin fit cos 20T - cos i;2t sin 2StT
(
= sin SZt (2 cos2 flT - 1) + cos Rt (2 sin RT cos RT)
= - sin Rt + 2 cos SZT 2 sin RT cos RT
( >
= - sin Ot + 2 cos RT sin i2(t - T)
or in digital form
Sn + ClSn-1 + C2S7-~-2 = 0 (6.52)
where cl = -2cosClT and c2 = 1.
This difference equation, obeyed by all sinusoids, can be exploited in
several different ways. In the most direct implementation it can be used as a
digital oscillator or tone generator, i.e., an algorithm to generate sinusoidal
signals. Given the desired digital oscillation frequency tid, amplitude A, and
initial phase 4, we precompute the coefficient
f-2
Cl = -2 cos flT = -2 cos - = -2 cos f&j
f5
and the first two signal values
SO = Asin
Sl = Asin(SZT + 4) = Asin(% -I- 4)
250 SYSTEMS
where Szdis the digital frequency (we will omit the subscript from here on).
The difference equations now recursively supply all signal values:
s2 = - (Wl + so>
s3 = - (w2 + Sl)
sq = - (ClS3 + s2)
and so on. This digital oscillator requires only one multiplication, one ad-
dition, and one sign reversal per sample point! This is remarkably efficient
when compared with alternative oscillator implementations such as approxi-
mation of the sine function by a polynomial, table lookup and interpolation,
or direct application of trigonometric addition formulas. The main prob-
lems with this implementation, like those of al purely recursive algorithms,
are those of accuracy and stability. Since each result depends on the previ-
ous two, numerical errors tend to add up, and eventually swamp the actual
calculation. This disadvantage can be rectified in practice by occasionally
resetting with precise values.
Another application of the difference equation (6.51) is the removal of an
interfering sinusoid. Given an input signal xn contaminated with an interfer-
ing tone at known frequency R; we can subtract the sinusoidal component
at this frequency by the following MA filter
Yn = Xn + ClX:,-1 + Xn-2
where cl is found from Q. The frequency response is found by substituting
an arbitrary complex exponential eiwn
iwn _ 2 cos neiw(n-I) + ,iw(n-2)
Yn = e
= 1 _ 2 cos ne-iw + ,-2iw ,iwn
( )
= e--L eiw - 2 cos 0 + emiw eiun
( >
-iw
(2 cos w - 2 cos Cl) eiwn
= e
IH(
Figure 6.11: The (squared) frequency response of the MA notch filter set to one-half the
Nyquist frequency. Note that no energy remains at the notched frequency.
EXERCISES
6.11.1 Show that the signal sn = eqn obeys the equation sn = as,-1 where a = eQ.
6.11.2 Show that the signal sn = sin(S2n) obeys the equation sn = ars,-1 +azsn-2
with coefficients ai determined by the equation 1 - alz- - CZ~Z-~= 0 having
solutions z = efm,
6.11.3 Show that if a signal is the sum of p exponentials
P
sn = &y&n
c
i=l
prediction a few seconds before noon. Of course when the system is tested
after noon your prediction turns out to be completely wrong! I think you
will agree that the game is only fair if we limit ourselves to the identification
of time-invariant systems.
Your opponent may still have a trick or two left! The system may have
been built to be sensitive to a very specific trigger. For example, for almost
every input signal the box may pass the signal unchanged; but for the trigger
signal the output will be quite different! A signal that is different from the
trigger signal in any way, even only having a slightly different amplitude
or having an infinitesimal amount of additive noise, does not trigger the
mechanism and is passed unchanged. You toil away trying a large variety
of signals and your best prediction is that the system is simply an identity
system. Then your opponent supplies the trigger as the test input and the
systems output quite astounds you.
The only sensible way to avoid this kind of pitfall is to limit ourselves to
linear systems. Linear systems may still be sensitive to specific signals. For
example, think of a box that contains the identity system and in parallel a
narrow band-pass filter with a strong amplifier. For most signals the output
equals the input, but for signals in the band-pass filters range the output is
strongly amplified. However, for linear systems it is not possible to hide the
trigger signal. Changing the amplitude or adding some noise will still allow
triggering to occur, and once the effect is observed you may home in on it.
So the system identification game is really only fair for linear time-
invariant systems, that is, for filters. It doesnt matter to us whether the
filters are MA, AR, ARMA, or even without memory; that can be deter-
mined from your measurements. Of course since the black box is a real
system, it is of necessity realizable as well, and in particular causal. There-
fore from now on we will assume that the black box contains an unknown
causal filter. If anyone offers to play the game without promising that the
box contains a causal filter, dont accept the challenge!
Our task in this section is to develop a winning strategy for the easy
case. Lets assume you are given one hour to examine the box in any way
you wish (short of prying off the top). At the end of precisely one hour
your opponent will reappear, present you with an input signal and ask you
what you believe the boxs response will be. The most straightforward way
of proceeding would be to quickly apply as many different input signals as
you can and to record the corresponding outputs. Then you win the game
if your opponents input signal turns out to be essentially one of the inputs
you have checked. Unfortunately, there are very many possible inputs, and
an hour is to short a time to test even a small fraction of them. To economize
254 SYSTEMS
we can exploit the fact that the box contains a linear time-invariant system.
If we have already tried input x7, there is no point in trying uxn or xnwm,
but this still leaves a tremendous number of signals to check.
Our job can be made more manageable in two different ways, one of
which relies on the time domain description of the input signal, and the
other on its frequency domain representation. The frequency domain ap-
proach is based on Fouriers theorem that every signal can be written as the
weighted sum (or integral) of basic sinusoids. Assume that you apply to the
unknown system not every possible signal, but only every possible sinusoid.
You store the systems response to each of these and wait for your opponent
to appear. When presented with the test input you can simply break it down
to its Fourier components, and exploit the filters linearity to add the stored
system responses with the appropriate Fourier coefficients.
Now this task of recording the system outputs is not as hard as it appears,
since sinusoids are eigensignals of filters. When a sinusoid is input to a filter
the output is a single sinusoid of the same frequency, only the amplitude
and phase may be different. So you need only record these amplitudes and
phases and use them to predict the system output for the test signal. For
example, suppose the test signal turns out to be the sum of three sinusoids
respectively. Then, since the filter is linear, the output is the sum of the
three responses, with the Fourier coefficients.
More generally, any finite duration or periodic test digital signal can be
broken down by the DFT into the sum of a denumerable number of complex
exponentials
N-l
1 Xkei+h
X7-b=- c
N
k=O
and the response of the system to each complex exponential is the same
complex exponential multiplied by a number Hk.
6.12. SYSTEM IDENTIFICATION-THE EASY CASE 255
The Hk are in general complex (representing the gains and phase shifts) and
are precisely the elements of the frequency response. A similar decomposition
solves the problem for nonperiodic analog signals, only now we have to test
a nondenumerable set of sinusoids.
The above discussion proves that the frequency response provides a com-
plete description of a filter. Given the entire frequency response (i.e., the
response of the system to all sinusoids), we can always win the game of
predicting the response for an arbitrary input.
The frequency response is obviously a frequency domain quantity; the
duality of time and frequency domains leads us to believe that there should
be a complete description in the time domain as well. There is, and we
previously called it the impulse response. To measure it we excite the system
with a unit impulse (a Dirac delta function s(t) for analog systems or a unit
impulse signal 6,,0 for digital systems) and measure the output as a function
of time (see equation 6.22). For systems without memory there will only be
output for time t = 0, but in general the output will be nonzero over an
entire time interval. A causal system will have its impulse response zero for
times t < 0 but nonzero for t 2 0. A system that is time-variant (and hence
not a filter) requires measuring the response to all the SUIs, a quantity
known as the Greens function.
Like the frequency response, the impulse response may be used to predict
the output of a filter when an arbitrary input is applied. The strategy is
similar to that we developed above, only this time we break down the test
signal in the basis of SUIs (equation (2.26)) rather than using the Fourier
expansion. We need only record the systems response to each SUI, expand
the input signal in SUIs, and exploit the linearity of the system (as we
have already done in Section 6.5). Unfortunately, the SUIs are not generally
eigensignals of filters, and so the systems outputs will not be SUIs, and we
need to record the entire output. However, unlike the frequency response
where we needed to observe the systems output for an infinite number of
basis functions, here we can capitalize on the fact that all SUIs are related by
time shifts. Exploiting the time-invariance property of filters we realize that
after measuring the response of an unknown system to a single SUI (e.g., the
unit impulse at time zero), we may immediately deduce its response to all
SUIs! Hence we need only apply a single input and record a single response
256 SYSTEMS
If we apply a unit impulse to a system and its output never dies down to
zero, we say that the system is Infinite Impulse Response (IIR). Systems of
the form (6.15), which we previously called IIR filters, can indeed sustain
an impulse response that is nonzero for an infinite amount of time. To see
this consider the simple case
Yn = Xn + ;Yn-1
which is of the type of equation (6.15). For negative times n the output is
zero, Yn = 0, but at time zero yc = 1, at time one yi = i and thereafter
yn is halved every time. It is obvious that the output at time r~ is precisely
Yn = 2 -+, which for large 72is extremely small, but never zero.
Suppose we have been handed a black box and measure its impulse
response. Although there may be many systems with this response to the
unit impulse, there will be only one filter that matches, and the coefficients
of equation (6.14) are precisely the impulse response in reverse order. This
means that if we know that the box contains a filter, then measuring the
impulse response is sufficient to uniquely define the system. In particular, we
neednt measure the frequency response since it is mathematically derivable
from the impulse response.
It is instructive to find this connection between the impulse response
(the time domain description) and the frequency response (the frequency
domain description) of a filter. The frequency response of the nonterminating
convolution system
00
Yn = c hixn-i
i=-00
is found by substituting a sinusoidal input for xn, and for mathematical
convenience we will use a complex sinusoid xn = eiwn. We thus obtain
H(U) xn = yn = 2 hkeiw(n-k)
k=-co
00
hk ,-iwk eiwn
=
c
(6.53)
k=-oo
= Hk Xn
here we have found the FT of the impulse response; hence the frequency
response and the impulse response are an FT pair. Just as the time and
frequency domain representations of signals are connected by the Fourier
transform, the simplest representations of filters in the time and frequency
domains are related by the FT.
EXERCISES
6.12.1 Find the impulse response for the following systems.
1. yn =xn
2. yn = Xn + Xn-2 +X72-4
3. yn = Xn + 2X,-1 + 3Xn-2
4. yn = xi aix,-i
5. Yn=Xn+Yn-1
6. Yn = xn + i(yn-1 + yn-2)
6.12.2 An ideal low-pass filter (i.e., one that passes without change signals under
some frequency but entirely blocks those above it) is unrealizable. Prove this
by arguing that the Fourier transform of a step function is nonzero over the
entire axis and then invoking the connection between frequency response and
impulse response.
6.12.3 When determining the frequency response we neednt apply each sinusoidal
input separately; sinusoid orthogonality and filter linearity allow us to apply
multiple sinusoids at the same time. This is what is done in probe signals
(cf. exercise 2.6.4). Can we apply all possible sinusoids at the same time and
reduce the number of input signals to one?
6.12.4 Since white noise contains all frequencies with the same amplitude, applying
white noise to the system is somehow equivalent to applying all possible
sinusoids. The white noise response is the response of a system to white
noise. Prove that for linear systems the spectral amplitude of the white noise
response is the amplitude of the frequency response. What about the phase
delay portion of the frequency response?
6.12.5 The fact that the impulse and frequency responses are an FT pair derives
from the general rule that the FT relates convolution and multiplication
FT(x * y) = FT(x)FT(y). P rove this general statement and relate it to the
Wiener-K hintchine theorem.
6.12.6 Donald S. Perfectionist tries to measure the frequency response of a system
by measuring the output power while injecting a slowly sweeping tone of
constant amplitude. Unbeknownst to him the system contains a filter that
passes most frequencies unattenuated, and amplifies a small band of frequen-
cies. However, following the filter is a fast Automatic Gain Control (AGC)
that causes all Donalds test outputs to have the same amplitude, thus com-
pletely masking the filter. Whats wrong?
6.13. SYSTEM IDENTIFICATION-THE HARD CASE 259
Yo = a()xo
Yl = aox + a1xo
Yo = aoxo
Yl = Npl + a120
Y2 = aox2 + a121 + u2xt-j (6.54)
Y3 = aox3 + am + a221 + ~3x0
Y4 = ~ox4+~lx3+a2x2+a3xl +u4q-)
Yl - aox
al =
x0
Y2 - aox2 - ~1x1
a2 = (6.55)
X0
a3 =
Y3 - aoxg - a122 - a2xl
X0
a4 =
Y4 - aox4 - alxg - a222 - aax1
x0
(6.56)
and identify the matrix containing the input values as being lower triangu-
lar and Toeplitz. The solution of (6.55) is simple due to the matrix being
lower triangular. Finding the Ith coefficient requires I multiplications and
subtractions and one division, so that finding all L + 1 coefficients involves
LL
2 (
L + 1) multiplications and subtractions and L + 1 divisions.
The above solution to the hard system identification problem was based
on the assumption that the input signal was exactly zero for n < 0. What
can we do in the common case when we start observing the signals at an
arbitrary time before which the input was not zero? For notational simplicity
lets assume that the system is known to be FIR with L = 2. Since we
6.13. SYSTEM IDENTIFICATION-THE HARD CASE 261
92 Xn-1 G-2
Xn+l Xn G-1 (6.58)
Xn+2 Xn+l Xn
or in other words -y = --
X a, where X is a nonsymmetric Toeplitz matrix. The
-
solution is obviously a = X-y but the three-by-three matrix is not lower
triangular, and so itsinv=ion is no longer trivial. For larger number of
coefficients L we have to invert an N = L + 1 square matrix; although most
direct N-by-N matrix inversion algorithms have computational complexity
O(N3), it is possible to invert a general matrix in O(NOg2 7, N O(N2a807)
time. Exploiting the special characteristics of Toeplitz matrices reduces the
computational load to O(IV2).
What about AR filters?
M
Yn = xn + C b,y,-,
m=l
or in matrix notation
or simply -y = -x + --
Y b. The answer this time is b = Y - ( -y - -x), which once
again necessitates inverting a nonsymmetric Toeplit; matrix.
Finally, the full ARMA with L = 2 and &! = 3
L M
Yn = xalxn-l+ C bmyn-m
l=O m=l
In order to average we multiply both sides by xn-q and sum over as many
n as we can get our hands on.
cn
Yn%-q = t
k=O
ak c%-kxn-q
n
Cx(-k) = Cx(k)
The deconvolution equations can now be written simply as
and are called the Wiener-Hopf equations. For L = 2 the Wiener-Hopf equa-
tions look like this:
Cx(O) Cx(-1) Cx(-2)
Cx(l> Cx(O) Cx(-1)
Cx(2) Cx(l)
and from the aforementioned symmetry we immediately recognize the matrix
as symmetric Toeplitz, a fact that makes them more stable and even faster
to solve.
For a black box containing an AR filter, there is a special case where the
input signal dies out (or perhaps the input happens to be an impulse). Once
the input is zero
Yn = 5 bmYn-m
??a=1
cn
YnYn-q = 5
m=l
bmxYn-mYn-q
n
264 SYSTEMS
WV c,(l) c,(2)
c,(l) c,(O) c,(l)
c,(2)c,(l) c,(O)
These are the celebrated Yule-Walker equations, which will turn up again
in Sections 9.8 and 9.9.
EXERCISES
6.13.1 Write a program that numerically solves equation (6.55) for the coefficients of
a causal MA filter given arbitrary inputs and outputs. Pick such a filter and
generate outputs for a pseudorandom input. Run your program for several
different input sequences and compare the predicted coefficients with the
true ones (e.g., calculate the squared difference). What happens if you try
predicting with too long a filter? Too short a filter? If the input is a sinusoid
instead of pseudorandom?
6.13.2 Repeat the previous exercise for AR filters (i.e., solve equation (6.60)). If the
filter seems to be seriously wrong, try exciting it with a new pseudorandom
input and comparing its output with the output of the intended system.
6.13.3 In the text we assumed that we knew the order L and M. How can we find
the order of the system being identified?
6.13.4 Assume that gn is related to xn by a noncausal MA filter with coefficients
a-M . . . aM. Derive equations for the coefficients in terms of the appropriate
number of inputs and outputs.
6.13.5 In deriving the Wiener-Hopf equations we could have multiplied by Y~.-~ to
get the equations
6.13.7 In an even harder system identification problem not only dont you have
control over the systems input, you cant even observe it. Can the system
be identified based on observation of the output only?
6.13.8 Assume that the observed input sequence X~ is white, i.e., that all autocor-
relations are zero except for CZ(0). What is the relationship between the
crosscorrelation CPZ(n) and the systems impulse response? How does this
simplify the hard system identification task?
What could be easier? If we wish we can even recover the impulse response
from the frequency response, by using equation (6.53). So it would seem best
to solve the easy system identification problem in the time domain and the
hard problem in the frequency domain.
266 SYSTEMS
One must be careful when using this frequency domain approach to the
hard system identification problem. To see why, think of an MA all-pass
system that simply delays the input by L samples yn = x~-L. Had we
observed the system when the input was of the form xn = sin(2nF + 4) for
any k we would conclude that we were observing the identity system yn = x,!
This mistake is obviously due to the input being periodic and the system
looking back in time by precisely an integer number of periods; equivalently
in the frequency domain this input consists of a single line, and thus we can
only learn about H(w) at this single frequency. The lesson to be learned
is more general than this simple example. In order to uniquely specify a
system the input must excite it at all frequencies. There are often frequencies
for which the system produces no output at all, and based on these we
certainly would not be able to identify the system. The unit impulse is a
single excitation that squeezes all possible information out of the system; due
to orthogonality and the eigenfunction property a single sinusoid contributes
only an infinitesimal amount of information about the system.
The frequency response is a great tool for FIR systems, but not as good
for IIR systems since they may become unstable. When an IIR systems
output increases without limit for a frequency w, this is a sign that its fre-
quency response is infinite there. For example, a typical frequency response
is depicted in Figure 6.12. As usual, the horizontal axis is from DC to half
the sampling rate. We see that the frequency response goes to zero for a
digital frequency of 0.2. This means that when the input is a sinusoid of
Figure 6.12: The frequency response of an ARMA filter with a zero at frequency 0.2fs
and a pole at 0.35f,.
6.14. SYSTEM IDENTIFICATION IN THE Z DOMAIN 267
Y(z)= c ynifn
n=-00
= C hk C xn-kZsn
= c00
k=-m n=-W
hkz-k 5 xmfm
k=-co m=-cm
= W) X(4
Y(4
w4 = x(z>
Using the time domain method of equation (6.55) after some work you
could deduce that the coefficients of the nonterminating convolution are
h, = ;-@+l) , but some inspiration would still be required before the ARMA
form could be discovered. So lets try the zT line of attack. The zTs of the
input and output are easily found by using equation (4.63).
1
X(z) = ZT un =n ROC Izl > 1
zT 2n+-1 =--- 1 1
p+i U7-i
1 - z-1 :: 1 - iz-l
i
= ROC Izl > ;
(1 - z-1)(1 - +z-)
w i
H(z) = x(z> = 1 _ 4,-l
Yn -- ;Yn-1 = +Xn
and equation (6.66) has magically appeared!
EXERCISES
6.14.1 As we shall learn in Chapter 18, modem signals are distorted by the tele-
phone lines through which they travel. This distortion can be modeled as a
filter, and removed by appropriate inverse filtering (equalizing). Can you ex-
plain why modems transmit known pseudonoise signals during their start-up
procedures?
6.14.2 You observe a system when its input consists of the sum of two different
sinusoids. Find two systems that cannot be distinguished based on this input.
Do the same for an input composed of M sinusoids.
6.14.3 What is the transfer function of two systems connected in series (cascaded
so that y = HOW,w = Hiz)? Of two systems connected in parallel (i.e., so
that y = Hlx + Hgy)?
6.14.4 Prove that ARMA systems commute.
6.14.5 Deconvolution is equivalent to finding the inverse system for a filter. Explain
how to carry out deconvolution using the transfer function.
6.14.6 Prove (as in exercise 6.4.1) that the inverse system of an MA filter is AR and
vice versa.
6.14.7 Many communication channels both distort the information carrying signal
by an unknown filter and add nonwhite noise to it. The frequency charac-
teristics of both the channel filter and the noise can be directly measured
by inputting a signal consisting of a comb of equidistant sinusoids each with
known amplitude and phase, and measuring the output at these frequencies.
The input signal is conveniently generated and the output recovered using
the DFT. In order to combat noise the procedure should be repeated N times
and the output averaged. We denote the input in bin k by XI, and the mea-
sured output at repetition m by Yi]. Explain how to measure the SNR of
bin k.
6.14.8 Continuing the previous exercise, a Frequency EQualizer (FEQ) tries to
remove the frequency distortion introduced by the channel filter by directly
multiplying each output Yk by complex number ek in order to recover the
input XI, = f?!kYk.Explain how to find the FEQ coefficients ek.
6.14.9 Continuing the previous exercises, the frequency magnitude response lHk12
(the ratio of the output to input energies) as measured at repetition m is
(Yy2
)Him1 I2 = -J+ . Express IHk I2 in terms of ek and SNRk.
k
270 BIBLIOGRAPHICAL NOTES
Bibliographical Notes
Signal processing systems are treated in all the standard signal processing texts,
[186, 185, 187, 200, 189, 252, 159, 1671, as well as books specifically on system
design [126].
The word convolution was used as early as 1935 by mathematicians, but seems
to have been picked up by the signal processing community rather later. Norbert
Wiener, in his classic 1933 text [277], uses the German Faltung noting the lack of
an appropriate English-language word. In his later book of the 1940s [278] there is
a conspicuous absence of the word. Rice, in an influential 1944-45 article on noise
[220] gives the FT of a product in an appendix, calling the convolution simply the
integral on the right. In 1958 Blackman and Tukey [19] use the word convolution
freely, although they mention several other possible names as well.
The impulse response, known in other fields as the Greens function, was first
published by Green in 1828 [86].
Amazingly, the Wiener-Hopf equations were originally derived in the early 1930s
to solve a problem involving radiation equilibrium in stars [281]. While working on
defense-related problems during World War II, Wiener discovered that these same
equations were useful for prediction and filtering. Several years before Eberhard
Hopf had returned to Nazi Germany in order to accept a professorship at Leipzig
that had been vacated by a cousin of Wieners who had fled Germany after the
rise of Hitler ([280]). Despite this turn of events Wiener always referred to the
Hopf-Wiener equations.
The great Cambridge statistician George Udny Yule formulated the Yule-Walker
equations for signals containing one or two sinusoidal components in the late 1920s
in an attempt to explain the periodicity of sunspot numbers [289]. A few years later
Sir Gilbert Walker expanded on this work [267], discovering that the autocorrela-
tions were much smoother than the noisy signal itself, and applying this technique
to a meteorological problem.
Otto Toeplitz was one of the founders of operator theory, as well as a great
teacher and historian of math [21, 141. In operator theory he was one of Hilberts
principle students, emphasizing matrix methods and considering Banachs methods
too abstract. In teaching he was a disciple of Felix Klein (who considered group
theory too abstract). In Bonn he would lecture to packed audiences of over 200
students, and was said to recognize each students handwriting and writing style. He
indirectly influenced the development of Quantum Mechanics by teaching his friend
Max Born matrix methods; Born later recognized that these were the mathematical
basis of Heisenbergs theory. Toeplitz was dismissed from his university position on
racial grounds after the Niirnberg laws of 1935, but stayed on in Germany until
1939 representing the Jewish community and helping minority students emigrate.
He finally left Germany in 1939, traveling to Jerusalem where he assumed the post
of scientific advisor to the Hebrew University, a position he held less than a year
until his death in 1940.
7
Filters
271
272 FILTERS
In this chapter we will learn how filters are specified in both frequency
and time domains. We will learn about fundamental limitations that make
the job of designing a filter to meet specifications difficult, but will not cover
the theory and implementation of filter design in great detail. Whole books
are devoted to this subject and excellent software is readily available that
automates the filter design task. We will only attempt to provide insight
into the basic principles of the theory so that the reader may easily use any
of the available programs.
IH(o IH(m)?
1.AurDu IH(m IH(c
Figure 7.1: Frequency response of ideal and nonideal filters. In (A) we see the low-pass
filters, in (B) the high-pass filters, in (C) the band-pass filters and in (D) the band-stop
(notch) filters.
future. On the left side of Figure 7.1 we show the frequency response of
ideal filters, while the right side depicts more realistic approximations to
the ideal response. Realistic filters will always have a finite transition region
between pass-bands and stop-bands, and often exhibit ripple in some or all
of these areas. When designing a filter for a particular application one has
to specify what amount of ripple and how much transition width can be
tolerated. There are many techniques for building both analog and digital
filters to specification, but all depend on the same basic principles.
Not all filters are low-pass, high-pass, band-pass, or band-stop, any fre-
quency dependent gain is admissible. The gain of a pre-emphasis filter in-
creases monotonically with frequency, while that of a de-emphasis filter de-
creases monotonically. Such filters are often needed to compensate for or
eliminate the effects of various other signal processing systems.
Filtering in the analog world depends on the existence of components
whose impedance is dependent on frequency, usually capacitors and induc-
tors. A capacitor looks like an open circuit to DC but its impedance de-
creases with increasing frequency. Thus a series-connected capacitor effec-
tively blocks DC current but passes high frequencies, and is thus a low-pass
filter. A parallel-connected capacitor short circuits high frequencies but not
DC or low frequencies and is thus a high-pass filter. The converse can be
said about series- and parallel-connected inductors.
Filtering in DSP depends on mathematical operations that remove or
emphasize different frequencies. Averaging adjacent signal values passes DC
and low frequencies while canceling out high frequencies. Thus averaging
behaves as a low-pass filter. Adding differences of adjacent values cancels
out DC and low frequencies but will pass signals with rapidly changing
signs. Thus such operations are essentially high-pass filters.
274 FILTERS
EXERCISES
7.1.1 Classify the following filters as low-pass, high-pass, band-pass, or notch.
1. Human visual system, which has a persistence of $ of a second
2. Human hearing, which cannot hear under 30 Hz or above 25KHz
3. Line noise filter used to remove 50 or 60 Hz AC hum
4. Soda bottle amplifying a specific frequency when air is blown above it
5. Telephone line, which rejects below 200 Hz and above 3800 Hz
7.1.2 Design an MA filter, with an even number of coefficients N, that passes a
DC signal (a, a, a,. . .) unchanged but completely kills a maximal frequency
signal (a, -a, a, -a,. . .). For example, for N = 2 you must find two numbers
gr and g2 such that gia + g2a = a but gas + 92(-u) = 0. Write equations
that the gi must obey for arbitrary N. Can you find a solution for odd N?
7.1.3 Design a moving average digital filter, with an even number of coefficients N,
that passes a maximal frequency signal unchanged but completely kills DC.
What equations must the gi obey now? What about odd N?
7.2. PHASE AND GROUP DELAY 275
7.1.4 The squared frequency response of the ideal low-pass filter is unity below the
cutoff frequency and zero above.
This type of delay is called bufler delay since it results from buffering the
inputs.
It is not difficult to show that if the impulse response is symmetric (or
antisymmetric) then the linear phase shift resulting from buffer delay is the
only phase distortion. Applying the symmetric noncausal FIR filter with an
odd number of coefficients
+L L
h,,, ,idn-m) = hoeiwn + 2eiwn c hi,1 cos(mw)
Yn = c
m=-L m=l
7.2. PHASE AND GROUP DELAY 277
so that the frequency response is real and thus has zero phase delay.
L
(the important step is isolating the imaginary portion) and the filter is seen
to be linear-phase, with phase shift corresponding to a time delay of L.
The converse is true as well, namely all linear-phase filters have impulse
responses that are either symmetric or antisymmetric. We can immediately
conclude that causal IIR filters cannot be linear-phase, since if the impulse
response continues to the end of time, and must be symmetric, then it must
have started at the beginning of time. This rules out the filter being causal.
From now on we will not consider a linear phase delay (constant time
delay) to be phase distortion. True phase distortion corresponds to nonlin-
earities in the phase as a function of frequency. To test for deviation from
linearity it is useful to look at the first derivative, since linear phase response
will have a constant derivative, and deviations from linearity will show up as
deviations from a constant value. It is customary to define the group delay
Figure 7.2: The difference between phase delay and group delay. In (A) we see the input
signal, consisting of the sum of two sinusoids of nearly the same frequency. (B) depicts
the output of a filter with unity gain, phase delay of vr, and zero group delay, while the
graph in (C) is the output of a filter with unity gain, no phase delay, but nonzero group
delay. Note that the local phase in (C) is the same as that of the input, but the position
of the beat amplitude peak has shifted.
In Figure 7.2 we see the effect of passing a signal consisting of the sum of
two sinusoids of nearly the same frequency through two filters. Both filters
have unity gain in the spectral area of interest, but the first has maximal
phase delay and zero derivative (group delay) there. The second filter has
zero phase delay but a group delay of one-half the beat period. Both filters
distort phase, but the phase distortions are different at the frequency of the
input signal.
EXERCISES
7.2.1 Show that an antisymmetric FIR filter (h, = -Ln) has zero phase and
when made causal has linear phase.
7.2.2 Prove that all linear-phase filters have impulse responsesthat are either sym-
metric or antisymmetric.
7.2.3 Assume that two filters have phase delay as a function of frequency @i(w)
and @Q(W).What is the phase delay of the two filters in series?What about
the group delay?
7.2.4 In a non-real-time application a nonlinear-phase filter is run from the end of
the signal buffer toward the beginning. What phase delay is introduced?
7.3. SPECIAL FILTERS 279
7.2.5 Stable IIR filters cannot be truly linear-phase. How can the result of the
previous exercise be used to create a filter with linear phase based on IIR
filtering? How can this technique be used for real-time linear-phase IIR fil-
tering with delay? (Hint: Run the filter first from the beginning of the buffer
to the end, and then back from the end toward the beginning.)
7.2.6 What is the phase delay of the IIR filter of equation (6.39)? What is the
group delay?
7.2.7 Can you think of a use for all-pass filters?
low frequency. Hence the task of smoothing is carried out in DSP by low-
pass filtering. The new interpretation of smoothing is that of blocking the
high-frequency noise while passing the signals energy.
The numerical analysis and DSP approaches are not truly incompatible.
For the usual case of evenly sampled data, polynomial smoothing can be
implemented as a filter, as was shown for the special case of a five-point
parabola in exercise 6.6.5. For that case the smoothed value at time n was
found to be the linear combination of the five surrounding input values,
Yn = a2n2 + aln + a0
For notational simplicity we will only consider retrieving the smoothed value
for n = 0, all other times simply requiring shifting the time axis.
The essence of the numerical analysis approach is to find the coefficients
a2, al, and a0 that make yn as close as possible to the 2L + 1 given xn
( 72 = --A.. +L). Th is is done by requiring the squared error
+L +L
E= C( Yn - Xn)2 = C ( a2n2 + a172 + a0 - 2n)2
n=-L n=-L
The B coefficients are universal, i.e., do not depend on the input xn,
and can be precalculated given L. It is obvious that if the data are evenly
distributed around zero (n = -L, -L + 1,. . . - 1,0, +l, . . . L - 1, L) then
7.3. SPECIAL FILTERS 281
Bij = 0 when i + j is odd, and the other required values can be looked up
in a good mathematical handbook.
+L
Cl = c n%
n=- L
+L
c2 = c n2Xn
n=-L
(7.3)
and the precise expressions for the D elements are also universal and can be
found by straightforward algebra.
vo = y-
1
v1 =
ig
a2
D2 = -2
D
x3oB2
v3 = 7
282 FILTERS
Now that we have found the coefficients ao, al, and a2, we can finally
find the desired smoothed value at n = 0
+L
YO = a2 = V&O + D&2 = c (Do + Dzn2)3h
n=-L
d
Y@> = ,,m (7.5)
a common use being the computation of the instantaneous frequency from
the phase using equation (4.67). The first approximation to the derivative
is the finite difference,
Yn = Xn - h-1
but for signals sampled at the Nyquist rate or only slightly above the sample
times are much too far apart for this approximation to be satisfactory. The
standard numerical analysis approach to differentiation is derived from that
for smoothing; first one approximates the input by some function, and then
one returns the value of the derivative of that function. Using the formal-
ism developed above we can find that in the parabolic approximation, the
derivative at n = 0 is given by
n=- L
Figure 7.3: Frequency and impulse responses of the ideal differentiation filter.
H(w) = iw (7 . 6)
This phase rotation is quite expected considering that the derivative of sine
is cosine, which is precisely such a 90 rotation. The impulse response, given
by the iFT of the frequency response,
i &t 1 e-i7d 1
= y.g -7r ( -- in > -it--/r--- (
(
it in >)
= 44
--- sin(7rt)
t 7rt2
is plotted in Figure 7.3.B.
We are more interested in digital differentiators than in the analog one
just derived. When trying to convert the frequency response to the digital
domain we run into several small snags. First, from the impulse response we
see that the ideal differentiator is unrealizable. Second, since the frequency
response is now required to be periodic, it can no longer be strictly linear,
but instead must be sawtooth with discontinuities. Finally, if the filter has an
284 FILTERS
h(t)
I 1 1 I I I I I =-t
Figure 7.4: Frequency and impulse responses of digital differentiation filters with even
and odd numbers of coefficients. In (A) we see the frequency response of an odd length
differentiator; note the linearity and discontinuities. (B) is the impulse response for this
case. In (C) we see the real and imaginary parts of the frequency response of an even
length differentiator. (D) is its impulse response; note that fewer coefficients are required.
(7.7)
which has both real and imaginary parts but is no longer discontinuous. We
now need to recalculate the impulse response.
The frequency and impulse responses for the odd and even cases are de-
picted in Figure 7.4. We see that FIR differentiators with an even number
of coefficients have no discontinuities in their frequency response, and hence
7.3. SPECIAL FILTERS 285
H(w)= J-
iw (7.9)
The standard Riemann sum approximation to the integral
nT
J0
x(t) dt = T(xo + x1 + . . . x,-~)
and well take T = 1 from here on. What is the frequency response of
this filter? If the input is xn = eiwn the output must be yn = H(w)eiwn
where H(w) is a complex number that contains the gain and phase shift.
Substituting into the previous equation
we find that
H(w) = 1-,-iw
lH(w)1* = l
2(1 - cos(w)
LH(w) = $(7r+w)
286 FILTERS
IH(
Figure 7.5: The (squared) frequency response of integrators. The middle curve is that
of the ideal integrator, the Riemann sum approximation is above it, and the trapezoidal
approximation below.
which isnt quite what we wanted. The phase is only the desired z at DC
and deviates linearly with w. For small w, where cos(w) N 1 - 2w2, 4 the
gain is very close to the desired 3, but it too diverges at higher frequencies
(see Figure 7.5). What this means is that this simple numeric integration is
relatively good when the signal is extremely oversampled, but as we approach
Nyquist both gain and phase response strongly deviate.
A slightly more complex numeric integration technique is the trapezoidal
rule, which takes the average signal value (x,-r + 2,) for the Riemann
rectangle, rather than the initial or final value. It too can be written as an
IIR filter.
Yn = Yn-1 + 4(%-l + %> (7.11)
Using the same technique we find
H(W)f2wn = Yn = yn-1 + !j(lXn-1 + 2,) = H(W)t2iw(n-1) + !j(,iw(n-l) + tZiwn)
LH(w) = f
so that the phase is correct, and the gain (also depicted in Figure 7.5) is
about the same as before. This is not surprising since previous signal values
7.3. SPECIAL FILTERS 287
contribute just as in the Riemann sum, only the first and last values having
half weight.
Integrators are always approximated by IIR filters. FIR filters cannot be
used for true integration from the beginning of all time, since they forget
everything that happened before their first coefficient. Integration over a
finite period of time is usually performed by a leaky integrator that grad-
ually forgets, which is most easily implemented by an IIR filter like that of
equation (6.39). While integration has a singular frequency response at DC,
the frequency response of leaky integration is finite.
Our final special filter is the Hilbert transform, which we introduced in
Section 4.12. There are two slightly different ways of presenting the Hilbert
transform as a filter. We can consider a real filter that operates on z(t)
creating y(t) such that z(t) = z(t) + iy(t) is the analytic representation, or
as a complex filter that directly creates z(t) from x(t). The first form has
an antisymmetric frequency response
H(w) = (7.12)
which means IH( = 1 and its phase is A$. The impulse response for
delay r is not hard to derive
2sin2 @(t-T))
w = -7r t- 7 (7.13)
except for at t = 0 where it is zero. Of course the ideal Hilbert filter is unre-
alizable. The frequency response of the second form is obtained by summing
X(w) with i times the above.
2 w>o
H(w) = (7.14)
0 WI0
The factor of two derives from our desire to retain the original energy after
removing half of the spectral components.
The Hilbert transform can be implemented as a filter in a variety of
ways. We can implement it as a noncausal FIR filter with an odd number of
coefficients arranged to be antisymmetric around zero. Its impulse response
2 sin2($t)
w =-
7r t
288 FILTERS
-W
Figure 7.6: Imaginary portion of the frequency response of a realizable digital Hilbert
filter with zero delay. The ideal filter would have discontinuities at both DC and half the
sampling frequency.
f&(t)= l
?r(t+ +)
but simplicity can be deceptive, and for the same amount of computation
odd order Hilbert filters have less error than even ones.
The trick in designing a Hilbert filter is bandwidth reduction, that is, re-
quiring that it perform the 90 phase shift only for the frequencies absolutely
required. Then the frequency response plotted in Figure 7.6 can be used as
the design goal, rather than the discontinuous one of equation (7.12).
EXERCISES
7.3.1 Generate a signal composed of a small number of sinusoids and approximate
it in a small interval by a polynomial. Compare the true derivative to the
polynomials derivative.
7.3.2 What are the frequency responsesof the polynomial smoother and differen-
tiator? How does the filter length affect the frequency response?
7.3.3 What is the ratio between the Riemann sum integration gain and the gain
of an ideal integrator? Can you explain this result?
7.3.4 Show that the odd order Hilbert filter when discretized to integer times has
all even coefficients zero.
7.4. FEEDBACK 289
7.4 Feedback
While FIR filters can be implemented in a feedfomvard manner, with the
input signal flowing through the system in the forward direction, IIR filters
employ fee&a&. Feedforward systems are simple in principle. An FIR with
N coefficients is simply a function from its N inputs to a single output; but
feedback systems are not static functions; they have dynamics that make
them hard to predict and even unstable. However, we neednt despair as
there are properties of feedback systems that can be easily understood.
In order to better understand the effect of feedback we will consider the
simplest case, that of a simple amplifier with instantaneous feedback. It is
helpful to use a graphical representation of DSP systems that will be studied
in detail in Chapter 12; for now you need only know that in Figure 7.7 an
arrow with a symbol above it represents a gain, and a circle with a plus sign
depicts an adder.
Were it not for the feedback path (i.e., were a = 0) the system would be
a simple amplifier y = Gz; but with the feedback we have
y=Gw (7.15)
where the intermediate signal is the sum of the input and the feedback.
W =x+ay (7.16)
Substituting
y=G(x+ay) =Gx+aGy
and solving for the output
G
Y =gx (7.17)
we see that the overall system is an amplifier like before, only the gain has
been enhanced by a denominator. This gain obtained by closing the feedback
Figure 7.7: The DSP diagram of an amplifier with instantaneous feedback. As will be
explained in detail in Chapter 12, an arrow with a symbol above it represents a gain, a
symbol above a filled circle names a signal, and a circle with a plus sign depicts an adder.
The feedforward amplifiers gain is G while the feedback path has gain (or attenuation) a.
290 FILTERS
Figure 7.8: An amplifier with delayed feedback. As will be explained in detail in Chap-
ter 12, a circle with zWN stands for a delay of N time units. Here the feedforward amplifiers
gain is G while the feedback path has delay of N time units and gain (or attenuation) a.
loop is called the closed loop gain. When a is increased above zero the closed
loop gain increases.
What if a takes precisely the value a = $? Then the closed loop gain
explodes! We see that even this simplest of examples produces an instability
or pole. Physically this means that the system can maintain a finite output
even with zero input. This behavior is quite unlike a normal amplifier; ac-
tually our system has become an oscillator rather than an amplifier. What
if we subtract the feedback from the input rather than adding it? Then for
a= $ the output is exactly zero.
The next step in understanding feedback is to add some delay to the
feedback path, as depicted in Figure 7.8. Now
Yn = Gwn
with
Wn = Xn + a&-N
where N is the delay time. Combining
and we see that for constant signals nothing has changed. What happens to
time-varying signals? A periodic signal xn that goes through a whole cycle,
or any integer number of whole cycles, during the delay time will cause
the feedback to precisely track the input. In this case the amplification will
be exactly like that of a constant signal. However, consider a sinusoid that
goes through a half cycle (or any odd multiple of half cycles) during the
delay time. Then yn-N will be of opposite sign to yn and the feedback will
destructively combine with the input; for aG = 1 the output will be zero!
The same is true for a periodic signal that goes through a full cycle (or
any multiple) during the delay time, with negative feedback (i.e., when the
feedback term is subtracted from rather than added to the input).
Wn = Xn - a&-N (7.19)
7.4. FEEDBACK 291
sn = e
while for aG < 1 the signal needs to decrease in the same fashion. This
is a general result; when the feedback gain is not unity the signals that
are optimally amplified or notched are exponentially growing or damped
sinusoids.
Continuing our argument it is easy to predict that if there are several de-
layed feedback paths in parallel then there will be several frequency regions
that are amplified or attenuated. We may even put filters in the feedback
path, allowing feedback at certain frequencies and blocking it at others. In-
deed this is the way filters are designed in analog signal processing; feedback
paths of various gains and phases are combined until the desired effect is
approximated.
292 FILTERS
Figure 7.9: The general feedback amplifier. The boxes represent general filters, with
transfer functions as marked. The amplifiers transfer function is H while that of the
feedback path is F.
In the most general setting, consider a digital system with transfer func-
tion H(z-l) to which we add a feedback loop with transfer function F(z-l),
as depicted in Figure 7.9. The closed loop transfer function is given by
H(z-l)
H(z-l) = 1 _ J(z-l)H(z-l) (7.20)
which has a pole whenever the denominator becomes zero (i.e., for those z
for which F(z-l) H (z-1> = 1). The value of z determines the frequency of
the oscillation.
EXERCISES
7.4.1 When the microphone of an amplification system is pointed toward the
speaker a squealing noise results. What determines the frequency of the
squeal? Test your answer. What waveform would you expect?
7.4.2 A feedback pole causesan oscillation with frequency determined by the delay
time. This oscillation is sustained even without any input. The system is
linear and time-invariant, and so is a filter; as a filter it cannot create energy
at a frequency where there was no energy in the input. Resolve this paradox.
7.4.3 What is the effect of a delayed feedback path with unity gain on a sinusoid of
frequency close, but not equal, to the instability? Plot the gain as a function
of frequency (the frequency response).
7.4.4 Find a signal that destabilizes a system with a delayed positive feedback path
and nonunity feedback gain.
7.4.5 Show that for G = ?, a = 1 a sinusoid of frequency corresponding to the
delay is amplified by the gain predicted by equation (7.17).
7.5. THE ARMA TRANSFER FUNCTION 293
7.4.6 What is the effect of a delayed feedback path with nonunity gain G on a
sinusoid of frequency corresponding to the delay? Plot the effective gain as a
function of G.
7.4.7 Simulate a system that has a causal MA filter in the feedback path. Start
with a low-pass filter, then a high-pass, and finally a band-pass. Plot the
frequency response.
l=O u=--00
! cpmZAm
/M
Y(z) =
m=O )
294 FILTERS
and comparing with equation (6.65) we find that the transfer function is the
ratio of two polynomials in 2-l.
(7.21)
where the D roots <i are in general complex numbers. When the coefficients
ci are real, the sum itself is always real, and so the roots must either be
real, or appear in complex conjugate pairs. Thus we can rewrite the transfer
function of the general ARMA filter as
(7.26)
7.5. THE ARMA TRANSFER FUNCTION 295
where the roots of the numerator <i are called zeros of the transfer function,
and the roots of the denominator 7rr, its poles. So other than an simple
overall gain G, we need only specify the zeros and poles to completely de-
termine the transfer function; no further information is needed.
From equations 7.23 and 7.24 we see that the transfer function of the
MA filter has zeros but no poles while that of the AR filter has poles but no
zeros. Hence the MA filter is also called an all-zero filter and the AR filter
is called an all-pole filter.
What is the meaning of these zeros and poles? A zero of the transfer
function is a complex number c = reiw that represents a complex (possibly
decaying or increasing) exponential signal that is attenuated by the ARMA
filter. Poles 7r represent complex exponential signals that are amplified by
the filter. If a zero or pole is on the unit circle, it represents a sinusoid that
is either completely blocked by the filter or destabilizes it.
Since the positions in the complex plane of the zeros and poles provide a
complete description of the transfer function of the general ARMA system,
it is conventional to graphically depict them using a pole-zero plot. In such
plots the position of a zero is shown by a small filled circle and a pole is
marked with an X. Poles or zeros at x = 0 or x = co that derive from the
xMdL factor in equation (7.25) are not depicted, but multiple poles and/or
zeros at the same position are. This single diagram captures everything one
needs to know about a filter, except for the overall gain.
A few examples are in order. First consider the causal equally weighted
L+ l-point average MA filter (since we intend to discard the gain we neednt
normalize the sum).
L
Yn = c G-1
I=0
By inspection the transfer function is
1 _ x-1-L-
H(z) = + = 1_ z-l =- $ z:l, l
z=o
and we seem to see L poles at the origin, the L+ 1 zeros of xLfl - 1 and a pole
at x = 1. The zeros are the L + 1 roots of unity, z = e i2x*, one of which
is x = 1 itself; hence that zero cancels the putative pole at z = 1. The L
poles at the origin are meaningless and may be ignored. We are therefore left
with L zeros equally spaced around the unit circle (not including x = l), as
displayed in Figure 7.10.A. It is not difficult to verify that the corresponding
sinusoids are indeed blocked by the averaging MA filter.
296 FILTERS
Figure 7.10: The pole-zero plots of two simple systems. In (A) we see the pole-zero plot
for the MA filter that averages with equal weights eight consecutive input values. In (B)
is the simple AR low-pass filter y,, = (1 - p)sn + fly,+ 1.
which has a trivial zero at the origin and a single pole at p, as depicted in
Figure 7.10.B.
As our last example we choose a general first-order section, that is, an
ARMA system with a single zero and a single pole.
has its zero at z = -2 and its pole at z = bl. To find the frequency response
we substitute x = elw
a0 + ale+
H(w)=
1 - bleviw
7.5. THE ARMA TRANSFER FUNCTION 297
(z - h) + (2 + h)
H(z) = a0
z- h
(2 + h)z
+ 1 -b z-L
1
a0 n=O
h, =
(a+ aoh)b, ~4 n#O
EXERCISES
7.5.1 Sometimes it is useful to write difference equations as yn = Gx, +C ulxn-l +
C b,y,-, where G is called the gain. Write the transfer function in rational-
function- and factored-form for this case.
7.5.2 Derive equation (7.21) more simply than in the text by using the time shift
relation for the zT.
7.5.3 Consider the system with a single real pole or zero. What signal is maximally
amplified or attenuated? Repeat for a complex pole or zero.
7.5.4 Calculate the transfer function H(z) for the noncausal MA system of equa-
tion (6.35). Relate this to the transfer function of the causal version and to
the frequency response (equation (6.36)) previously calculated.
7.5.5 Show that stable ARMA filters have all their poles inside the unit circle.
7.5.6 Prove that real all-pass filters have poles and zeros in conjugate reciprocal
locations.
7.5.7 Show that the first-order section is stable when Ibi) < 1 both by considering
the pole and by checking the impulse response.
7.5.8 Plot the absolute value of the frequency response of the first-order section
for frequencies between DC and Nyquist. When is the filter low-pass (passes
low frequencies better than highs)?
7.5.9 If the maximum input absolute value is 1, what is the maximal output abso-
lute value for the first-order section? If the input is white noise of variance
1, what is the variance of the output of the first-order section?
298 FILTERS
7.5.10 In general, when breaking down ARMA systems into first-order sections the
zeros and poles may be complex. In such caseswe most often use real-valued
second-order sections instead.
a0 + alz-l + a2zm2
W) = 1 _ b-l
1 - b2r2
What is the frequency response for the second-order section with complex
conjugate poles?
The pole-zero plot directly depicts the transfer function, but the fre-
quency response is also easily inferred. Think of the unit circle as a circular
railroad track with its height above sea level representing the gain at the
corresponding frequency. In this analogy poles are steep mountains and ze-
ros are craters. As the train travels around the track its height increases and
decreases because of proximity to a mountain or crater. Of course at any
position there may be several poles and/or craters nearby, and the overall
height is influenced by each of them according to its distance from the train.
Now lets justify this analogy. Substituting x = & into equation (7.26) we
find that the frequency response of an ARMA systems is
rI~l@~- c>
H(w)= Gn~=l(,i~_ n,) (7.27)
kl m=l
The Zth factor in the numerator of the magnitude is the distance between
the point on the unit circle and the Zth zero, and the mth factor in the
denominator is the distance to the mth pole. The magnitude is seen to be
the product of the distances to all the zeros divided by the product to all
the poles. If one of the zeros or poles is very close it tends to dominate, but
in general the trains height is influenced by all the mountains and craters
according to their distances from it. The Zth term in the numerator of the
angle is the direction of the vector between the point on the unit circle and
the lth zero and the mth term in the denominator is the angle to the mth
pole. Therefore the phase of the frequency response is seen to be the sum
of the angles to all the zeros minus the sum of the angles to the poles. If
one of the zeros or poles is very close its angle changes rapidly as the train
progresses, causing it to dominate the group delay.
Suppose we design a filter by some technique and find that a pole is
outside the unit circle. Is there some way to stabilize the system by moving
it back inside the unit circle, without changing the frequency response?
The answer is affirmative. Let the pole in question be ~0 = PC?. You can
convince yourself that the distance from any point on the unit circle to ~0 is
exactly P2 times the distance to ;~rb= $ele, the point along the same radius
7.6. POLE-ZERO PLOTS 301
but with reciprocal magnitude. Thus to within a gain term (that we have
been neglecting here) we can replace any pole outside the unit circle with
its reciprocal conjugate 7r&. This operation is known as reflecting a pole.
We can also reflect a zero from outside the unit circle inward, or from the
inside out if we so desire. For real filters we must of course reflect both the
pole and its complex conjugate.
Lets see how the concept of a pole-zero plot enables us to design some
useful filters. Assume we want a DC blocker, that is, a filter that blocks DC
but passes AC frequencies. A first attempt might be to simply place a zero
at DC
discarding the term representing a zero at z = 0; but this filter is simply the
finite difference, with frequency response
Here p < 1 but the closer ,0 is to unity the sharper the notch will be. There
is a minor problem regarding the gain of this filter. We would like the gain to
be unity far away from DC, but of course pole-zero methods cannot control
the gain. At x = -1 our DC blocker has a gain of
z- 1
-z-z -1-1 1
x- P -1-p 1-g
angles but slightly reduced radius. We can also make a sharp band-pass filter
by reversing the roles of the zeros and poles. Wider band-pass or band-stop
filters can be approximated by placing several poles and/or zeros along the
desired band. Every type of frequency-selective filter you can imagine can
be designed by careful placement of poles and zeros.
EXERCISES
7.6.1 Although practically every filter you meet in practice is ARMA, they are
not the most general LTI system. Give an example of a linear time-invariant
system that is not ARMA.
7.6.2 Make a pole-zero plot for the system
(z - 4(x - ;I
H(z) = (2 - ?Yl)(z - 5)
.
where CL= em and T 2 1. Sketch the frequency response. What kind of filter
is this?
7.6.3 Why did we call XL the reciprocal conjugate? Prove that the distance from
any point on the unit circle to ~0 is exactly P2 times the distance to the
reciprocal conjugate 7rb.
7.6.4 A stable system whose inverse is stable as well is said to be minimum phase.
What can you say about the pole-zero plot of a minimum phase system?
7.6.5 Prove that reflecting poles (or zeros) does not change the frequency response.
7.6.6 What can be said about the poles and zeros of an all-pass filter? What is the
connection between this question and the previous one?
7.6.7 A notch filter can be designed by adding the outputs of two all-pass filters
that have the same phase everywhere except in the vicinity of the frequency
to be blocked, where they differ by 180. Design a notch filter of the form
H(z) = 4 (1 + A( z)) where A(z) is the transfer function of an all-pass filter.
How can you control the position and width of the notch?
7.6.8 Consider the DC blocking IIR filter yh = 0.9992(x1, - 21~~1) + 0.9985yk-r.
Draw its frequency response by inputting pure sinusoids and measuring the
amplitude of the output. What is its pole-zero plot?
7.7. CLASSICAL FILTER DESIGN 303
IH(
A
Figure 7.11: Desired frequency response of the analog low-pass filter to be designed. The
pass-band is from f = 0 to the pass-band edge fr,, the transition region from fP to fS, and
the stop-band from the top-band edge fS to infinity. The frequency response is halfway
between that of the pass-band and that of the stop-band at the cutoff frequency fC. The
maximal ripple in the pass-band is 6, and in the stop-band 6,.
We will first focus on the simplest case, that of an analog low-pass filter.
Our ideal will be the ideal low-pass filter, but that being unobtainable we
strive toward its best approximation. The most important specification is the
cutoff frequency fc, below which we wish the signal to be passed, above which
we wish the signal to be blocked. The pass-band and stop-band are separated
by a transition region where we do not place stringent requirements on the
frequency response. The end of the pass-band is called fp and the beginning
304 FILTERS
jH(0)12 = 1
lwf>12 * 1 for f < fc
lwf>2 x 0 for f > fc
Wf)12 + 0 for f + 00
are the requirements for an analog low-pass filter. The first functional forms
that come to mind are based on arctangents and hyperbolic tangents, but
these are natural when the constraints are at plus and minus infinity, rather
than zero and infinity. Classical filter design relies on the form
which when substituted back into equation (7.28) gives the slowest filter
depicted in Figure 7.12. The other filters there are derived from
P(f)= (;)2
7.7. CLASSICAL FILTER DESIGN 305
IH(
Figure 7.12: Frequency response of analog Butterworth low-pass filters. From bottom to
top at low frequencies we have order N = 1,2,3,5,10,25,00.
and are called the Butterworth low-pass filters of order n. It is obvious from
the figure that the higher n is the narrower the transition.
Butterworth filters have advantages and disadvantages. The attenuation
monotonically increases from DC to infinite frequency; in fact the first 2N- 1
derivatives of IH(f) I2 are identically zero at these two points, a property
known as maximal flatness. An analog Butterworth filter has only poles and
is straightforward to design. However, returning to the design specifications,
for the transition region A to be small enough the order N usually has to
be quite high; and there is no way of independently specifying the rest of
the parameters.
In order to obtain faster rolloff in the filter skirt we have to give some-
thing up, and that something is the monotonicity of IH(f)12. A Butterworth
filter wastes a lot of effort in being maximally flat, effort that could be put
to good use in reducing the size of the transition region. A filter that is al-
lowed to oscillate up and down a little in either the pass-band, the stop-band
or both can have appreciably smaller A. Of course we want the deviation
from our specification to be minimal in some sense. We could require a
minimal squared error between the specification and the implemented filter
IH( IH(
A
\
A C
f ,f
IH( IH(
f IE
Figure 7.13: Frequency response of low-pass equiripple designs. In (A) we see an FIR
filter designed using the Remez algorithm for comparison purposes. In (B) we the IIR
Chebyshev design, in (C) the inverse Chebyshev and in (D) the elliptical design.
and like the Butterworth approximation, the analog Chebyshev filter is all-
pole. The inverse Chebyshev (or Chebyshev II) filter is equiripple in the
stop-band but maximally flat in the pass-band. Its polynomial is
the opposite. For both types of Chebyshev filters the parameter S sets the
ripple in the equiripple band. For the inverse Chebyshev, where the equirip-
ple property holds in the stop-band, the attenuation is determined by the
ripple; lower ripple means higher stop-band rejection.
Finally, the elliptical filter is equiripple in both pass-band and stop-
band, and so approximates the desired response in the minimax sense for all
frequencies. Its polynomial is not a polynomial at all, but rather a rational
function UN(i). These functions are defined using the elliptical functions
(see Appendices A.8 and A.lO). Taking the idea from equation (A.59), we
define the function
u dw (u > z snk r snq-l u (7.29)
( ( 0
and when r and the complete elliptical integrals & and Kq obey certain
relations that we will not go into here, this function becomes a rational
function.
(u~-u2)(u~-u2)*..(u~~~I-u2)
(l-u:212)(1-u~u2)...(1-~~~-~~2)
N even
&V(u) = a2 (7.30)
u(?.+-u)(?L;-+.(u2 4)
(1-u;u2)(l-u~u2)~.*(l-u;~u2)
N odd
IW)12= l
1+h(k)
is equiripple in both the pass-band and the stop-band.
It is useful to compare the four types of analog filter-Butterworth,
Chebyshev, inverse Chebyshev, and elliptical. A very strong statement can
be made (but will not be proven here) regarding the elliptical filter; given any
three of the four parameters of interest (pass-band ripple, stop-band ripple,
transition width, and filter order) the elliptical filter minimizes the remain-
ing parameter. In particular, for given order N and ripple tolerances the
elliptical filter can provide the steepest pass-band to stop-band transition.
The Butterworth filter is the weakest in this regard, and the two Chebyshev
308 FILTERS
types are intermediate. The Butterworth filter, however, is the best approxi-
mation to the Taylor expansion of the ideal response at both DC and infinite
frequency. The Chebyshev design minimizes the maximum pass-band ripple,
while the inverse Chebyshev maximizes the minimum stop-band rejection.
The design criteria as we stated them do not address the issue of phase
response, and none of these filters is linear-phase. The elliptical has the
worst phase response, oscillating wildly in the pass-band and transition re-
gion (phase response in the stop-band is usually unimportant). The Butter-
worth is the smoothest in this regard, followed by the Chebyshev and inverse
Chebyshev.
Although this entire section focused on analog low-pass filter, the prin-
ciples are more general. All analog filters with a single pass-band and/or
stop-band can be derived from the low-pass designs discussed above. For
example, we can convert analog low-pass filter designs into high-pass filters
by the simple transformation f --$ j. Digital filters are a somewhat more
complex issue, to be discussed in the next section. For now it is sufficient to
say that IIR filters are often derived from analog Butterworth, Chebyshev,
inverse Chebyshev, or elliptical designs. The reasoning is not that such de-
signs are optimal; rather that the theory of the present section predated DSP
and early practitioners prefered to exploit well-developed theory whenever
possible.
EXERCISES
7.7.1 Show that a Butterworth filter of order N is maximally flat.
7.7.2 All Butterworth filters have their half gain (3 dB down) point at fc. Higher
order N makes the filter gain decrease faster, and the speed of decrease is
called the rollofl , Show that for high frequencies the rolloff of the Butter-
worth filter is 6 dB per octave (i.e., the gain decreases 6 dB for every doubling
in frequency) or 20 dB per decade. How should N be set to meet a specifica-
tion involving a pass-band end frequency fpr a stop-band start frequency fs,
and a maximum error tolerance 6?
7.7.3 Show that the 2N poles of IH( for the analog Butterworth filter all lie
on a circle of radius fc in the s-plane, are equally spaced, and are symmetric
with respect to the imaginary axis. Show that the poles of the Chebyshev I
filter lie on an ellipse in the s-plane.
7.7.4 The HPNA 1.0 specification calls for a pulse consisting of 4 cycles of a 7.5
MHz square wave filtered by a five-pole Butterworth filter that extends from
5.5 MHz to 9.5 MHz. Plot this pulse in the time domain.
7.8. DIGITAL FILTER DESIGN 309
IH( = f* f,
where a and fc are parameters. What type of filter is this and what is the
meaning of the parameters?
7.7.6 Repeat the previous exercise for these filters.
Wf)12 = &+
IH( = *
C
7.7.7 Show that in the pass-band the Chebyshev filter gain is always between ,-&
and h so that the ripple is about 4b2 dB. Show that the gain falls mono-
tonically in the stop-band with rolloff 20N dB per decade but always higher
than the Butterworth filter of the same order.
7.7.8 We stated that an analog low-pass filter can be converted into a high-pass
filter by a simple transformation of the frequency variable. How can band-
pass and band-stop filters be similarly designed by transformation?
undeniably FIR filter coefficients; but such an approach is only helpful when
the impulse response quickly dies down to zero. Also numeric transformation
of N frequency values will lead to a filter that obeys the specification at the
exact frequencies we specified, but at in-between frequencies the response
may be far from what is desired. The main trick behind filter design is how to
constrain the frequency response of the filter so that it does not significantly
deviate from the specification at any frequency.
We should note that this malady is not specific to time domain filter-
ing. Frequency domain filtering uses the FT to transfer the signal to the
frequency domain, performs there any needed filtering operation, and then
uses the iFT to return to the time domain. We can only numerically perform
a DFT for a finite number of signal values, and thus only get a finite fre-
quency resolution. Multiplying the signal in the frequency domain enforces
the desired filter specification at these frequencies only, but at intermedi-
ate frequencies anything can happen. Of course we can decide to double
the number of signal times used thus doubling the frequency resolution, but
there would still remain intermediate frequencies where we have no control.
Only in the limit of the LTDFT can we completely enforce the filter spec-
ification, but that requires knowing the signal values at all times and so is
an unrealizable process.
At its very outset the theory and practice of digital filter design splits
into two distinct domains, one devoted to general IIR filters, and the other
restricted to linear-phase FIR filters. In theory the general IIR problem
is the harder one, and we do not even know how to select the minimum
number of coefficients that meet a given specification, let alone find the
optimal coefficients. Yet in practice the FIR problem is considered the more
challenging one, since slightly suboptimal solutions based on the methods of
the previous section can be exploited for the IIR problem, but not for the
FIR one.
Lets start with IIR filter design. As we mentioned before we will not
attempt to directly optimize filter size and coefficients; rather we start with
a classical analog filter design and bring it into the digital domain. In order
to convert a classical analog filter design to a digital one, we would like to
somehow digitize. The problem is that the z-plane is not like the analog
(Laplace) s-plane. From Section 4.10 we know that the sinusoids live on
the imaginary axis in the s-plane, while the periodicity of digital spectra
force them to be on the unit circle in the x-plane. So although the filter was
originally specified in the frequency domain we are forced to digitize it in
the time domain.
7.8. DIGITAL FILTER DESIGN 311
=g1sIT
response by taking the iLTDFT
hn -7r
H(eiw)eiwn du (7.33)
312 FILTERS
and that these h, are the coefficients of the convolution in the time do-
main. Therefore, the theoretical frequency responses of the ideal low-pass,
high-pass, band-pass, and band-stop filters already imply the coefficients of
the ideal digital implementation. Assuming a noncausal filter with an odd
number of coefficients, it is straightforward to find the following.
GG n=O
low-pass: h, =
!$ si&nw,) n#O
1-y n=O
high-pass: h, = (7.34)
{ - 9 sinc(nw,) n#O
w2-wL n=O
band-pass: h, =
{ p sinc(nw2) If. *~ sinc(nwr) n#O
l+F n=O
band-stop: h, =
F sinc(nwi) - F sinc(nw2) n # 0
to be as narrow as possible. These two wishes must be traded off because the
uncertainty theorem limits how confined the window can simultaneously be
in the time and frequency domains. In order to facilitate this trade-off there
are window families (e.g., kaiser and Dolph-Chebyshev) with continuously
variable parameters.
So the windowing method of FIR filter design goes something like this.
IH(f)l A IH(f)l B
0 T-
0.1 0.2
-20
40
40
do
-100* -woJ
Figure 7.14: FIR design by window method vs. by Remez algorithm. (A) is the frequency
response of a 71-coefficient low-pass filter designed by the window method. (B) is a 41-
coefficient filter designed by the Remez algorithm using the same specification. Note the
equiripple characteristic.
314 FILTERS
EXERCISES
7.8.1 Some digital filter design programs assume a sampling frequency (e.g., 8000
Hz). Can these programs be used to design filters for systems with different
sampling frequencies?
7.8.2 Obtain a good filter design program and design an IIR low-pass filter using
the four classical types from Section 7.7. What happens as you force the tran-
sition region to shrink in size? What is the effect of fp for a given transition
region width? Plot the phase response and group delay. How can you make
the phase response more linear?
7.8.3 Design a low-pass FIR using the same criteria as in the previous exercise.
Compare the amount of computation required for similar gain characteristics.
7.9. SPATIAL FILTERING 315
7.8.4 Repeat the previous two questions for a narrow band-pass filter.
7.8.5 An extremely narrow FIR low-pass filter requires a large number of coeffi-
cients, and hence a large amount of computation. How can this be reduced?
7.8.6 The impulse responses in equations (7.34) were for odd N. For even N the
ideal frequency responses must be shifted by a half-integer delay eig before
applying equation (7.33). Find the ideal impulse responses for even N.
7.8.7 What are the coefficients of the ideal differentiator and Hilbert filters for even
and odd N?
those from other angles. The problem with such directional antennas is that
changing the preferred direction involves physically rotating the antenna to
point the desired way. Beamforming is a technique, mainly utilized in trans-
mission and reception of sonar and radar signals, for focusing transmitted
energy or amplifying received energy without having to physically rotate
antennas. This feat is performed by combining a number of omnidirectional
sensors (antennas, microphones, hydrophones, or loudspeakers depending on
the type of wave).
Figure 7.15: Beamforming to separate two sinusoidal signals of the same frequency. The
sensor array consists of two antennas separated by the distance traveled by the wave
during half a period. Each sensor is connected to a phase shifter and the phase shifted
signals are summed.
Figure 7.16: A wave impinging upon an array of M = 5 sensors spaced d apart. The
parallel lines represent the peaks of the sinusoids and hence there is one wavelength X
between each pair. The wave arrives at angle 0 from the normal to the line of the sensors.
It is obvious from the geometry that when X = dsin8 the wave takes the same value at
all of the sensors.
IH(d12
OA.
-5-
-10 -
.15-
-20-
-25 -
-30
-35
40
Figure 7.17: Angle response of a phased array. We depict the square of the response in
dB referenced to the zero degree response for a phased array with A4 = 32 and ni = $.
The phased array is pointed to 8 - 0 and the horizontal axis is the angle 4 in degrees.
where cp is the phase at the first sensor, and k is the DOA variable corre-
sponding to angle 4. Substituting this into equation (7.36)
M-l
A ei~eiwneikme-ilcm
2, = c
m=O
M-l
= Aei~$wn C &(k-n)m
m=O
1 - ,&Wk-4
= A &&wn
1 - ei(k-n)
I2*=
I sin M7rf(sin 4 - sin 0)
sin 7rf(sin q5- sin 0)
*
(7.37)
So the phased array acts as a spatial filter that is really quite similar to
a regular frequency domain filter. The angle response of equation (7.37) is
analogous to the frequency response of a frequency filter, and the high side-
lobes in Figure 7.17 can be attenuated using techniques from filter design,
such as windowing.
Our discussion has focused on simple sinusoidal waves; what if we need
to pull in a complex wave? If the wave consists of only two frequency com-
ponents, we can build two separate phased arrays based on the same sensors
and add their results, or equivalently a single phased array with two delays
per sensor. A little thought should be sufficient to convince you that arbi-
trary waves can be accommodated by replacing the simple phase delay with
full FIR filters. In this way we can combine spatial and frequency filtering.
Such a combined filter can select or reject a signal based on both its spectral
and spatial characteristics.
EXERCISES
7.9.1 Direction fixing can also be performed using time of arrival (TOA) techniques,
where the time a signal arrives at multiple sensors is compared. We use
both phase differences and TOA to locate sound sources with our two ears,
depending on the frequency (wavelength) of the sound. When is each used?
How is elevation determined? (Hint: The external ear is not symmetric.) Can
similar principles be exploited for SONAR echolocation systems?
7.9.2 Bats use biological sonar as their primary tool of perception, and are able
to hunt insects at night (making the expression blind as a bat somewhat
frivolous). At first, while searching for insects, they emit signals with basic
frequency sweeping from 28 KHz down to 22 KHz and duration of about
10 milliseconds, Once a target is detected the sounds become shorter (about
3 milliseconds) in duration but scan from 50 KHz down to 25 KHz. While
attempting to capture the prey, yet a third mode appears, of lower bandwidth
and duration of below 1 millisecond. What is the purpose of these different
cries? Can similar nrincinles be used for fighter aircraft radar?
320 BIBLIOGRAPHICAL NOTES
Bibliographical Notes
Filters and filter design are covered in all standard DSP texts [186, 185, 200, 1671,
and chapters 4 and 5 of [241], as well as books devoted specifically to the subject
[191]. Many original papers are reprinted in [209, 40, 411.
The original Parks-McClellan FIR design program is described and (FOR-
TRAN) source code provided in [192, 1651. Extensions and portings of this code
to various languages are widely available. After the original article appeared, much
follow-on work appeared that treated the practical points of designing filters, in-
cluding differentiators, Hilbert transforms, etc. [208, 213, 212, 78, 2071.
Exercise 7.2.5 is based on [197].
8
Nonfilters
Filters have a lot going for them. In the previous chapter we have seen that
they are simple to design, describe and implement. So why bother devoting
an entire chapter to the subject of systems that are not filters?
There are two good reasons to study nonfilters-systems that are ei-
ther nonlinear, or not time-invariant, or both. First, no system in the real
world is ever perfectly linear; all linear analog systems are nonlinear if you
look carefully enough, and digital signals become nonlinear due to round-off
error and overflow. Even relatively small analog nonlinearities can lead to ob-
servable results and unexpected major nonlinearities can lead to disastrous
results. A signal processing professional needs to know how to identify these
nonlinearities and how to correct them. Second, linear systems are limited
in their capabilities, and one often requires processing functions that sim-
ply cannot be produced using purely linear systems. Also, linear systems are
predictable; a small change in the input signal will always lead to a bounded
change in the output signal. Nonlinear systems, however, may behave chaot-
ically, that is, very small changes in the input leading to completely different
behavior!
We start the chapter with a discussion of the effects of small nonlineari-
ties on otherwise linear systems. Next we discuss several nonlinear filters,
a term that is definitely an oxymoron. We define& a filter as a linear and
time-invariant system, so how can there be a nonlinear filter? Well, once
again, we are not the kind of people to be held back by our own definitions.
Just as we say delta function, or talk about infinite energy signals, we
allow ourselves to call systems that are obviously not filters, just that.
The mixer and the phase locked loop are two systems that are not filters
due to not being time-invariant. These systems turn out to very important
in signal processing for telecommunications. Our final topic, time warping,
is an even more blatant example of the breakdown of time invariance.
322 NONFILTERS
8.1 Nonlinearities
Lets see what makes nonlinear systems interesting. We start by considering
the simplest possible nonlinearity, a small additive quadratic term, which
for analog signals reads
EA2
y(t) = Asin + 2 + &c3(2wt) (8 .4)
We see here three terms; the first being simply the original unscathed signal,
the other two going to zero as E + 0. The second term is a small DC
component that we should have expected, since cos2 is always positive and
thus has a nonzero mean. The final term is an attenuated replica of the
original signal, but at twice the original frequency! This component is known
as the second harmonic of the signal, and the phenomenon of creating new
frequencies which are integer multiples of the original is called harmonic
generation. Harmonic generation will always take place when a nonlinearity
is present, the energy of the harmonic depending directly on the strength
of the nonlinearity. In some cases the harmonic is unwanted (as when a
nonlinearity causes a transmitter to interfere with a receiver at a different
frequency), while in other cases nonlinearities are introduced precisely to
obtain the harmonic.
We see here a fundamental difference between linear and nonlinear sys-
tems. Time-invariant linear systems are limited to filtering the spectrum of
the incoming signal, while nonlinear systems can generate new frequencies.
What would have happened had the nonlinearity been cubic rather than
quadratic?
y(t) = z(t) + a3(t)
8.1. NONLINEARITIES 323
You can easily find that there is third harmonic generation (i.e., a signal
with thrice the original frequency appears from nowhere). A fourth order
nonlinearity
y(t) = 5(t) + m4(t)
will generate both second and fourth harmonics (see equation (A.33)); and
nth order nonlinearities generate harmonics up to order n. Of course a gen-
eral nonlinearity that can be expanded in a Taylor expansion
y(t) = z(t) + 62x2(t) + eg3(t) + E4X4(Q + ** * (8 . 5)
m = Al cos(qt) + A2 cos(w2t)
+A; cos2 (wl t) + A; cos2 (w2t)
+2A1A2 cos(qt) cos(w2t)
= Al cos(qt) + A2 cos(w2t)
+A: cos2 (wl t) + A; cos2 (w2t)
+A& cos ((WI + wz)t)
we see harmonic generation for both frequencies, but there is also a new
nonlinear term, called the inter-modulation product, that is responsible for
the generation of sum and difference frequencies. Once again we see that
nonlinearities cause energy to migrate to frequencies where there was none
before.
More general nonlinearities generate higher harmonics plus more com-
plex intermodulation frequencies such as
4 + 2w2, 24 + w2,
pw - w21, pJJ2 - w,
EXERCISES
8.1.1 Show exactly which harmonics and intermodulation products are generated
by a power law nonlinearity y(t) = z(t) + &(t).
8.1.2 Assume that the nonlinearity is exponential y(t) = z(t) + eect) rather than
a power law. What harmonics and intermodulation frequencies appear now?
0 1x1< 8
y=C&)= 2 else (8.7)
{
The center clipper is also obviously nonlinear, and although at first sight
its purpose is hard to imagine, it has several uses in speech processing. The
first relates to the removal of unwanted zero crossings. As we will see in
Section 13.1 there are algorithms that exploit the number of times a signal
crosses the time axis, and/or the time between two such successive zero
crossings. These algorithms work very well on clean signals, but fail in the
8.2. CLIPPERS AND SLICERS 325
EXERCISES
8.2.1 Apply a center clipper with a small threshold to clean sampled speech. Do
you hear any effect? What about noisy speech? What happens as you increase
the threshold? At what point does the speech start to sound distorted?
8.2.2 Determine experimentally the type of harmonic generation performed by the
clipper and the center clipper.
8.2.3 There is a variant of the center clipper with continuous output as a function
of input, but discontinuous derivative. Plot the response of this system. What
are its advantages and disadvantages?
8.2.4 When a slicer operates on sampled values a question arises regarding values
exactly equidistant between two integer values. Discuss possible tactics.
8.2.5 A resetting filter is a nonlinear system governed by the following equations.
Yn = Xn+Wn
-8 yn < -8
rn = ; IYnkQ
1 Yn > Q
%a = $/n-l -rn-1
Explain what the resetting filter does and how it can be used.
A
A . __ _.
__
______ __
_.__ __
___ __
__
__
___. _.
h
I
Figure 8.1: Comparison of a linear filter with a median filter. In (A) we have the orig-
inal corrupted square wave signal; in (B) the signal has been filtered using a symmetric
noncausal FIR low-pass filter; and in (C) we see the effect of a median filter.
corrupted signal values closer to their correct levels, but also changes sig-
nal values that were not corrupted at all. In particular, low-pass filtering
smooths sharp transitions (making the square wave edges less pronounced)
and disturbs the signal in the vicinity of the outlier. The closer we wish the
outlier to approach its proper level, the stronger this undesirable smoothing
effect will be.
An alternative to the low-pass filter is the median filter, whose effect is
seen in Figure 8.1.C. At every time instant the median filter observes signal
values in a region around that time, similar to a noncausal FIR filter. How-
ever, instead of multiplying the signal values in this region by coefficients,
the median filter sorts the signal values (in ascending order) and selects me-
dian, i.e., the value precisely in the center of the sorted buffer. For example,
if a median filter of length five overlaps the values 1,5,4,3,2, it sorts them
into 1,2,3,4,5 and returns 3. In a more typical case the median filter over-
laps something like 2,2,2,15,2, sorts this to 2,2,2,2,15 and returns 2; and
at the next time instant the filter sees 2,2,15,2,2 and returns 2 again. Any
isolated outlier in a constant or slowly varying signal is completely removed.
Why doesnt a median filter smooth a sharp transition between two
constant plateaus ? As long as more than half the signal values belong to
one side or the other, the median filter returns the correct value. Using an
odd-order noncausal filter ensures that the changeover happens at precisely
the right time.
What happens when the original signal is not constant? Were the lin-
early increasing signal . . . 1,2,3,4,5,6,7,8,9,10.. . to become corrupted to
328 NONFILTERS
EXERCISES
8.3.1 Prove that the median filter is not linear.
8.3.2 Median filtering is very popular in image processing. What properties of com-
mon images make the median filter more appropriate than linear filtering?
8.3.3 The conditional median filter is similar to the median filter, but only replaces
the input value with the median if the difference between the two is above
a threshold, otherwise it returns the input value. Explain the motivation
behind this variant.
8.3.4 Graphically explain the names dilation, erosion, opening, and closing by con-
sidering 0, l-valued signals.
8.3.5 Explain how morphological operations are implemented for image processing
of binary images (such as fax documents). Consider kernels of different
shapes, such as a 3*3 square and a 5-pixel cross. Program the four operations
and show their effect on simple images.
which is equivalent to a single FIR filter with coefficients equal to the con-
volution g t h.
In order for a cascaded system to be essentially different from its con-
stituents we must introduce nonlinearity. Augmenting the FIR filter with a
330 NONFILTERS
Xn
Yn
Figure 8.2: A general nonlinear two-layer feedforward system. Although not explicitly
shown, each connection arc represents a weight. NL stands for the nonlinearity, for exam-
ple, the sgn or tanh function.
hidden values, and finally activate one of 10 output units, thereby expressing
its opinion as to the digit that was uttered. Since humans can perform
this task we are confident that there is some system that can implement
the desired function from input samples to output logical values. Since the
aforementioned theorem states that (assuming a sufficient number of hidden
units) three-layer MLPs can implement arbitrary systems, there must be a
three-layer MLP that imitates human behavior and properly classifies the
spoken digits.
How are MLP systems designed? The discussion of this topic would lead
us too far astray. Suffice it to say that there are training algorithms that
when presented with a sufficient amount of data can accomplish the required
system identification. The most popular of these algorithms is backpropaga-
tion, (backprop) which iteratively presents an input, computes the present
output, corrects the internal weights in order to decrease the output error,
and then proceeds to the next input-output pair.
How many hidden units are needed to implement a given system? There
are few practical rules here. The aforementioned theorem only says that
there is some number of hidden units that allows a given system to be
emulated; it does not inform us as to the minimum number needed for all
specific cases, or whether one, two, or three layers are needed. In practice
these architectural parameters are often determined by trial and error.
332 NONFILTERS
EXERCISES
8.4.1 Using linear threshold units we can design systems that implement various
logic operations, where signal value 0 represents false and 1 true. Find
parameters wi, ~2, and cp such that y = 0 (~1x1 + ~2x2 - cp) implements
the logical AND and logical OR operations. Can we implement these logical
operations with linear systems?
8.4.2 Of the 16 logical operations between two logical variables, which can and
which cant be implemented?
8.4.3 Find a multilayer system can implements XOR.
8.4.4 What is the form of curves of equal output for the perceptron of equa-
tion (8.9)? What is the form of areas of the same value of equation (8.8)?
What is the form of these areas for multilayer perceptrons formed by AND
or OR of different simple perceptrons? What types of sets cannot be imple-
mented? How can this limitation be lifted?
8.4.5 What are the derivatives of the sigmoid functions (equations (8.11) and
(8.9))? Show that a(x) = c(x) (1 - g(x)). Can you say something similar
regarding the tanh sigmoid?
8.5 Mixers
A mixer is a system that takes a band-pass signal centered around some fre-
quency fo, and moves it along the frequency axis (without otherwise chang-
ing it) until it is centered around some other frequency fi. Some mixers
may also invert the spectrum of the mixed signal. In Figures 8.3 and 8.4 we
depict the situation in stylized fashion, where the triangular spectrum has
become prevalent in such diagrams, mainly because spectral inversions are
obvious. In older analog signal processing textbooks mixing is sometimes
called heterodyning. In many audio applications the term mixing is used
when simple weighted addition of signals is intended; thus when speaking
to audio professionals always say frequency mixing when you refer to the
subject of this section.
8.5. MIXERS 333
Figure 8.4: The effect of mixing a narrow-band analogsignal with spectral inversion. In
(A) we seethe spectrum of the original signal centered at frequency fe, and in (B) the
mixed and inverted signal at frequency fr. Note how the triangular spectral shapeassists
in visualizing the inversion.
Many receivers start by mixing the received signal down by fC to obtain the
simpler form of equation (4.65)
from which the amplitude and phase can be recovered using the techniques
of Section 4.12.
The phase is intimately connected with the carrier frequency so that the
mixing stage is obviously required for proper phase recovery. Even were there
to be a mixer but its frequency to be off by some small amount Af the phase
would be misinterpreted as 2nA f t+#(t) along with the unavoidable jumps of
27r. The amplitude signal is apparently independent of the carrier frequency;
can we conclude that no mixer is required for the recovery of amplitude-
modulated signals? No, although mistuning is much less destructive. The
reason a mixer is required is that the receiver sees many possible transmitted
signals, each with its own carrier frequency fc. Isolation of the desired signal
is accomplished by downmixing it and injecting it into a narrow low-pass
filter. The output of this filter now contains only the signal of interest and
demodulation can continue without interference. When you tune an AM or
FM radio in order to hear your favorite station you are actually adjusting
a mixer. Older and simpler receivers allow this downmix frequency to be
controlled by a continuously rotatable (i.e., analog) knob, while more modern
and complex receivers use digital frequency control.
Telephone-quality speech requires less than 4 KHz of bandwidth, while
telephone cables can carry a great deal more bandwidth than this. In the
interest of economy the telephone network compels a single cable to simul-
taneously carry many speech signals, a process known as multiplexing. It is
obvious that we cannot simply add together all the signals corresponding to
the different conversations, since there would be no way to separate them
at the other end of the cable. One solution, known as Frequency Domain
Multiplexing (FDM), consists of upmixing each speech signal by a different
offset frequency before adding all the signals together. This results in each
signal being confined to its own frequency band, and thus simple band-pass
filtering and mixing back down (or mixing first and then low-pass filter-
ing) allows complete recovery of each signal. The operation of building the
FDM signal from its components involves upmixing and addition, while the
extraction of a single signal requires downmixing and filtering.
Sometimes we need a mixer to compensate for the imperfections of other
mixers. For example, a modem signal transmitted via telephone may be
upmixed to place it in a FDM transmission, and then downmixed before
8.5. MIXERS 335
s(t) = AeiWt
w i(U-Ld)t
= AeiWtei(U-W)t
= AeiWt
= s/(t)
Note that the multiplying signal is sinusoidal at the frequency shift frequency
and thus the system is not time-invariant.
336 NONFILTERS
s(t) = c Akeiwkt
k
When a signal is mixed down in frequency until it occupies the range from
DC up to its bandwidth, it is said to have been downmixed to low-pass.
When we go even further and set the signals center frequency to zero, we
have downmixed to zero.
So it seems that mixing is actually quite simple. The problems arise when
we try to mix real-valued signals rather than analytic ones, or digital signals
rather than analog ones. To illustrate the problems that arise, consider first
the mixing of real signals. Since real signals have symmetric spectra, we have
to look at both positive and negative frequencies to understand the whole
story.
A I I I
-30 0 f0
Figure 8.5: A real signal at frequency fo, whose spectrum is depicted in (A), is moved
to frequency fl by complex mixing. When a signal is multiplied by a complex exponential
all frequency components are shifted in the same direction, as seen in (B).
B I I I
-fl 0 fl
Figure 8.6: A real signal at frequency fe, whose spectrum is depicted in (A), is moved to
frequency fr by real mixmg. When a real signal is multiplied by a real sinusoid its positive
and negative frequency approach each other, as seen in (B).
I I I
-fP fl
Figure 8.7: A real signal after destructive downmixing. Once the spectrum overlaps itself
information is lost.
I I I I I I I
In I I I I I I
Figure 8.8: A digital mixer. Here a real-valued digital signal is mixed by a complex
digital exponential.
EXERCISES
8.5.1 Diagram all the cases of mixing real or complex digital signals by real or
complex oscillators.
8.5.2 We originally claimed that a mixer generates new frequencies due to its being
time-invariant but linear. Afterward when discussing its analog implementa-
tion we noted that the product is generated by a time-invariant nonlinearity.
Reconcile these two statements.
8.5.3 There are two techniques to mix a real signal down to zero. The signal can
be converted to the analytic representation and then multiplied by a complex
exponential, or multiplied by the same complex exponential and then low-
pass filtered. Demonstrate the equivalence of these two methods. What are
the practical advantages and disadvantages of each approach?
clock. The receiver need only set its clock precisely according to this sinusoid
and the discrepancy problem vanishes. This operation of matching clocks is
called synchronization, often shortened to synching (pronounced sinking)
or synching up. Synchronization of the receivers clock to the transmitters
has to be maintained continuously; even if properly initially matched, non-
synched clocks will drift apart with time, introducing bit slips and insertions.
The accurate synching up of the receivers clock depends critically on
obtaining a clean signal from the transmitter. A naive DSP approach would
be to use a very narrow-band band-pass filter centered on f to recover
the clock signal and reject as much noise as possible. Such an attempt is
doomed to failure since we dont know f (were we to know f there wouldnt
be anything to do). Setting an extremely sharp band-pass filter centered on
the receivers estimate f may leave the true f outside the filter bandwidth.
Of course we could use a wider filter bandwidth, but that would increase the
noise. What we really need is to find and track the received signals center
frequency. That is what the PLL does.
In order to build a PLL we first need some basic building blocks. The
first is traditionally called a Voltage-Controlled Oscillator (VCO). Like an
ordinary oscillator the VCO outputs a real sinusoid, but unlike the oscillators
we have seen before the VCO has an input as well. With zero input the
VCO oscillates at its natural frequency wg, but with nonzero input z(t)
the VCO outputs instantaneous frequency changes to wg + v(t). It is now
straightforward to express the VCO output y(t) in terms of its input z(t).
The analog VCO is thus controlled by the voltage at its input, and hence its
name; the digital version should properly be called a Numerically-Controlled
Oscillator (NCO), but the name VCO is often used even when no voltages
are evident.
The next basic subsystem has two inputs where it expects two pure
sinusoids; its output is proportional to the difference in frequency between
the two. There are many ways to implement this block, e.g., one could use
two frequency demodulators (Section 4.12) and an adder with one input
negated. A more devious implementation uses a mixer, a special notch filter
and an amplitude demodulator. The VCO output is used to downmix the
input to zero; the mixer output is input to a filter with gain 1~1 so that
when the input frequency matches the VCO there is no output, while as the
deviation increases so does the amplitude; finally the amplitude demodulator
outputs the desired frequency difference.
340 NONFILTERS
x : frequency L
F integrator - vco =- Y
difference 1r
+
Figure 8.9: The frequency-locked loop (FLL). The output is a sinusoid that tracks the
frequency of the input signal.
Using the two special blocks we have defined so far we can already make a
first attempt at a system that tracks sinusoidal components (see Figure 8.9).
We will call this system a Frequency-Locked Loop (FLL), as its feedback
loop causes it to lock onto the frequency of the input signal. Consider what
happens when a sinusoid with frequency w > wg is applied to the input
(previously zero). At first the frequency difference block outputs w - wg,
and were this to be input to the VCO it would change its frequency from
wo to wo + w - wo = w. Unfortunately, this correct response is just an
instantaneous spike since the difference would then become zero and the
VCO would immediately return to its natural frequency. The only escape
from this predicament is to integrate the difference signal before passing it
to the VCO. The integral maintains a constant value when the difference
becomes zero, forcing the VCO to remain at w.
The FLL can be useful in some applications but it has a major drawback.
Even if the input is a pure sinusoid the FLL output will not in general
precisely duplicate it. The reason being that there is no direct relationship
between the input and output phases. Thus in our bit rate recovery example
the FLL would accurately report the rate at which the bits are arriving, but
could not tell us precisely when to expect them. In order to track the input
signal in both frequency and phase, we need the more sensitive phase-locked
loop. Looking carefully at our FLL we see that the frequency difference
is integrated, returning a phase difference; the PLL replaces the frequency
difference block of the FLL with an explicit phase difference one.
The phase difference subsystem expects two sinusoidal inputs of approx-
imately the same frequency and outputs the phase difference between them.
One could be built similarly to the frequency difference block by using two
phase demodulators and an adder with one input negated; however, there are
approximations that are much easier to build for analog signals and cheaper
to compute for digital ones. The most common approximate difference block
shifts the phase of one input by 90 and multiplies the two signals.
8.6. PHASE-LOCKED LOOPS 341
It the low-pass filters the output to remove the double frequency component.
The filtered product is proportional to
sin@1 - 42) N 41 - 42
Figure 8.10: Characteristics of three phase difference blocks. The ideal phase differ-
ence subsystem has its output vary like a sawtooth as a function of the phase difference
A4=&-&.Th e simple product subsystem has sinusoidal characteristic, while the xor
comparator has a triangular one. The important feature of all these blocks is that for small
phase differences the characteristic is linear.
342 NONFILTERS
Figure 8.11: The phase-locked loop, or PLL. The output is a sinusoid that tracks the
phase of the input signal.
No matter how we build the phase detector, the proper way to use it
is depicted in Figure 8.11. If the input is truly a sinusoid of the VCOs
natural frequency, the phase difference output causes the VCO frequency to
momentarily increase in order to catch up with the input or decrease to let
the input catch up. The PLL is even more useful when the input is noisy.
In this case the phase difference varies erratically but the low-pass filter
smooths the jumps so that the VCO only tracks the average input phase.
Quite noisy signals can be applied provided the low-pass filter is sufficiently
narrow.
What if the input frequency doesnt equal the VCO natural frequency?
Small constant frequency differences can be thought of as constantly chang-
ing phase differences, and the phase corrections will cause the VCO to oscil-
late at the average input frequency. If the frequency difference is larger than
the low-pass filter bandwidth the VCO will receive zero input and remain at
its natural frequency, completely oblivious to the input. For input frequen-
cies in the capture range the VCO does get some input and starts moving
toward the input frequency. The difference then further decreases, allowing
more energy through the filter, and the PLL snaps into lock. Once locked
the phase difference is DC and completely passed by the filter, thus main-
taining lock. If the input frequency varies the VCO automatically tracks it
as long as it remains in the tracking range.
The low-pass filter used in the PLL is usually of the IIR type. When the
phase detector is of the product type a single low-pass filter can be used
both for the filtering needed for the PLLs noise rejection and for rejecting
the double frequency component. When the double frequency rejection is
not required we may be able to skip the filter altogether. In this case there
is still a feedback path provided by the PLL architecture, and so the PLL
is said to be of first order. If the IIR filter has a single pole the additional
pole-like behavior leads us to say that the PLL is of second order. Higher
orders are seldom used because of stability problems.
8.7. TIME WARPING 343
EXERCISES
8.6.1 Simulate the FLLs frequency behavior by assuming a VCO natural fre-
quency, inputting some other frequency, and using simple addition to in-
tegrate. Simulate a slowly varying input frequency. How far can the input
frequency be from the natural frequency?
8.6.2 Adding a clipping amplifier between the frequency difference and the integra-
tor of the FLL makes the FLL have two operating regions, acquisition and
tracking. Analyze the behavior of the system in these two regions.
8.6.3 Compare the PLL and FLL from the aspects of frequency acquisition range,
steady state frequency, and steady state phase error.
8.6.4 Explain how to use the PLL to build a frequency synthesizer, that is, an
oscillator with selectable accurate frequency.
8.6.5 What effect does decreasing a PLLs low-pass filter bandwidth have on the
capture range, the acquisition time, and robustness to noise?
nl I
i
SI I
e
S
#S
01 I
4 I
Now we fill in each of the blank squares with the minimal cost to get
to that square. The bottom-left square is initialized to zero since we start
there, and all the rest of the squares will get values that can be computed
recursively. We finally arrive at the top right square, and the value there
will be the total cost, namely the Levenshtein distance.
The recursive step involves comparing three components. One can enter
a square from its left, corresponding to a deletion from the dictionary word,
by taking the value to its left and adding one. One can enter a square
from underneath, corresponding to an insertion into the dictionary word,
by taking the value underneath it and incrementing. Finally, one can enter
a square from the square diagonally to the left and down; if the letter in
the dictionary word at the bottom of the column is the same as the letter
in the string at the beginning of the row, then there is no additional cost
and we simply copy the value from the diagonal square. If the letters differ,
a substitution is needed and so we increment the value diagonally beneath.
In this fashion each square gets three possible values, and we always choose
the minimum of these three.
Lets try this out on our example. We start with the table from above,
initialize the bottom left square, and trivially fill in the lowest row and
leftmost column.
346 NONFILTERS
rlll I I I
PO123456789
Oprocessing
Now we can continue filling in the entire table, and find (as we previously
discovered in a rather undisciplined fashion) that the Levenshtein distance
is indeed 3.
n8766544435
i7655433355
~6544323445
e5433233345
~4322222345
~3211223456
02101234567
r1012345678
PO123456789
From the table we can discover more than simply the total distance,
we can actually reconstruct the optimal sequence of operations. Indeed the
optimal set of deletions, insertions, and substitutions pops out to the eye as
the path of minimal cost through the table. At first there seem to be many
optimal paths, but quite a few of these correspond to making a deletion
and insertion instead of some substitution. The true path segments are the
ones that contributed the minimal cost transitions. Thus to find the true
path you start at the end point and retrace your steps backward through
the table; we can save redundant computation by storing in each square
not only its cost but the previous square visited. The only ambiguities that
remain correspond to squares where more than one transition produced the
same minimal cost; in our example changing the dictionary c to the incorrect
8.7. TIME WARPING 347
EXERCISES
8.7.1 The game of doublets was invented in 1879 by Lewis Carroll (the mathe-
matician Charles Lutwidge Dodgson 1832-1898). The aim of the game is to
convert a word into a related word in the minimal number of substitution
steps; However, each step must leave an actual word. For example, we can
change hate into love, in three steps in the following way: hate have lave
love. Show how to make a cat into a dog in three steps, how an ape can
evolve into a man in five steps, and how to raise four to five by a seven step
procedure. four foul fool foot fort fore fire five. How many steps
does it take to drive the pig into the sty?
8.7.2 In more complex implementations of spelling checkers further types of errors
may be added (e.g., reversal of the order of two letters). Can the dynamical
programming algorithm still be used to determine the Levenshtein distance?
8.7.3 An alternative method for comparing time-warped signals is the Markov
model approach. Here we assume that the signal is generated by a Markov
model with states 01,02 . . . 0~. When the model is in state 0, it has prob-
ability u~,~ of staying in the same state, probability u~,~+I of transitioning
to state Om+r, and probability u~,~+z of skipping over state Om+r directly
to state Om+2. When the model is in state 0, it outputs a characteristic
signal segment sm. Write a program that simulates a Markov model and run
it several times. Do you see how the time warping arises?
348 BIBLIOGRAPHICAL NOTES
8.7.4 An extension to the above model is the HiddenMarkov Model (HMM). The
HMM states are hidden since they do not uniquely correspond to an output
signal segment; rather when the model is in a state 0, it has probability
b,l of outputting signal sl. Extend the program of the previous exercise to
generate HMM signals. Why is the HMM more realistic for speech?
Bibliographical Notes
Although there are a lot of books that deal with things that are not filters,
there are very few such that happen to treat signal processing.
Median and morphological filters are mostly discussed in books on image
processing, but see [68, 258, 1571.
Multilayer perceptrons were introduced in [225], and popularized in the
books by the same authors [168, 1691, although the basic idea had beed
previously discovered by several researchers. A popular short introduction
is [150].
Phase-locked loops are usually discussed in books on digital communi-
cations, e.g., [242, 1991.
Time warping and HMM are discussed in texts on speech recognition,
e.g., [204, 1761.
9
Correlation
Our study of signal processing systems has been dominated by the concept
of convolution, and we have somewhat neglected its close relative the cor-
relation. While formally similar (in fact convolution by a symmetric FIR
filter can be considered a correlation as well), the way one should think
about the two is different. Convolution is usually between a signal and a
filter; we think of it as a system with a single input and stored coefficients.
Crosscorrelation is usually between two signals; we think of a system with
two inputs and no stored coefficients. The difference may be only in our
minds, but nonetheless this mind-set influences the way the two are most
often used.
Although somewhat neglected we werent able to get this far without
mentioning correlations at all. We have already learned that crosscorrela-
tion is a measure of similarity between two signals, while autocorrelation is
a measure of how similar a signal is to itself. In Section 5.6 we met the auto-
correlation for stochastic signals (which are often quite unlike themselves),
and in Section 6.13 we used the crosscorrelation between input and output
signals to help identify an unknown system.
Correlations are the main theme that links together the present chapter.
We first motivate the concept of correlation by considering how to compare
an input signal to a reference signal. We find that the best signal detector
is the correlator. After formally defining both crosscorrelation and autocor-
relation and calculating some examples, we prove the important Wiener-
Khintchine theorem, which relates the autocorrelation to the power spectral
density (PSD).
Next we compare correlation with convolution and discover that the op-
timal signal detector can be implemented as a matched filter. The matched
filter was invented for radar and a digression into this important applica-
tion is worthwhile. The matched filter is good for signal detection, but for
cleaning up a partially unknown signal we need the Wiener filter, which is
also based on correlations.
349
350 CORRELATION
From these examples we see that comparison and detection are essen-
tially the same. The simplest detector is implemented by comparing the out-
put of a comparator to a threshold. Complex detectors may employ more
sophisticated decision elements, but still require the basic comparison mech-
anism to function.
Signal detection and comparison are nontrivial problems due to the pres-
ence of noise. We know how to build filters that selectively enhance defined
frequency components as compared to noise; but how do we build a system
that selectively responds to a known but arbitrary reference signal? Our
first inclination would be to subtract the input signal sn from the desired
reference rn, thus forming an error signal en = rn - sn. Were the error signal
to be identically zero, this would imply that the input precisely matches the
reference, thus triggering the signal detector or maximizing the output of
the signal comparator. However, for an input signal contaminated by noise
%a = rn + vn, we can not expect the instantaneous error to be identically
zero, but the lower the energy of the error signal the better the implied
match. So a system that computes the energy of the difference signal is a
natural comparator.
This idea of using a simple difference is a step in the right direction,
but only the first step. The problem is that we have assumed that the input
signal is simply the reference signal plus additive noise; and this is too strong
an assumption. The most obvious reason for this discrepancy is that the
amplitude of the input signal is usually arbitrary. The strength of a radar
return signal depends on the cross-sectional area of the target, the distance
from the transmitter to the target and the target to the receiver, the type
and size of the radar antenna, etc. Communications signals are received
after path loss, and in the receiver probably go through several stages of
analog amplification, including automatic gain control. A more reasonable
representation of the input signal is
sn= Am + U,
where A is some unknown gain parameter.
In order to compare the received signal sn with the reference signal rn it
is no longer sufficient to simply form the difference; instead we now have to
find a gain parameter g such that rn - gsn is minimized. We can then use
the energy of the resulting error signal
En = m$rn - $I%>
as the final match criterion. How can we find this g? Assuming for the
352 CORRELATION
moment that there is no noise, then for every time YZwe require
7n 1
g =-=-
Sn A (9.1)
in addition to the weaker constraint that the error energy be zero.
C( rn - gqJ2 = 0 (9.2)
n
cn
r~-2gCTnSn+g2CS~=0
n n
C r2 Er
g2=-=F
c n sn s
Gs = l/z% (9.4)
in the absence of noise. When the input signal does not precisely match the
reference, due to distortion or noise, we have lCrSI < +dm. The cross-
correlation CTSis thus an easily computed quantity that compares the input
signal to the reference, even when the amplitudes are not equal. A compara-
tor can thus be realized by simply computing the correlation, and a signal
detector can be implemented by comparing it to dm (e.g., requiring
Unfortunately we have not yet considered all that happens to the ref-
erence signal before it becomes an input signal. In addition to the additive
noise and unknown gain, there will also usually be an unknown time shift.
For communications signals we receive a stream of signals to compare, each
offset by an unknown time delay. For the radar signal the time delay derives
9.1. SIGNAL COMPARISON AND DETECTION 353
from the round-trip time of the signal from the transmitter to the target
and back, and is precisely the quantity we wish to measure. When there is
a time shift, a reasonable representation of the input signal is
Sn- - Arn+m + vn Vn
where A is the gain and m < 0 the time shift parameter.
In order to compare the received signal sn with the reference signal rn
we can no longer simply compute a single crosscorrelation; instead we now
have to find the time shift parameter m such that
EXERCISES
9.1.1 Formulate the concept of correlation in the frequency domain starting from
spectral difference and taking into account an arbitrary gain of the spectral
distribution. What happens if we need to allow an arbitrary spectral shift?
9.1.2 Give a complete algorithm for the optimal detection of a radar return sn given
that the transmitted signal rn was sent at time Ti, returns are expected to
be received before time T2, and the correlation is required to be at least y.
Note that you can precompute E,. and compute Es and CTB(rn)in one loop.
9.1.3 Design an optimal detector for the V.34 probe signal introduced in exer-
cise 2.6.4. The basic idea is to perform a DFT and implement a correlator
in the frequency domain by multiplying the spectrum by a comb with 21
pass-bands (of suitable bandwidth). However, note that this is not indepen-
dent of signal strength. You might try correcting this defect by requiring the
correlation to be over 80% of the total signal energy, but this wouldnt work
properly since, e.g., answer tone (a pure 2100 Hz tone) would trigger it, be-
ing one of the frequenciesof the probe signal. What is wrong? How can this
problem be solved?
354 CORRELATION
Definition: crosscorrelation
The crosscorrelation between two real signals x and y is given by
Definition: autocorrelation
The autocorrelation of a real signal s is given by
cs (7)
~(7) = m A D cS(m) = -cs (f-4 (9.7)
cs (0)
These definitions are consistent with those of Section 5.6 for the case
of stationary ergodic signals. In practice we often approximate the autocor-
relation of equation (5.22) by using equation (9.6) but with the sum only
over a finite amount of time. The resulting quantity is called the empirical
autocorrelation. The correlation is also somewhat related to the covariance
matrix of vector random variables, and strongly related to the convolution,
as will be discussed in the next section.
Before discussing properties of the correlations, lets try calculating a
few. The analog rectangular window
s(t)1=
1 ItI < 1
0 else
9.2. CROSSCORRELATION AND AUTOCORRELATION 355
Figure 9.1: The autocorrelation of an analog rectangularly shaped signal. In (A) the
signal is depicted while the autocorrelation is in (B). Note that the autocorrelation is
symmetric and has its maximal value at the origin.
~~(7)
- Jm
--oo
s*(t)s(t - T)dt A D C&n) f E s;snmm (9 .9)
72=--00
triangular wave of the same period. This too is quite general-the autocor-
relation of a periodic signal is periodic with the same period; and since the
lag-zero autocorrelation is a global maximum, all lags that are multiples of
the period have globally maximal autocorrelations. This fact is precisely the
secret behind using autocorrelation for determining the period of a periodic
phenomenon. One looks for the first nonzero peak in the autocorrelation as
an indication of the period. The same idea can be used for finding Fourier
components as well; each component contributes a local peak to the auto-
correlation.
As our final example, lets try a digital autocorrelation. The signal b, is
assumed to be zero except for n = 1. . . 13 where it takes on the values f 1.
. ..0.0,+1,+1,+1,+1,+1,-1,-1,+1,+1,-1,+1,-1,+1,0,0 ,... (9.10)
Its autocorrelation is easily computed to be C(0) = 13, C(m) = 0 for odd r-n
in the range -13 < m < 13, C(m) = 1 for even nonzero m in this range, and
all other autocorrelations are zero. We see that the autocorrelation is indeed
maximal at m = 0 and symmetric, and in addition the highest nonzero-
lag correlations are only 1. Signals consisting of 44 values with this last
property (i.e., with maximal nontrivial autocorrelation of v or less) are
called Barker codes, and are useful for timing and synchronization. There is
no known way of generating Barker codes and none longer than this one are
known.
The definitions for autocorrelation or crosscorrelation given above in-
volve integrating or summing over all times, and hence are not amenable
to computation in practice. In any case we would like to allow signals to
change behavior with time, and thus would like to allow correlations that
are defined for finite time durations. The situation is analogous to the prob-
lem that led to the definition of the STFT, and we follow the same tactic
here. Assuming a rectangular window of length N, there are N terms in the
expression for the zero lag, but only N - 1 terms contribute to the lag 1
correlation slsa + szsr + . . . + SN.-~SN-~, and only N - m terms in the lag
m sum. So we define the short-time autocorrelation
&x%-m (9.11)
where now the zero lag is the power rather than the energy. This quantity is
often called the unbiased empirical autocorrelation when it is looked upon
as a numerical estimate of the full autocorrelation.
9.3. THE WIENER-KHINTCHINE THEOREM 357
EXERCISES
9.2.1 What is the connection between autocorrelation defined here for determin-
istic signals and the autocorrelation we earlier defined for stochastic signals
(equation (5.22))?
9.2.2 What is the crosscorrelation between a signal s(t) and the impulse s(t)?
9.2.3 Compute and draw the crosscorrelation between two analog rectangular sig-
nals of different widths.
9.2.4 Compute and draw the crosscorrelation between two analog triangular sig-
nals.
9.2.5 Show that CyZ(m) = C&,(-m).
9.2.6 Prove that the autocorrelation is symmetric and takes its maximum value at
the origin, where it is the energy. Show that IcZy(m)( 5 1.
9.2.7 Can you find Barker codes of length 5, 7, and ll? What are their autocorre-
lations?
9.2.8 What is the proper generalization of crosscorrelation and autocorrelation to
complex signals? (Hint: The autocorrelation should be phase independent.)
9.2.9 Prove that the autocorrelation of a periodic signal is periodic with the same
period.
9.210 Prove that zero mean symmetric signals have zero odd lag autocorrelations.
9.2.11 Assume gn = x,+1. What are the connections between CZy(m), C,(m) and
c,(m)?
9.2.12 Derive the first few autocorrelation values for sn = Asin(wn + 4).
9.2.13 Generalize the previous exercise and derive the following expression for the
general autocorrelation of the sinusoid.
A2
G(m) = (wn+m) = yj- cos(wm)
The PSD of a signal is the absolute square of its FT, but it is also can
be considered to be the FT of some function. Parsevals relation tells us
that integrating the PSD over all frequencies is the same as integrating the
square of the signal over all times, so it seems reasonable that the iFT of
the PSD is somehow related to the square of the signal.
Could it be that the PSD is simply the FT of the signal squared? The DC
term works because of Parseval, but what about the rest? We dont have
to actually integrate or sum to find out since we can use the connection
between convolution and FT of a product FT(zy) = X * Y (equation (4.18)
or (4.46)). Using the signal s for both z and y we see that the FT of s2(t)
is S*S = $S(w - Q)S(Q)dO, which is not quite the PSD lSl2 = S*S =
S(-w)S(w) (for real signals), but has an additional integration. We want to
move this integration to the time side of the equation, so lets try s *s. From
equation (4.19) or (4.47) we see that the FT of s * s is S2(w) which is even
closer, but has both frequency variables positive, instead of one positive and
one negative. So we need something very much like s * s but with some kind
of time variable inversion; that sounds like the autocorrelation!
So lets find the FT of the autocorrelation.
EXERCISES
9.3.1 The period of a pure sinusoid is evident as a peak in the autocorrelation and
hence its frequency is manifested as a peak in the power spectrum. This is
the true basis for the connection between autocorrelation and PSD. What
can you say about the autocorrelation of a general periodic signal? What is
the autocorrelation of the sum of two sinusoidal components? Can you see
the PSD connection?
9.3.2 Express and prove the Wiener-Khintchine theorem for digital signals.
9.3.3 Generalize the Wiener-Khintchine theorem by finding the FT of the cross-
correlation of two signals z(t) and y(t).
EXERCISES
9.4.1 Express the processing gain in decibels when the DFT is performed using a
2m point FFT.
9.4.2 In the text we tacitly assumed the signal frequency to be precisely at a bin
center. If this is not the case a window function 20~ (see Section 13.4) must
be employed. Show that with a window the signal energy is enhanced by
(C, UJ,)~ while the noise energy is increased by C, wi thus resulting in a
processing gain of the ratio of these two expressions.
9.4.3 Build a detector for a signal that consists of the equally weighted sum of two
sinusoids. Is it worthwhile taking the phases into account? What if the signal
is the weighted sum of the two sinusoids?
9.4.4 Extend the technique of the previous exercise and build a DFT-based detector
for a completely general signal.
cup of tea. Usually he would take the kettle from the cupboard, fill it with
water, put it on the fire, and when the water boiled, pour it into a cup and
drop in a tea bag. One day he found that someone had already boiled the
water. He stared perplexed at the kettle and then smiled. He went to the
sink, poured out the boiling water, returned the kettle to the cupboard and
declared triumphantly: The problem has been reduced to one we know how
to solve.
How can we reduce the problem of a signal in colored noise to the one for
which the matched filter is the optimal answer? All we have to do is filter the
contaminated signal xn by a filter whose frequency response is the inverse
of this noise spectrum. Such a filter is called a whitening filter, because it
flattens the noise spectrum. The filtered signal XL = sk + u; now contains
an additive white noise component uk, and the conditions required for the
matched filter to be optimal are satisfied. Of course the reference signal sk
is no longer our original signal s,; but finding the matched filter for sk is
straightforward.
EXERCISES
9.5.1 Create a sinusoid and add Gaussian white noise of equal energy. Recover the
sinusoid by averaging. Experiment with inaccurate registration. Now recover
the sinusoid by a DFT. What advantages and disadvantages are there to this
method? What happens if the frequency is inaccurately known?
9.5.2 Build a matched filter to detect the HPNA 1.0 pulse (see exercise 7.7.4). Try
it out by synthesizing pulses at random times and adding Gaussian noise.
HPNA 1.0 uses PPM where the information is in the pulse position. How
precisely can you detect the pulses time of arrival?
9.5.3 Compare the time domain matched filter with a frequency domain detector
based on the FFT algorithm. Consider computational complexity, processing
delay, and programming difficulty.
our newly acquired knowledge of matched filters we can now present im-
proved radar signals and receivers.
Radar pulses need to have as much energy as possible in order to increase
the probability of being detected, and thus should be long in duration. In
order to increase a radars range resolution we prefer narrow pulses since
its hard to tell when exactly a wide pulse arrives. How can we resolve this
conflict of interests? The basic idea is to use a wide pulse but to modulate it
(Le., to change its characteristics with time). The output of a filter matched
to this modulation can be made to be very short in duration, but containing
all the energy of the original pulse.
To this end some radars vary their instantaneous frequency linearly with
time over the duration of the pulse, a technique known as FM chirp We
demonstrate in Figure 9.2 the improvement chirp can bring in range reso-
lution. The pulse in Figure 9.2-A is unmodulated and hence the matched
filter can do no better than to lock onto the basic frequency. The output of
such a matched filter is the autocorrelation of this pulse, and is displayed in
Figure 9.2.B. Although theoretically there is a maximum corresponding to
the perfect match when the entire pulse is overlapped by the matched fil-
ter, in practice the false maxima at shifts corresponding to the basic period
Figure 9.2: The autocorrelation of pulses with and without chirp. In (A) a pulse with
constant instantaneous frequency is depicted, and its wide autocorrelation is displayed in
(B). In (C) we present a pulse with frequency chirp; its much narrower autocorrelation is
displayed in (D) .
364 CORRELATION
make it difficult to determine the precise TOA. In contrast the chirped pulse
of Figure 9.2.C does not match itself well at any nontrivial shifts, and so
its autocorrelation (Figure 9.2.D) is much narrower. Hence a matched filter
built for a chirped radar pulse will have a much more precise response.
Chirped frequency is not the only way to sharpen a radar pulses auto-
correlation. Barker codes are often used because of their optimal autocorre-
lation properties, and the best way to embed a Barker code into a pulse is
by changing its instantaneous phase. Binary Phase Shift Keying (BPSK),
to be discussed in Section 18.13, is generated by changing a sinusoidal sig-
nals phase by 180, or equivalently multiplying the sinusoid by -1. To use
the 13-bit Barker code we divide the pulse width into 13 equal time inter-
vals, and assign a value f 1 to each. When the Barker code element is +l
we transmit + sin(wt), while when it is -1 we send - sin(&). This Barker
BPSK sharpens the pulses autocorrelation by a factor of 13.
Not all radars utilize pulses; a Continuous Wave (CW) radar transmits
continuously with constant amplitude. How can range be determined if echo
arrives continuously? Once again by modulating the signal, and if we want
constant amplitude we can only modulate the frequency or phase (e.g., by
chirp or BPSK). Both chirp and BPSK modulation are popular for CW
radars, with the modulation sequence repeating over and over again without
stopping. CW radars use LFSR sequences rather than Barker codes for a very
simple reason. Barker codes have optimal linear autocorrelation properties,
while maximal-length LFSR sequences can be shown to have optimal circular
autocorrelation characteristics. Circular correlation is analogous to circular
convolution; instead of overlapping zero when one signal extends past the
other, we wrap the other signal around periodically. A matched filter that
runs over a periodically repeated BPSK sequence essentially reproduces the
circular autocorrelation.
EXERCISES
9.6.1 Plot, analogously to Figure 9.2, the autocorrelation of a pulse with a 13-bit
Barker code BPSK.
9.6.2 What is the circular autocorrelation of the LFSR15 sequence?
9.6.3 What is the difference between coherent and incoherent pulse radars? In what
way are coherent radars better?
9.7. THE WIENER FILTER 365
and if the noise is white then we further define its constant power spectral
density to be Vu = $Y watt per Hz. The overall signal-to-noise ratio is the
ratio of the energies
SNR = 2 (9.12)
Y
but we can define time- and frequency-dependent SNRs as well.
2
I V>l
SNR(t) = -!--- IW >I2
SNW) = Iv(f)lz (9.13)
Iv@)I2
366 CORRELATION
Finally, the observed signal is the sum of the signal plus the noise.
x(t) = s(t) + v(t) X(f) = S(f) + w (9.14)
Well start with the simple case of a relatively pure sinusoid of frequency
fo in white noise. The signal PSD consists of a single narrow line (and
its negative frequency conjugate), while the noise PSD is a constant VO;
accordingly the SNR is 3. What filter will optimally detect this signal
given this noise? Looking at the frequency-dependent SNR we see that the
signal stands out above the noise at fo; so it makes sense to use a narrow
band-pass filter centered on the sinusoids frequency J-J. The narrower the
filter bandwidth BW, the less noise energy is picked up, so we want BW
to be as small as possible. The situation is depicted in Figure 9.3.A where
we see the signal PSD represented as a single vertical line, the noise as a
horizontal line, and the optimum filter as the smooth curve peaked around
the signal. The signal-to-noise ratio at the output of the filter
(9.15)
is greater than that at the input by a factor of & . For small B W this is a
great improvement in SNR and allows us to detect the reference signal even
when buried in very high noise levels.
Now lets complicate matters a bit by considering a signal with two equal
spectral components, as in Figure 9.3.B. Should we use a filter that captures
both spectral lines or be content with observing only one of them? The two-
component filter will pass twice the signal energy but twice the noise energy
as well. However, a filter that matches the signal spectrum may enhance the
time-dependent SNR; the two signal components will add constructively at
some time, and by choosing the relative phases of the filter components we
can make this peak occur whenever we want. Also, for finite times the noise
spectrum will have local fluctuations that may cause a false alarm in a single
filter, but the probability of that happening simultaneously in both filters
is much smaller. Finally, the two-component filter can differentiate better
between the desired signal and a single frequency sinusoid masquerading as
the desired signal.
Were one of the frequency components to be more prominent than the
other, we would have to compensate by having the filter response H(f) as
9.7. THE WIENER FILTER 367
IH( IH(
4 A
I
A
I i -f I \ ) \ -f
IH( IH(
A A
I I
B
I I \ -f
Figure 9.3: Expected behavior of an optimum filter in the frequency domain. In all the
figures we see the PSD of the reference signal and noise, as well as the Wiener filter. The
various cases are discussed in the text.
depicted in Figure 9.3.C. This seems like the right thing to do, since such a
filter emphasizes frequencies with high SNR. Likewise Figure 9.3.D depicts
what we expect the optimal filter to look like for the case of two equal signal
components, but non-white noise.
How do we actually construct this optimum filter? Its easier than it
looks. From equation (9.14) the spectrum at the filter input is S(f) + V(f),
so the filters frequency response must be
S(f) (9.16)
H(f) = S(f) + V(f)
in order for the desired spectrum S(f) to appear at its output. This fre-
quency response was depicted in Figure 9.3. Note that we can think of this
filter as being built of two parts: the denominator corresponds to a whitening
filter, while the numerator is matched to the signals spectrum. Unlike the
whitening filter that we met in the matched filter detector, here the entire
signal plus noise must be whitened, not just the noise.
This filter is a special case of the Wiener filter derived by Norbert Wiener
during World War II for optimal detection of radar signals. It is a special
case because we have been implicitly assuming that the noise and signal are
368 CORRELATION
k
so that given CsZ and CZ we can solve for h, the Wiener filter in the time
domain. To compare this filter with our previous results we need to transfer
the equations to the frequency domain, using equation (4.47) for the FT of
a convolution.
Here PsZ(w) is the FT of the crosscorrelation between s(t) and x(t) , and
Px(w) is the PSD of x(t) (i.e., FT of its autocorrelation). Dividing we find
the full Wiener filter.
f+) = psz(w) (9.17)
w4
For uncorrelated noise Ps&) = P&) and Pz(w) = P&) + P&) and so
the full Wiener filter reduces to equation (9.16).
The Wiener filter only functions when the signals being treated are sta-
tionary (i.e., Psx and Ps are not functions of time). This restriction too can
be lifted, resulting in the K&nun filter, but any attempt at explaining its
principles would lead us too far astray.
9.8. CORRELATION AND PREDICTION 369
EXERCISES
9.7.1 Assume that the signal s(t) has constant PSD in some range but the noise
y(t) is narrow-band. Explain why we expect a Wiener filter to have a notch
at the disturbing frequency.
9.7.2 An alternative to the SNR is the signal-plus-noise-to-noise-ratio S+NNR.
Why is this ratio of importance? What is the relationship between the overall
S+NNR and SNR? What is the relationship between the Wiener filter and
the frequency-dependent S+NNR and SNR?
and then differentiate and set equal to zero. We find that the optimal linear
prediction is
b = -s(l) = c,(l) (9.19)
cs (0)
the normalized autocorrelation coefficient for lag 1. Substituting this back
into the expression for the average square error, we find
d; = cm - c,2(1) (9.20)
( > cs (0)
so that the error vanishes when the lag 1 correlation equals the energy.
EXERCISES
9.8.1 Wiener named his book The Extrapolation, Interpolation and Smoothing of
Stationa y Time Series with Engineering Applications. Wieners extrapola-
tion is what we have called prediction. What did he mean by interpolation
and smoothing ?
9.8.2 Find the optimal linear prediction coefficients when two lags are taken into
account.
s,, = bls,.+1 + bzsn-2
9.9. LINEAR PREDICTIVE CODING 371
Here e, is the portion of the signal not predictable based on the signals own
history, G is an arbitrarily introduced gain, and bm are called the Linear
Predictive Coding (LPC) coefficients. Note that most people use a for these
coefficients, but we reserve a for FIR coefficients; some people use a minus
sign before the sum (i.e., use what we call ,0 coefficients).
Equation (9.21) has a simple interpretation; the signal sn is obtained
by filtering the unpredictable signal e, by a all-pole filter with gain G and
coefficients bm. The e, is called the excitation signal since it excites the
filter into operation. Since the filter is all-pole it enhances certain excited
frequencies; these amplified frequencies are responsible for the non-flat spec-
trum and nontrivial autocorrelation of predictable signals. For speech sig-
nals (see Section 11.3) the excitation e, is the glottal excitation; for voiced
speech (e.g., vowels) this is a periodic set of pulses created by the vocal
chords,.while for unvoiced speech (e.g., h) it is a noise-like signal created by
constricting the passage of air. For both cases the mouth and nasal cavities
act as a filter, enhancing frequencies according to their geometry.
In order to compress the signal we need an algorithm for finding the
M + 1 parameters G and bm given a buffer of N samples of the signal
N-l Looking carefully at equation (9.21) we note a problem. There
{ Sn 1n=O*
are too many unknowns. In order to uniquely determine the coefficients bm
we need to know both the observed speech signal sn and the excitation
en. Unfortunately, the latter signal is usually inaccessible; for speech signals
372 CORRELATION
for which s = 0 (the zero signal) is a solution. It is the only solution if the
excitation was truly always zero; but due to the IIR nature of the filter, other
possibilities exist if the excitation was once nonzero, even if zero during the
duration of the present buffer. For speech the excitation is not truly zero,
so even when we find the coefficients bm we can only approximately predict
the next signal value.
M
sn = c bmsn-m (9.23)
m=l
The error of this approximation is called the residual signal
M
7-n = sn - sn = sn, - c bmsn-m = 5 Pmsn-m (9.24)
(where ,& E 1 and pm = -bm), and the correct LPC coefficients minimize
this residual. Note that the residual is obtained by FIR filtering the input
signal, with the filter coefficients being precisely pm. This all-zero filter is
usually called the LPC analysis filter and it is the inverse filter of the
LPC synthesis filter that synthesizes the speech from the excitation (see
Figure 9.4). The analysis filter is also called the LPC whitening filter, the
residual being much whiter than the original speech signal, since the linear
predictability has been removed.
There is another way of looking at the residual signal. Rather than taking
no excitation and treating the residual as an error signal, we can pretend
that there is excitation but take the error to be precisely zero. What must
the excitation be for s, to be the correct signal value? Comparing equations
(9.24) and (9.21) we see that rn = Gen, the residual is simply the excitation
amplified by the gain. Thus when analyzing voiced speech we see that the
residual is usually small but displays peaks corresponding to the vocal chord
pulses.
9.9. LINEAR PREDICTIVE CODING 373
e+Ft-+s s = = t- all-zero = r
filter Li F
Figure 9.4: LPC synthesis and analysis filters. The synthesis filter synthesizes the signal
s,, from the excitation e,, while the analysis filter analyzes incoming signal s,, and outputs
the residual error signal rn. The synthesis and analysis filters are inverse systems to within
a gain.
(9.27)
(9.28)
These are, of course, the Yule-Walker equations for the LPC coefficients.
The sum in the autocorrelations should run over all times n. This is
problematic for two reasons. First, we are usually only given an input signal
buffer of length IV, and even if we are willing to look at speech samples
outside this buffer, we cannot wait forever. Second, many signals including
speech are stationary only for short time durations, and it is only sensible
to compute autocorrelations over such durations. Thus we must somehow
limit the range of times taken into account in the autocorrelation sums.
This can be done in two ways. The brute-force way is to artificially take all
signal values outside the buffer to be zero for the purposes of the sums. A
somewhat more gentle variant of the same approach uses a window function
(see Section 13.4) that smoothly reduces the signal to zero. The second
way is to retain the required values from the previous buffer. The first way
is called the autocorrelation method and is by far the most popular; the
second is called the covariance method and is less popular due to potential
numerical stability problems.
The autocorrelation method allows the sum in the MSE to be over all
times, but takes all signal values outside the buffer se. . . s~-l to be zero.
Since the error en in equation (9.24) depends on A4 + 1 signal values, it can
only be nonzero for n = 0. . . N + M - 1. Accordingly, the MSE is
N+M-1
E= c et
n=O
EXERCISES
9.9.1 What is the approximation error for the covariance method?
9.9.2 Equation (9.22) predicts sn based on M previous values s,+r, ~~-2,. . . s,+M
and is called the forward predictor. We can also predict (postdict?) s,-M
based on the next M values s+M+r, . . . , s,+ 1, sn. This surprising twist on
LPC is called backward linear prediction. Modify equation (9.22) for this case
(call the coefficients c,). What is the residual?
9.9.3 Show that the MSE error can be written E = C, si + C,=, b, C, snsnmm
and thus for the autocorrelation method E = C,(O) + Cz=1 bmCs(m).
9.9.4 Show that assuming the input to be an impulse G&Q the gain is given by
the error as given in the previous exercise.
9.9.5 Use the LPC method to predict the next term in the sequence 1, a, 02, a3,. . .
for various 0 < c1!< 1. Repeat for cy > 1. Does the LPC method always
correctly predict the next signal value?
down a pipe consisting of two cylinders. Recalling from Section 7.9 that
traveling waves are functions of s - wt, we can express the incoming wave
for this one directional case as foilowL
The reflected wave in the first cylinder will be sinusoid of the same frequency
but traveling in the opposite direction and reduced in amplitude
where the reflection coefficient k is the fraction of the wave that is reflected.
Since the energy is proportional to the signal squared, the fraction of the
waves energy that is reflected is k2, while the wave energy that continues
on to the second cylinder is whatever remains.
Now for a little physics. The $ for sound waves can represent many
different physical quantities (e.g., the average air particle displacement, the
air particle velocity, the pressure). Well assume here that it represents the
velocity. Physically this velocity must be continuous across the interface
between the two sections, so at the interface the following must hold.
Lets check to see that this result is reasonable. If the second cylinder
shrinks to zero area (closing off the pipe) then k = 1 and the wave is entirely
reflected, as it should be. If there really is no interface at all (i.e., Al = AZ)
then k = 0 and no energy is reflected. If AZ >> A1 then k = -1, which seems
unreasonable at first; but an open-ended pipe has zero pressure at its end,
and so the wave reflects but with a phase reversal.
It isnt hard to generalize our last result to a pipe with many sections.
The reflection coefficient at the interface between section i and section i + 1
is
Ai - Ai+1
ki = (9.34)
Ai + Ai+1
What does all this have to do with solving the Yule-Walker equations for
the LPC coefficients in the autocorrelation method? The LPC coefficients
b, are not the only way of describing an all-pole system; the area ratios,
the reflection coefficients, and many others (including an interesting set to
be discussed in the next section) can be used instead. Since all of these
parameter sets contain exactly the same information, it follows that we can
derive any set from any other set. Many of the parameter sets are related by
linear transformations, and hence the conversion is equivalent to multiplying
by a matrix. We will now show that the connection between the reflection
and LPC coefficients can be expressed as a recursion that is the most efficient
way of deriving both.
How can equation (9.29) be solved recursively? For simplicity well drop
the subscript identifying the signal, but we have to add superscripts identi-
fying the recursion depth. The first case is simple (for further simplicity we
have dropped the subscript)
where we have defined kl z br]. Lets assume we have already solved the
mth case
..
blnm]
9.10. THE LEVINSON-DURBIN RECURSION 379
and lets write this C[]b[ml = ~1~1.W e are now interested in the (m + l)th
= -
case
where we have drawn in delimiters that divide the equations into two parts:
C(l)
04
= C(2)
.. (9.35)
LC(m)
1
and
(9.36)
0 0 ... 0 1
0 0 ... 1 0
J = ii;;;
= i 0 1 ... 0 0
1 0 ... 0 0 1
and noting that it commutes with Toeplitz matrices, we can finally write
the following recursion for the LPC coefficients
[ F)=(L-km+li)[
fl (9.37)
where k, E bk].
In the statistics literature the k variables are called partial correlation or
PARCOR coefficients, since they can be shown to measure the correlation
between the forward and backward prediction errors (see exercise 9.9.2).
Later we will show that they are exactly the reflection coefficients.
Were we to know km+1 this recursion would produce all the other new
b[+l given the old b[ml. So we have reduced the problem of finding the
LPC coefficients to the problem of finding the PARCOR coefficients. Yet it
is obvious from equation (9.36) that the converse is also true, kLm+l] can be
derived from the lower b[+] coefficients. So lets derive a recursion for the
ks and try to eliminate the bs.
First we rewrite equation (9.36) as
=J + km+lC(O) = C(mtl)
-c - --
Jb+I
-
+ km+lC(O) = C(m + 1)
9.10. THE LEVINSON-DURBIN RECURSION 381
C(m + 1) - c JbLrnl l
km+l =
C(O) - c T$T-
C(m+lG-_ c * __
Jb[ml
=
Em -
identifying the MSE in the denominator. After following all the above the
reader will have no problem proving that the MSE obeys the simplest re-
cursion of all.
E m+l = (1~ k&+,)Ern (9.38)
Lets now group together all the recursive equations into one algorithm
that computes the k and b coefficients for successively higher orders until
we reach the desired order M.
forp = m - 1 down to 1
&/I t b[m-1l _ k b[-ll
P m m-u
Em + (l-kg)Em-r
for p + 1 to M
b t bLM1
To see how the algorithm works lets run through it for the case of two
coefficients .
( g; g, ) (i:)=( 2)
The first iteration is easy.
E. = C(O)
b[l] = ICI= -C(1)
C(O)
382 CORRELATION
E _ C2(O>- C2(l>
1-
C(O)
C(2)C(O) - C2(1)
bp = k2=
C2(0) - V(l)
$1 = $1 - k2 = WC(O) - ww
C2(0) - C2(1)
EXERCISES
9.10.1 Prove equation (9.34) for a pipe with multiple sections taking into account
the reflected wave from the next interface.
9.10.2 Transmission lines have both voltage and current traveling waves, the ratio
between the voltage and current being the impedance 2. At a splice where
the impedance changes a reflected wave is generated. Express the reflection
coefficient in terms of the impedances. Explain the limiting cases of shorted
and open circuited cables.
9.10.3 Prove equation (9.38) for the MSE.
9.10.4 Solve the three-coefficient problem on paper using the Levinson-Durbin re-
cursion.
9.10.5 Show that the complexity of the Levinson-Durbin algorithm is O(i@) rather
than O(M3) as for non-Toeplitz systems.
9.10.6 Levinson originally solved the more general problem of solving the equations
TX = -y where T is Toeplitz but unrelated to -y. Generalize the recursion
--
G solve this problem. (Hint: You will need another set of recursions.) How
much more computationally complex is the solution?
9.11. LINE SPECTRAL PAIRS 383
= x2 - 2cos($) + 1
Pm = am + a,--WI
a, = 2Prn + qm) where (9.39)
%?I = a??X- aM--m
(if M is even the middle coefficient appears in pm only). When we are dealing
with polynomials that have their constant term equal to unity, we would like
the polynomials pm and qm to share this property. To accomplish this we
need only pretend for a moment that am is a polynomial of order M + 1 and
use the above equation with a~+1 = 0.
Pm = am + aM+l-m
a, = $<Pm + Qm) where (9.40)
qT7-t = am- aM+l-772
Figure 9.5: The zeros of a polynomial and of its palindromic and antipalindromic com-
ponents. The Xs are the zeros of a randomly chosen tenth order polynomial (constrained
to have its zeros inside the unit circle). The circles and diamonds are the zeros of the p(z)
and q(z). Note that they are all on the unit circle and are intertwined.
9.11. LINE SPECTRAL PAIRS 385
44 = $(P(X)
+ q(2)) where
and it is not hard to show that if all the zeros of a(x) are inside the unit circle,
then all the zeros of p(x) and of q(x) are on the unit circle. Furthermore,
the zeros of p(x) and q(x) are intertwined, i.e., between every two zeros
of p(x) there is a zero of q(x) and vice versa. Since these zeros are on the
unit circle they are uniquely specified by their angles. For the polynomial
in the denominator of the LPC frequency response these angles represent
frequencies, and are called the LSP frequencies.
Why are the LSP frequencies a useful representation of the all-pole filter?
The LPC coefficients are not a very homogeneous set, the higher-order bm
being more sensitive than the lower-order ones. LPC coefficients do not
quantize well; small quantization error may lead to large spectral distortion.
Also the LPC coefficients do not interpolate well; we cant compute them at
two distinct times and expect to accurately predict them in between. The
zeros of the LPC polynomial are a better choice, since they all have the same
physical interpret at ion. However, finding these zeros numerically entails a
complex two-dimensional search, while the zeros of p(x) and q(x) can be
found by simple one-dimensional search techniques. In speech applications
it has been found empirically that the LSP frequencies quantize well and
interpolate better than all other parameters that have been tried.
EXERCISES
9.11 .l Lets create a random polynomial of degree M by generating M + 1 random
numbers and using them as coefficients. We can now find the zeros of this
polynomial and plot them in the complex plane. Verify empirically the hard-
to-believe fact that for large M most of the zeros are close to the unit circle
(except for large negative real zeros). Change the distribution of the random
number generator. Did anything change? Can you explain why?
9.11.2 Prove that if all the zeros of U(Z) are inside the unit circle, then all the zeros
of p(z) and of a(~) are on the unit circle. (Hint: One way is write the p and
Q polynomials as a(z) (1 f h(x)) w here h(x) is an all-pass filter.) Prove that
the zeros of p(z) and q(z) are intertwined. (Hint: Show that the phase of
all-pass filter is monotonic, and alternately becomesx (zero of p) and 0 (zero
of cl>*>
9.11.3 A pipe consisting of M + 1 cylinders that is completely open or completely
closed at the end has its last reflection coefficient kM+i = f 1. How does this
relate to the LSP representation?
386 CORRELATION
9.11.4 Generate random polynomials and find their zeros. Now build p(x) and q(x)
and find their zeros. Verify that if the polynomial zeros are inside the unit
circle, then those of p and q are on the unit circle. Is there a connection
between the angles of the polynomial zeros and those of the LSPs?
9.11.5 The Greek mathematician Apollonius of Perga discovered that given two
points in the plane ~1 and ~2, the locus of points with distances to zi and ~2
in a fixed ratio is circle (except when the ratio is fixed at one when it is a
straight line). Prove this theorem. What is the connection to LSPs?
The lath moment function of the digital stationary stochastic signal s is the
average of the product of Ic + 1 signal values, at time lags defined by the
moment functions parameters.
The first-order moment function is simply
MA] = (sn)
However, the third moment can be significantly different from this as well.
For instance, a signal that is generated by
(where V, is some driving noise signal) will have a nontrivial third moment
function with just these lags.
388 CORRELATION
Similarly the fourth and higher moment functions give the probability
that the signal takes on 1 values at four or more times. In practice, in-
terpretation of numeric moment function data is complex because of the
contributions from lower-order moments, as in equations (9.42) and (9.43).
For example, if 0 and 1 are equally probable, we expect to observe 1 at two
different times with a probability of one-quarter; only deviations from this
value signify that there is something special about the lag between the two
times. Likewise, to really understand how connected four different times
are, we must subtract from the fourth moment function all the contribu-
tions from the third-order moments, but these in turn contain portions of
second-order moments and so on. The way to escape this maize of twisty
little passages is to define cumulants.
The exact definition of the cumulant is a bit tricky since we have to
keep track of all possible groupings of the time instants that appear in the
moment function. For this purpose we use the mathematical concept of a
partition, which is a collection of nonempty sets whose union is a given set.
For example, in the third moment there are three time instances no = n,
721 = n+ml, and n2 = n + m2, and these can be grouped into five different
partitions. PI = {(n17n2,n3)), P2 = e-Q>, @2r n3)), p3 = {(n2), (w, n3)),
p4 = {(n2), (ni,n2)}, and P5 = {(nr), (ns), (n3)). Well use the symbol Sij
for the jth set of partition Pi (e.g., $1 = (ni) and &2 = (nl,n2)), Ni the
number of such sets (Ni = 1, N2 = N3 = N4 = 2 and N5 = 3)) and Nij for
the number of elements in a set (e.g., Nrr = 3, N51 = 1). We can now define
the cumulant
C[I
s = C(-l)~~-l(N~ - l)! ~ ~~Nij(S~j) (9.44)
i j=l
where the sum is over all possible partitions of the Ic time instants.
It will be convenient to have a special notation for the signal with its
DC component removed, $ E s - (s) . The first few cumulants can now be
exmessed as follows:
as expected,
c;21(m)= M~21(?n)
- (My)2 = (&$n+m)
which is the autocovariance rather than the autocorrelation,
9.12. HIGHER-ORDERSIGNAL PROCESSING 389
CL31(m1, m2) =
Mu](mw2) - A@] (M;](n) + AI;] ( n+m1) + M;21(n+m2)
>
+ 2(A4q3
= (&Gn+ml &l+?nz >
a surprisingly simple result,
c~4](ml,m2,m3) = itd~4](ml,m2,m3) - . l . - 6(Mi11)4 =
(&&+ml &l+m2 &z+ma >
-&m,)c; PI(??%2--m3)
-q
PI(rn2)C$ PI(m3-74 - cg
PI (ms)C4 PI(7w-m2)
which is somewhat more complex. For the special case of a zero mean signal
and ml = rn2 = ma, Cl21 is the variance, Cl31 the skew, and Ci41 the
kurtosis.
Other than their interpretability, the cumulants are advantageous due
to their convenient characteristics. The most important of these, and the
reason they are called cumulants, is their additivity.
PI
Cz+&-m, m2,. . . mk-1) = Cikl(ml, m2,. . . m&l) + $(ml, m2,. . . m&l)
It is easy to see that this characteristic is not shared by the moment func-
tions. Another nice feature is their blindness to DC components
where a is any constant. Like the moments, cumulants are permutation blind
C~kl(m,l, m,,, . . . mcrk.l) = C,PI (ml, m2, . . . mk-1)
where cri is any permutation of 1. . . k - 1; and scale according to their order.
Now that we have defined them, we can show that cumulants are truly
useful. Assume that we have a non-Gaussian signal distorted by Gaussian
noise. Standard signal processing does not take advantage of the higher-
order statistics of the signal, and can only attempt to separate the signal
from the noise in the power spectral domain. However, cumulants of the
third and higher orders of the noise will vanish exactly, while those of the
signal will not, thus providing a more powerful tool for recovery of such a
signal. For example, higher-order matched filters can be used as sensitive
detectors of the arrival of non-Gaussian signals in Gaussian noise.
We know from Section 8.1 that intermodulation products are produced
when two sinusoids enter a nonlinearity. Assume we observe several fre-
quency components in the output of a possibly nonlinear system; is there
any way to tell if they are intermodulation frequencies rather than indepen-
dent signals that happen to be there? The fingerprint of the phenomenon is
that intermodulation products are necessarily phase coupled to the inputs;
but such subtle phase relations are lost in classical correlation-based anal-
ysis. By using higher-order cumulants intermodulation frequencies can be
identified and the precise nature of system nonlinearities classified.
In Sections 6.12 and 6.13 we saw how to perform correlation-based sys-
tem identification when we had access to a systems input and output.
Sometimes we may desire to identify a system, but can only observe its
output. Amazingly, this problem may be tractable if the input signal is non-
Gaussian. For example, if the unknown system is an N tap FIR filter,
N-l
Yn = c hmxn-m + vn
m=O
the input x is zero mean but with nonzero third-order cumulant, and the
output y is observed contaminated by additive Gaussian (but not necessarily
white) noise v, then the systems impulse response can be derived solely from
the outputs third-order cumulants.
hm =
EXERCISES
9.12.1 Find all the partitions of four time instants and express CP](mi, na2, ma) in
terms of moments.
9.12.2 Consider the three systems (0 < a, b < 1)
yn = &a - (U + b)xn-1 + abxn-2
Yn = Xn - (a + b)xn+l + ubxn+2
The cumulants are the coefficients of the Taylor expansion of the logarithm of
this function. Derive the first few cumulants according to this definition and
show that they agree with those in the text. Derive the additivity property
from this new definition.
9.12.5 In the text we mentioned the application of higher-order signal processing
to the identification of intermodulation products. Let cpr, (~2 and (~3 be in-
dependent uniformly distributed random variables and define two stochastic
signals
&I = cm (wn + 91) + cos (w2n + 92) cos ((w + w2)n + (p1 + $92))
each of which has three spectral lines, the highest frequency being the sum
of the lower two. The highest component of ~1~1could be an intermodulation
product since it is phase-locked with the other two, while that of ~1~1is an
unrelated signal. Show that both signals have the same autocorrelation and
power spectrum, but differ in their third-order cumulants.
392 BIBLIOGRAPHICAL NOTES
Bibliographical Notes
Matched filters are covered in most books on communications theory, e.g. [242, 951.
Wieners first expositions of the Wiener-Khintchine theorem were in mathe-
matical journals [276] but he later wrote an entire book on his discoveries [277].
The co-discoverer of the theorem was Aleksandr Khintchine (or Khinchin), whose
Mathematical Foundations of Information Theoy was translated into English from
the original Russian in 1957.
The second volume of Norbert Wieners autobiography [280] has fascinating
background information on Wieners work at MIT during the World War II years.
His 1942 report, entitled Extrapolation, Interpolation and Smoothing of Stationary
Time Series, was suppressed because of possible military applications, and finally
released only in 1949 [278]. Even though written to be more understandable than
the former paper, its mathematics, more familiar to physicists than engineers, was
so difficult to the latter audience that it was commonly called the yellow peril.
Levinson both explained Wieners results to a wider audience [146] and translated
the formalism to the digital domain. While accomplishing this second task he in-
vented his recursion [147], although digital hardware capable of computing it did
not exist at the time.
The invention of LPC is due to Bishnu Atal of Bell Labs [lo], who was mainly
interested in its use for compression of speech [9]. The LSP frequencies are due
to Itakura of NTT Research Labs [log] (but dont bother checking the original
reference, its only an abstract).
Higher-order signal processing is the subject of a book [181] and numerous
review articles [173, 1821. [33] d iscusses partitions in a simple way, and includes
source code for computing the number of partitions of n objects. Cumulants were
introduced in statistics by Fisher in the 1930s and in use in physics at about the
same time. The idea of higher-order spectra as the FT of cumulants dates back
to Kolmogorov, but the nomenclature polyspectra is due to Tukey. The use of
cumulants for output-only system identification is due to Georgios Giannakis [72].
A few references to the extensive literature on applications of cumulants include
noise cancellation [49]; system identification [73, 651; blind equalization [235, 2361;
and signal separation [286, 287, 1081.
10
Adaptation
393
394 ADAPTATION
Yn = xn + bn-k (10.1)
10.1. ADAPTIVE NOISE CANCELLATION 395
with k z 2400 and h < 1. Of course the delay need not be an integer number
of samples, and indeed in a closed room we will get multiple noise echoes
due to the sound waves bouncing off the walls and other surfaces. Each such
echo will arrive at a different time and with a different amplitude, and the
total effect is obtained by adding up all these contributions. We will return
to the effect of multiple echoes later.
Lets try to regain the desired clean lecturers voice signal from the noisy
received signal yn and the reference signal qn. Lets assume at first that we
know the delay k having measured the distance between the microphones,
but have no information regarding the gain h. We can try to subtract out
the interference
Xn = Yn - Rh-k (10.2)
with zn representing our attempt at recovering xn. This attempt is depicted
in Figure 10.1, using a self-explanatory graphical technique to be presented
more fully in Chapter 12. We know that this could work; were we to know
h we could set e = h and
c3cQ(~)
=c XnQn-ln
E;i:=)~;=~(xn+r,-k)2=~x;+2~xnr,-k+~r~-k
n n n n n
but the cross term is precisely lag Ic of the correlation between x, and rn
that was assumed to be zero.
10.1. ADAPTIVE NOISE CANCELLATION 397
Continuing
Yn = xn + x hkqn-k (10.4)
k
as depicted in Figure 10.2. Once again we are assured that this can be
successful, since selecting ek = hk will guarantee Zn = Xn. Viewed in this
light, the problem of noise removal is equivalent to the finding of an unknown
filter, with the filter coefficients possibly varying in time.
398 ADAPTATION
Following the same path as before we find that due to the assumption of
lack of correlation between X~ and qn, the energy of the attempted recon-
struction is the sum of two parts.
Es!
=cxi +c (C(hk
n
- e.,,qnsk)2
n k
The first is the energy of the desired signal xn and the second is the energy
of the residual interference. As a function of the vector of coefficients, the
energy E(el, e2,. . . eN> is a hyperparaboloid with a single global minimum
to be found. Once again this minimum corresponds to the desired signal.
How does one find this minimum in practice? When there was only a
single coefficient e to be found, this was a relatively easy job. For example,
we could start with any arbitrary e and then try moving along the e axis
by some positive or negative amount. If the energy decreases then we keep
moving in the same direction; otherwise we move in the opposite direction.
If after several steps that decrease the energy, it starts to rise again, then we
have gone too far; so we reduce the step size and home in on the minimum.
The more general case can also be solved by arbitrarily moving around
and checking the energy, but such a strategy would take a long time. With
one variable there were just two directions in which to move, while with
N coefficients there are an infinite number of directions. However, since we
know that the energy surface in ek space is a hyperparaboloid, we can (with
only a little extra work) make a good guess regarding the best direction.
The extra work is the calculation of the gradient of the energy in ek space,
VE(el, es,. . . eN> . Recall that the gradient of a surface is the multidimen-
sional extension of the derivative. The gradient of a function is a vector
that points in the direction the function increases most rapidly, and whose
length is proportional to the steepness of the function. At a maximum or
minimum (like the base of the energy paraboloid) the gradient is the zero
vector. Were we to be interested in finding a maximum of the energy, the
best strategy would be to move in the direction of the gradient. Any other
direction would not be moving to higher energy values as quickly. In order
to find the energys minimum we have to reverse this strategy and move in
the direction opposite the gradient. This technique of finding a minimum
of a function in N-dimensional space is called steepest descent or gradient
descent, and will be more fully explained in Section 10.5.
10.1. ADAPTIVE NOISE CANCELLATION 399
Figure 10.3: Cancelling filtered noise by an inverse filter (equalizer). This time the filter
ek is adapted to equal the inverse of the distorting filter hk. When successfully adapted
the output of filter e equals the input of h so that the interference is subtracted from the
desired signal.
EXERCISES
10.1.1 Unlike the air-conditioner,the audience is not located at one well-defined
location. Can the audience noise be removed in a manner similar to the air-
conditioner noise?
10.1.2 Build a random signal and measure its energy. Add to it a sinusoid and mea-
sure the resulting energy. Did the energies add? Subtract from the combined
signal the same sinusoid with varying amplitudes (but correct phase). Graph
the energy as a function of amplitude. What curve did you get? Keep the
correct amplitude but vary the phase. Is the behavior the same?
10.1.3 Electrocardiographs are required to record weak low-frequency signals and
are often plagued by AC line frequency pickup (50 or 60 Hz). Were there
are no desired signal components near this frequency a sharp notch filter
would suffice, however generally an adaptive technique should be employed.
Since we can directly measure the AC line sinusoid, the problem is reduced to
finding the optimum gain and phase delay. Explain how to solve this problem.
Simulate your solution using a stored waveform as the desired signal and a
slowly amplitude- and phase-varying sinusoid as interference.
10.1.4 A frequency agile notch filter can remove periodic interference (of unknown
frequency) from a nonperiodic desired signal without a separate reference
signal. Explain how this can be done.
400 ADAPTATION
*
telephone f---t hybrid hybrid u - telephone
1 2
4
Figure 10.4: The telephone hybrid. At both ends of the telephone connection are two
wire channels, but in between the conversation is carried over four-wire circuits.
10.2. ADAPTIVE ECHO CANCELLATION 401
switch =
AL
1
comparator out
2
1r
Figure 10.5: The telephone echo suppressor. The top line represents one direction of
the four wire telephone channel, and the bottom line the other. When the upper signal
is greater than the lower one the comparator gives positive output, thus keeping the
upper path open but suppressing the lower signal. When the lower signal is greater the
comparator output is negative and the switches open the lower path but shut the upper.
When the delay of line echo is short, it simply combines with the sidetone
and is not noticeable. However, when the delay becomes appreciable line echo
becomes quite annoying. Most people find it disconcerting to hear their own
voice echoing back in their ear if the delay is over 30 milliseconds. An echo
suppressor is a simple device that combats line echo by disconnecting one
side of the conversation while the other side is talking. The functioning of
an echo suppressor is clarified in Figure 10.5. Echo suppressors often cause
conversations to be carried out as if the telephone infrastructure were half-
duplex rather than full-duplex. Such conversations are unnatural, with each
side lecturing the other without interruption, rather than engaging in true
dialog. In addition, echo suppressors totally disrupt the operation of data
communications devices such as faxes and modems, and must be disabled
before these devices can be used. A Line Echo Canceller (LEC) is a more
complex device than an echo suppressor; it enables full-duplex conversations
by employing adaptive DSP algorithms.
How does an echo canceller work? Like the adaptive noise canceller, the
basic idea is that of subtraction; since we know the original signal that has
been fed back, we need only subtract it out again. However, we need to know
the delay, attenuation, and, more generally, the filter coefficients before such
subtraction can be carried out.
Full-duplex modems that fill all of the available bandwidth and use a
single pair of wires for both directions always experience echo. Indeed the
echo from the nearby modulator may be as strong as the received signal,
and demodulation would be completely impossible were it not to be removed
effectively. Hence a modem must remove its own transmitted signal from the
received signal before attempting demodulation.
402 ADAPTATION
: A/D k NLP =
,r ,r
li
to
NEAR z hybrid doubletalk adaptation
filter - FAR
END detector mechanism END
Figure 10.6: A digital telephone network line echo canceller (LEC). In this diagram only
signal flow lines are indicated; invisible are the logic indications sent from the double-
talk detector to the adaptation mechanism and NLP, and the fact that the adaptation
mechanism sets the filter coefficients and the NLP threshold.
The filter processor places digital samples from the far-end into a static
buffer (called the X register in LEC terminology), convolves them with the
filter (called the H register), and outputs the echo estimate to be subtracted
from the near-end samples.
The adaptation mechanism is responsible for adapting the filter coeffi-
cients in order to reproduce the echo as accurately as possible. Assume that
the far-end subscriber is talking and the near-end silent. In this case the
entire signal at the input to the subtracter is unwanted echo generated by
the nearby hybrid and the near-end telephone. Consequently, the adaptation
mechanism varies the filter coefficients in order to minimize the energy at the
output of the subtracter (the place where the energy is measured is marked
in the figure). If the far-end is quiet the adaptation algorithm automatically
abstains from updating the coefficients.
When the double-talk detector detects that both the near-end and far-
end subscribers are talking at the same time, it informs the adaptation
mechanism to freeze the coefficients. The Geigel algorithm compares the ab-
solute value of the near-end speech plus echo to half the maximum absolute
value in the filters static buffer. Whenever the near-end exceeds the far-end
according to this test, we can assume that only the near-end is speaking.
The nonlinear processor (NLP) is a center clipper (see equation (8.7)),
that enables the LEC to remove the last tiny bit of perceived echo. For
optimal functioning the center clipping threshold should also be adapted.
Although the LEC just described is somewhat complex, the basic filter is
essentially the same as that of the adaptive noise canceller. In both cases a
filtered reference signal is subtracted from the signal we wish to clean up, and
in both cases the criterion for setting the coefficients is energy minimization.
These two characteristics are quite general features of adaptive filters.
404 ADAPTATION
EXERCISES
10.2.1 Why is an acoustic echo canceller usually more complex than an LEC?
10.2.2 Why is the phone network LEC designed to cancel echo from the transmitted
signal, rather than from the received signal?
10.2.3 Describe the following performance criteria for echo cancellers: convergence
speed, ERLE (echo return loss enhancement), and stability (when presented
with narrow-band signals). The minimum performance of acoustic echo can-
tellers is detailed in ITU-T standard G.167, and that of LECs in G.165 and
G.168. Research, compare, and contrast these standards.
10.2.4 Assume that each tap of the echo cancelling FIR filter takes a single instruc-
tion cycle to calculate, that each coefficient update takes a single cycle as
well, and that all the other elements are negligible. Estimate the maximum
and typical computational complexities (in MIPS) required to echo cancel a
standard voice channel (8000 samples per second) assuming a 16-millisecond
tail in which echoes can occur.
10.2.5 Explain the Geigel algorithm for double-talk detection. Why isnt it sufficient
to compare the present near-end to a single far-end value? Why compare to
half the maximum far-end? How does it differ from the comparator in the
echo suppressor? How can it be improved?
9(t)
x(t) - channel * : (> equalizer - = 4t>
Figure 10.7: An equalizer for digital communications signals. The original signal z(t)
transmitted through the communications channel, and subject to additive noise y(t), is
received as signal y(t). The purpose of the equalizer is to construct a signal z(t) that is as
close to z(t) as possible.
Figure 10.8: The eye pattern display graphically portrays the effect of ISI, noise and
possibly other impairments on the receivers capability to properly decode the symbol. In
the present diagram the eye is open and proper decoding is possible.
406 ADAPTATION
Figure 10.9: The effect of increasing intersymbol interference, The filtered channel out-
put is superposed over the original signal. In (A) (th e mildest channel) the received signal
is close to the ideal signal. In (B) the bandwidth has been reduced and symbol recovery
has become harder. In (C) proper symbol recovery is not always likely. In (D) (the harshest
channel) symbol recovery has become impossible.
Why do channels cause the eyes to close? Channels limit the bandwidth
of signals that pass through them, and so ideal symbols will never be ob-
served at the channel output. Mild channels merely smooth the symbol-to-
symbol jumps, without impairing our ability to observe the proper symbol
value far from transitions, but channels with long impulse responses smear
each symbol over many symbol intervals, as seen in Figure 10.9. As a re-
sult the channel output at any given time is composed not only of the de-
sired symbol, but of contributions of many previous symbols as well, a phe-
nomenon known as InterSymbol Interference (ISI). When the IS1 is strong
the original information cannot be recovered without equalization.
At first glance the adaptation of an equalizer would seem to be com-
pletely different from the applications we discussed in previous sections. In
the previous cases there was an interfering signal that contaminated the
signal of interest; here the source of contamination is the signal itself! In
the previous cases there was a reference signal highly correlated to the con-
taminating signal; here we observe only a single signal! Notwithstanding
10.3. ADAPTIVE EQUALIZATION 407
EXERCISES
10.3.1 An alternative to equalization at the receiver as illustrated in Figure 10.7 is
Tomlinson equalization, where the inverse filter is placed at the transmitter.
What are the advantages and disadvantages of this approach? (Hints: What
happens if the channels frequency response has zeros? How can the equalizer
be adapted?)
10.3.2 DDE is not the only way to adapt an equalizer. Blind equalization uses gen-
eral characteristics of the signal, without making explicit decisions. Assume
the symbol for a 0 bit is -1 and that for a 1 bit is +l. How can the fact that
the square of both symbols is unity be used for blind equalization? Describe
a blind equalizer for a constant amplitude signal that encodes information in
its phase.
10.3.3 Signal separation is a generalization of both equalization and echo cancel-
lation. The task is to separate the signal mixtures and recover the original
signals. Let xi be the original signals we wish to recover, and yi the observed
combination signals. The most general linear two-signal case is
Yl = h * xl + hn * x2 (10.6)
Y2 = h * XI + hzz * x2
where hii are the self-filters (which need to be equalized) and the hi#j the
cross-filters (which need to be echo-cancelled). Generalize this to N combi-
nations of N signals. What conditions must hold for such problems to be
solvable?
408 ADAPTATION
We have introduced this rather unusual vector notation in order to keep our
discussion as general as possible. Using the dot product we can consider d to
be the output of an FIR filter, in which case x are N consecutive values of
a signal; the output of a phased array (see Section 7.9), in which case x are
values of N different signals received simultaneously by N sensors; or a-two-
class linearly separable pattern recognition discrimination function. In this
last application there are objects, each of which has N measurable numerical
features, xi . . . XN. Each object belongs to one of two classes, and pattern
recognition involves identifying an objects class. Two classes are called lin-
early separable when there is a linear function d(:[l) = C,=, v,x~~] that
is positive for all objects belonging to one class and negative for all those
belonging to the other.
When using this new notation the N coefficients are called weights,
and v a weight vector. In all three cases, the adaptive filter, the adaptive
beamformer, and the two-class discriminator, our task is to find this weight
vector given example inputs x [ml and outputs dImI. Since this is still the
system identification problemryou know that the optimum solution will be
given by the Wiener-Hopf equations (6.63). However, we beg your indulgence
10.4. WEIGHT SPACE 409
as our aim is to rederive these equations in a way that will be more suitable
for adaptive filters.
Assume that after seeing m - 1 inputs and respective desired outputs we
manage to come up with some weight vector w. - Then upon observing the
mth example, we predict the output to be
(10.8)
and if the desired output is really Jrnl our output gm is in error by 614 =
&4 - ,bq
Consider now the abstract iv-dimensional vector space of all possible
weight vectors w. Before our opponent allows us to observe the system
all weight vectors are possible, and all points in weight space are equally
plausible. After we have observed a single input-output example only a small
subset of weight space remains as plausible weight vectors, since most weight
vectors would produce outputs differing significantly from the observed one.
We can pick any point in this subset of plausible weight vectors as our guess
w. Each successive input-output example we observe reduces the size of the
subset of plausible weight vectors; indeed, were there no noise, after seeing
N different examples the subset would have been reduced to a single point.
This picture is encouraging, but doesnt provide us a practical heuristic
with which to find good weight vectors. To do so we now define the cost (or
loss) function L(w). This cost function is defined for every weight vector
in weight space, id is simply a measure of how plausible a weight vector
w really is. A highly plausible weight vector should have a low cost, while
one that noticeably violates the desired examples would be assigned a high
cost. An obvious candidate for the cost function is the Mean Squared Error
WE)
L(w)
- = ( (6[Q2) (10.9)
the averaging being done over all the observed examples. From its definition
the MSE is always nonnegative, and in the absence of noise there is a single
weight vector for which the MSE is precisely zero. This weight vector is
precisely the weight vector your opponent used, and by finding it you win
the game. In the presence of noise there will generally not be any weight
vectors with precisely zero MSE, but your best guess will be the weight
vector with the Minimum Mean Squared Error (MMSE).
Now you have a strategy with which to proceed. For each example m
take the input -z irnl, calculate the corresponding output ~1~1 for every weight
410 ADAPTATION
where the expectation ((~@l)~) simply means adding up all the errors and
dividing by the number of examples. Substituting the basic relation (10.8)
we find
L(u) = ((d[m1)2)
- 2 ( dLml
Fwnxn)+ (F F w-wn~~)
= ( (d[m])2) - 2c 20, (d[m]xn) + cc wnq (xnxz)
n n 1
L(w)
- = D2 - 2 C Wn(Cdz)n + C C WnWl(C,>nl (10.11)
n n 1
= D2 - 2~. Cdz -I- -v-
wC,w
--
10.4. WEIGHT SPACE 411
To find the minimum of the cost function we need to use the gradient
operator
(10.12)
and set the gradient of the cost equal to zero.
0 = VL(w)
- = -2(cdz>n + 2 ~(&)dW = -2c& -j- 2c,
-- w
1
Solving, we find the following set of N equations
i.e. -c&g = --
&ut (10.13)
L(w)
- < L(w)
- (10.16)
If the cost function indeed decreases at each step, we must eventually arrive
at a minimum.
The simplest type of iterative step is gradient descent, where the new
guess is found by moving in the direction in which the cost function de-
creases the fastest. To do this we compute the gradient VL( -w), which is
412 ADAPTATION
Figure 10.10: The effect of different step sizes on constant step size gradient descent.
Small steps waste iterations while large steps may overshoot the minimum.
the direction in which the cost function increases most rapidly, and move
in the opposite direction. More sophisticated methods exploit the matrix of
second derivatives (the Hessian) as well, but even just calculating and stor-
ing the N-by-N matrix can be prohibitive in high dimensions. All of these
methods require inverting the Hessian matrix, an operation that is not only
computationally costly, but numerically problematic.
In the simplest type of gradient descent we move some arbitrary step
size p at every step of the algorithm.
W1
w4 (10.17)
- = If - ppL(w)l
In general, this is often not a good idea (see Figure 10.10) since where the
gradient is steep this step size may be overly small requiring us to take many
small steps where one large one would have sufficed, while where the gradient
is shallow we may overshoot the minimum and need to reverse direction at
the next iteration. Alternatively, we can save computation by moving some
fraction of the value of the gradient
wt = w - XVL(w) (10.18)
- -
which is a logical thing to do if the gradient gets larger as we go further
from the minimum. There are more complex techniques that search along
the line to determine how far to move (requiring much more computation),
or vary the step size depending on the absolute value of the gradient or the
difference between the present gradient direction and that of the previous
iteration.
10.5. THE LMS ALGORITHM 413
We have seen that the MMSE weight vector can be found explicitly
via the Wiener-Hopf equations, or numerically using minimum finding tech-
niques such as gradient descent. Both of these methods assume that the un-
derlying system is time-invariant. When the system can constantly change
we require MMSE finding methods that can dynamically adapt to these
changes. The rest of the chapter will be devoted to such methods. It is
an interesting coincidence of alliteration that the equations that constitute
the simplest adaptive adaptation of the Wiener-Hopf equations is called the
Widrow-Hoff equation.
EXERCISES
10.4.1 Assume that there is but a single weight w, so that the Wiener-Hopf equation
is simply c& = w*C,. Show that the cost function as a function of this w
is a simple nonnegative parabola with a single minimum. For what weight is
the cost precisely zero?
10.4.2 Assume that there are two weights wr and ~2. Show that the cost function
surface is a paraboloid with a single minimum.
10.4.3 What is the computational complexity of the solution in (10.14)?
10.4.4 Try directly solving the Wiener-Hopf equations for the case of simple aver-
aging (i.e., the unknown coefficients are all *). Generate some large number
of input-output pairs, compute the correlations, and use the matrix inversion
technique of the previous exercise to solve. Have an opponent supply some
random -w and try to discover it.
10.4.5 Show that the MMSE weight vector decorrelates the error from the input
vector, (i.e., for w* the error 61ml and the input z lrnl obey (61~1> = 0).
What is the deeper meaning of this statement, sometimes called the orthog-
onality principle? What can be said about the error-output correlation?
We can thus expressed the MSE cost function as the sum of M nonnegative
single example terms, which can be zero only if all the individual terms are
zero. As an inference the gradient of this cost function must also be the
sum of single example terms! The problem is that moving w in the direction
dictated by one example, although decreasing the present &tribution, may
increase the contributions from other examples! In particular we may move
the weight vector in order to optimize for some input-output example, and
then move it right back for the next example; but when we get close enough
to the global minimum everything should work out fine.
So lets investigate the single example gradient. Its nth coordinate is
(VLLl), = Z$
where we have used the chain rule, equation (10.19) and equation (10.8).
Substituting this into equation (10.18) we find
Initialize: wll = 0
Loop until c&verged :
get new input 321ml
- and desired output dlml
compute new output ylml = 2~1~1 . ~1~1
calculate error @l = d[ml-- y[mJ
correct weight vector -wlm+ll = wlml+
- XS[mlzlml
-
is called the LMS algorithm. LMS stands for Least Mean Squared, referring
to our attempt at finding an MMSE solution.
Unlike our attempts at finding the MMSE in the previous section, the
LMS algorithm is an adaptive algorithm. If the true weights v vary slowly in
time, the LMS algorithm will follow these changes, approxr%ating at each
instant the best weight vector for that time. Of course if the underlying
system varies too rapidly, even an adaptive algorithm may not be able to
keep up.
The LMS algorithm is by far the most popular adaptive algorithm, and
the Widrow-Hoff equation appears in many contexts, although sometimes it
may be hidden. There is an easy way to recognize Widrow-Hoff in disguise;
all the correction terms contain the same output error term, while each
weight correction term multiplies it by its own input. Remember that the
complete correction term is a constant times the input times the output
error.
In order to get a feel for the use of the LMS algorithm, lets try a
simple example. Well take a three-dimensional case with the true weight
vector w = (5, i, i), and start with an initial guess of w = (0, 0,O). Now
assuming that the first input is -x = (1 , 1,l) , well be told that the desired
output is
d =
- w
while the present system outputs w -- l x = (0, 0,O) . (1, 1,l) = 0. The output
erroristhusS=d-y= l.IfweuseX= i the corrections will be as follows.
-w * w
- + X6x- = (O,O,O) + $ 1 (l,l, 1) = (i, h, ;,
416 ADAPTATION
Lets take the same input and perform another iteration. The output is
--w*x = ( i, ;, $) * (l,l, 1) = $
wn
f
l/3
0 m
0 10 20 30 40
Figure 10.11: The convergence of the Widrow-Hoff algorithm. Here the correct weight
vector has three components equal to 4, the initial guess is zero, the inputs are random,
the desired output is observed with 10% noise,and X = O,l, We seethat the weights ~1,
~2, and wg converge slowly but are close to the proper values after m FZ 50 iterations.
10.5. THE LMS ALGORITHM 417
We just used the word convergence without specifying what this means.
We may mean that the difference between the derived weight vector and the
true one must converge to zero, or that the MSE must converge to zero, but
for the LMS algorithm we usually mean that the mean squared error should
converge to some small value (nonzero due to the noise). How do we know
when the weights have converged? In Figure 10.11 we see that the changes
have become much less drastic after about thirty cycles, but can we really
be sure that this isnt a temporary phenomenon? Around cycle 15 there was
a short stretch where the weights did not change much for a few cycles, but
afterward the changes returned. In practice it really can be a tricky decision,
and usually the convergence criterion can be concocted only after in-depth
study of the particular problem at hand.
Assuming that the LMS algorithm does indeed converge, can we be sure
that it will converge to the right answer? Happily we can prove that the
expectation of the weight vector approaches the MMSE solution as M in-
creases. To show this we first slightly rewrite the n th component of the
Widrow-Hoff equation.
(wrm])
- = x 2 (I - Kq C& + (I= - Acqrn lJl
CL=0= = - c -
418 ADAPTATION
This last equation, being a matrix equation, must be true in all coordinate
systems, in particular in a coordinate system where the input autocorrela-
tion matrix is diagonal. In this system I - XC, is also diagonalized. The
diagonal elements l- Xc, will be less thai unitzn absolute value if X obeys
2
o<x<c (10.22)
vmax = Tr Cx = NE,
Z
where Ez is the energy of the input signal. This leads to a useful approximate
range for X.
2
o<x<- (10.24)
(E 2 )
10.5. THE LMS ALGORITHM 419
Up to now our discussion has been completely general; we end this section
by restricting the general discussion of input-output pairs to the system
identification case. By using the observed recent inputs and output of an
FIR system we can combine FIR convolution and LMS adaptation, thus
defining the standard adaptive FIR filter. We can make a new input-output
example for ewemJ new time instant
EXERCISES
10.5.1 It is easy to extend the derivation of the delta rule to nonlinear combinations,
the most important of which is the sigmoidal nonlinearity of equation (8.11).
Show that in this casethe delta rule reads w, + wn + X6Y(l - y)zn. (Hint:
Use a further chain rule and exercise 8.4.5).
10.5.2 Assume the unknown weights of a three-parameter linear combination are
w = ($, $,O) and that the inputs are x = (l,O,O), (O,l,O), (O,O, 1) over and
over again. Simulate this system with x = 4 and no noise. Try other values
for X and add noise. How fast can you make the LMS algorithm converge?
What happens if X is too large? What happens if we multiply all the inputs
by 100?
10.5.3 Although LMS finds the best direction, its choice of constant step size seems
overly primitive. A more sophisticated approach would be to search for min-
imal cost along the gradient direction. Compare LMS and this line-search
gradient algorithm on a simulated problem. How many cycles are required
for convergence? How many output and error evaluations?
10.5.4 Equation (10.20) seemsto require two multiplications and one addition for
each tap update. Show how this can be reduced. Compare the computational
complexity of LMS update to that of running the FIR filter. Why do some
DSP processorshave an LMS instruction?
10.5.5 What is necessary to make the LMS algorithm work for complex-valued sig-
nals? What is the complexity compared to a real signal?
420 ADAPTATION
10.5.6 In exercise 10.1.3 we discussed the cancellation of power line noise from weak
medical signals. Even when a narrow notch filter could be used LMS fil-
ters may require less computation. For example, assume that an ECG is
a quasiperiodic with period about one Hz. For the purposes of simulation,
model the ECG signal as two sinusoidal cycles one after the other, the first
with period one-tenth of the whole period, and the second filling the remain-
ing 0.9, but with amplitude one-tenth of the first. Add some white noise
and a nominal power line frequency with total energy about the same as the
desired signal. Try to remove the power line signal with a static FIR notch
filter; how many coefficients are required? Now use an LMS filter; how many
taps are required now?
(10.25)
where Ex is the input signals energy. One way of thinking about NLMS is
to cast it in standard LMS form with a normalized X;
accordingly NLMS is LMS with the step size tuned individually for each
inputs energy. In many applications NLMS converges faster than vanilla
LMS. More good news about NLMS is that it converges when 0 < p < 2,
so we neednt estimate input energy or autocorrelation eigenvalues. In fact,
p = 1 is just about always best. One drawback is that NLMS requires the
10.6. OTHER ADAPTIVE ALGORITHMS 421
by adding up the error times input for all m in the block; then once the
entire block has been seen a single correction
is performed.
Block LMS is preferable to vanilla LMS when the input signal fluctuates
rapidly, but converges more slowly for relatively stationary signals. To con-
vince yourself of this latter fact think of a block of length M in which the
signal is constant. Standard LMS will perform M separate iterations while
block LMS essentially performs only the first of these.
A compromise between BLMS and vanilla LMS is LMS with momentum.
In this variant we smooth the weight changes by a kind of AR filtering
1
cM pbl - w . ,[my (10.28)
Mm=1 - -
where M is taken small enough that the underlying system does not vary
appreciably during the M time instants. This policy automatically provides
422 ADAPTATION
-w = Z
c,-led, (10.30)
where the correlations appearing here are only slightly different from the
usual ones.
p-mZ[mlZ[ml
Zcz = 5 - -
m=l
M
(pM-mdbd x [ml
cdz = c -
m=l
The useful thing about exponential weighting is that these quantities can
be built up recursively.
Cz Lrnl = (&p-l1 + ,JmlJml
- - (10.31)
= ipFh b-11 + &d x[m]
-
Now if we only knew how to recursively update the inverse autocorrela-
tion matrix -P s C/l we could substitute these recursions into (10.30) to
obtain a re&&ion=fr the weight vector. Luckily, there are ways to recur-
sively update the inverse of a matrix of this sort. Using the matrix inversion
lemma (A.100) with A = cpCZlm-ll, g = D = ~1~1 and C = 1 we find
= C = - =
@pb-11 &d -&-d p[m-11
(pc,b11 + &4JJ4 -l = ~-1~In-11 -
( Z - - > (P-~&lp[ -m&T+ 1
which looks messy but contains only quantities known at step m of the
recursion. In order to clean it up a little we define the gain vector
pb-11 -xb.4
p.4 =, (10.32)
xIml;b-llxbl
-
+ cp
=
10.6. OTHER ADAPTIVE ALGORITHMS 423
in terms of which we can rewrite the recursion for the inverse autocorrelation.
Now the expression for the gain vector looks terrible, but it really has a
simple interpretation. Rearranging equation (10.32)
,bl = (C,byC&,bl
-
= pTl cpcd,bd + &4&d)
L
= ~p[ml ch b-11 + &d piA &4
= = -
Initialize: -w[Ol = 0
Loop until converged:
get new input -x Iml and desired output 61m]
compute new output ~1~1 = wlml .x[~l
calculate error 61ml = d[~l- grmJ
compute gain vector /cl] using equation (10.32)
correct weight vector using equation (10.35)
update inverse autocorrelation P
=
using equation (10.33)
Hence RLS can be thought of as LMS with a very intelligent adaptive step
size. This step size is a matrix, and hence takes care of cost function surfaces
in weight space that are steep in some directions but flat in others. The step
size is optimized at every step to ensure rapid convergence.
Each iteration of the RLS algorithm is more complex than an iteration
of the LMS algorithm, and indeed RLS is often impractical for real-time
applications. However, the RLS algorithm will normally converge faster than
the LMS one, at least when the noise is small. When there is strong additive
noise a long period of averaging is necessary in order to average out the
noise, and so RLS cannot significantly decrease the number of iterations
needed.
As with many recursive update formulas, the RLS updates can accumu-
late numerical error, eventually leading to a noninvertible CZ. This usually
isnt a problem when only a few w are needed, but becomeszolerable when
we must continuously update weFght vectors. One solution to this problem is
to iteratively update the Cholesky decomposition of CZ (see equation (A.94))
rather than the matrix itself. Another is the so-called QR-RLS algorithm,
which multiplies the equations by an orthogonal matrix in order to keep
them triangular; but further discussion of these topics would take us too far
astray.
BIBLIOGRAPHICAL NOTES 425
EXERCISES
10.6.1 Compare the LMS, NLMS, and RLS algorithms on benchmarks of your
choice. Which is fastest? Which is most robust?
10.6.2 One way of ameliorating the numeric difficulties of NLMS is by using the
following regularization.
2
w=w+p&-=---
- - E -I- Ez
Experiment with NLMS and regularized NLMS for signals with large dynamic
range.
Bibliographical Notes
There are many good textbooks on adaptive signal processing. A classic text is that
of Widrow and Stearns [275] and a more recent text is that of Haykin [96].
The invention of adaptive filters was necessitated by the conversion of the phone
to digital technologies and the consequent problems of echo cancellation and adap-
tive differential coding of speech. Adaptive beamforming [272] and equalization
were not far behind, and the use of LMS in the related field of pattern recognition
helped enrich the field.
In 1960, Bernard Widrow and one of his graduate students, Marcian (Ted) Hoff,
presented the Widrow-Hoff approach [274] which is basically the LMS algorithm.
Widrow went on to adapt this approach to become one of the fathers of the neu-
ral network; Hoff joined Intel and went on to become one of the fathers of the
microprocessor.
The applications we presented are covered by review articles of note; adaptive
noise cancelling in [273, 591, echo cancellation in (87, 1781, and equalizers in (202,
2031.
Adaptive equalization for digital communications was originally developed by
Lucky at Bell Labs [152, 1531 and was the main reason for the increase of speed of
telephone modems from 2400 b/s to 9600 b/s.
The technique for equalization at the transmitter discussed in exercise 10.3.1
was developed in the early 1970s by Tomlinson and by Harashima in Japan [260,92].
11
At first it may seem a bit unusual to find a chapter on biological signal pro-
cessing in a book dedicated to digital signal processing; yet this is in reality
no more peculiar than motivating DSP by starting with the analogous prin-
ciples of analog signal processing. Indeed the biological motivation should
be somewhat closer to our hearts (or eyes, ears and brains). In this book we
have chosen to introduce analog and digital signal processing together, but
have confined our discussion of biological signal processing to this chapter.
In the first two sections we examine how we map external signal param-
eters into internal (biological/psychological) representations. This question
belongs to the realm of psychophysics, the birth of which we describe. Our
senses are highly sensitive and yet have a remarkably large dynamic range;
we would like to understand and emulate this ability. We will see that a
form of universal compression is employed, one that is useful in many DSP
contexts.
The majority of the signals we acquire from the outside world and pro-
cess in our brains are visual, and much interesting signal processing takes
place in our visual system. Much has been discovered about the function-
ing of this system but here we concentrate on audio biological mechanisms
since the focus of this book is one-dimensional signals. Hearing is the sense
with the second largest bandwidth, and speech is our primary method of
communications. We will devote a section each to speech production and
perception mechanisms. In a later chapter we will study a DSP model of
speech production that is based on this simplified biology.
After studying the signal input and output mechanisms we proceed to
the processing apparatus, namely the brain. We discuss the basic processor,
the neuron, and compare its architecture with that of processors with which
we are more familiar. We introduce a simple model neuron and the concept
of a neural network, and conclude with a performance comparison of man
vs. machine.
427
428 BIOLOGICAL SIGNAL PROCESSING
AW=KW K x 0.034
11.1. WEBERS DISCOVERY 429
This means that in order for a change in weight to be noticeable, one has to
add a specific percentage of the present weight, not an absolute weight value.
This radically changed the way Weber understood the JND. He set out
to check the dependence of other sensitivity thresholds on total stimulus
intensity and found similar relationships.
AI = KII (11.1)
In each case the ratio KI, called Webers constant, was different, but the
linear dependence of the JND on total stimulus was universal.
Although this relationship surprised Weber it really is quite familiar.
Have you ever lain awake in the middle of the night and heard the ticking of
a clock or the barking of a distant dog? These sounds are not heard during
the day when the ambient noise is higher, but seem quite loud at night when
the total stimulus is low. Yet they must be there during the day even if not
discernible. It is simply that the addition of the ticking or distant barking to
the other sounds does not increase the total sound by a sufficient percentage.
You get out of bed and open the window. You remember how the stars
were so bright when you were a child, yet seem so dim now. During the day
you cant see them at all. Yet they must be there during the day even if
not discernible. It is simply that with no light from the sun the starlight
is a more significant fraction of the total stimulus. With the expansion of
cities and the resulting light pollution the stars are disappearing, and one
has to go further and further out into the countryside in order to see them.
You close the window and strike a match in the dark room. The entire room
seems to light up, yet had you struck the same match during the day no
change in illumination would have been noticed.
Lets now consider the sequence of physical values that are perceivably
different. Think of turning on the radio and slowly increasing the volume
until you just begin to hear something. You then turn a bit more until you
notice that the sound has definitely grown louder. Continuing this way we
can mark the points on the volume control where the sound has become
noticeably louder. A direct application of Webers law tells us that these
marks will not be evenly spaced.
Assume for the purpose of argument that the particular stimulus we
are studying just becomes detectable at one physical unit 10 = 1 and that
Webers constant for this stimulus is a whopping 100%. Then the second
distinguishable level will be 11 = 2 because any value of I that adds less
than one unit is indistinguishable from lo. Continuing, we must now add
KI1 = 2 units to the existing two in order to obtain the third distinguishable
430 BIOLOGICAL SIGNAL PROCESSING
EXERCISES
11.1.1 Try Webers coin experiment. Can you measure Webers constant?
11.1.2 Write a computer program that presents a random rectangle on one part of
the graphics screen, and allows subjects to reproduce it as closely as possible
somewhere else on the screen. What is K here?
11.1.3 Allow a subject to listen for a few seconds to a pure sinusoid of constant
frequency and then attempt to adjust the frequency of a sinusoid to match it.
What is K here? Repeat the experiment with amplitude instead of frequency.
11.1.4 Patterns of dots can be hidden by randomly placing large numbers of dots
around them. The original pictures stand out if the dots are of different color
or size, are made to slowly move, etc. Devise an experiment to determine
different peoples thresholds for detecting patterns in random dot pictures.
by observing the world with their senses. So even scientists are completely
dependent on the subjective in order to arrive at the objective.
The English philosopher Berkeley was fond of saying esse est percipi,
that is, existence is being perceived. We have all heard the famous conun-
drum about a tree falling in a forest not making a sound if there is no one
around to hear it. A physical signal that is not captured by our senses might
as well not exist. This capturing of physical signals and their translation
into internal representations is called perception.
The connection between physical signals and psychological manifesta-
tions is by no means simple. The cover of this book looks the same in di-
rect sunlight, under a fluorescent lamp, and by the light of a candle. Your
mothers voice sounds the same outside, in a train car, and over the phone.
Your friend seems the same height when he is standing close to you, when
he has walked across the street, and even on television. In all these cases the
physical signals varied widely but the internal psychological representation
remained the same. Our perception of quite different physical phenomena
may be the nearly the same.
Is it possible to say anything quantitative about internal psychologi-
cal representations? Can feelings be measured? Surely our perceptions and
thoughts are personal and unobservable to the outside world. How then can
we talk about representing them quantitatively? Although consideration of
such questions has convinced many sages to completely reject psychophysics,
these very same questions can be raised regarding much of modern science.
We cannot directly observe quarks, electrons, protons, or even atoms, but
we become convinced of their existence by indirectly perceiving their effects.
Individual cells cannot be seen, but biologists are convinced of their exis-
tence. We cannot hold the Milky way galaxy in our hand, yet astronomers
have deduced its existence. Feelings may not be openly witnessed, but their
existence may be inferred from psychophysical experiments.
Notwithstanding the importance and wide applicability of Webers law, it
is not a true psychophysical law. Psychophysical laws should relate external
physical signals to internal psychological representations. Webers law relates
the intensity threshold AI to the total stimulus I, both of which are physical
entities. Yet another step is needed to make a true psychophysical law.
The first direct attempt to quantitatively pin down feelings was made by
one of Webers students, Gustav Theodor Fechner. Fechner initially studied
medicine, but after graduation was more involved in physics. Webers discov-
eries retriggered his interest in psychophysics. Fechner started studying color
perception, and later performed a series of experiments on the persistence
of color after a bright light has been removed.
432 BIOLOGICAL SIGNAL PROCESSING
Y=AlogI+B (11.3)
i.e., that apart from an additive constant that derives from the minimum
biological sensitivity, the psychological intensity is proportional to the loga-
rithm of the physical intensity.
We know that the logarithm is an extremely compressive function. A log-
arithmic psychophysical connection would explain the fantastic ranges that
our senses can handle. Under proper conditions we can hear a sound that
corresponds to our ear drum moving less than the diameter of a hydrogen
atom, and we can actually see single photons. Yet we can also tolerate the
sound of a jet engine corresponding to 1012 times the minimum intensity
and see (for short periods of time as Fechner learned) direct sunlight 15 or-
ders of magnitude stronger. In order to quantitatively compare two signals
that may differ by such large amounts we introduce the BeZ (named after
Alexander Graham), defined as the base 10 logarithm of the ratio of the
powers of the two signals. In other words, if the power of the second signal
is greater than that of the first by a factor of ten, we say that it is one Be1
(1 B) stronger. In turns out that the Be1 is a bit too large a unit for most
purposes, and so we usually use the decibel (dB), which is ten times smaller.
Pl
d(m) = lOlog g (11.4)
Since power is the integral of the square of the signal values, if we know RMS
signal values we can directly compute the difference between two signals.
Sl
d(m) = 2Olog g (11.5)
The JND for strong sounds is about 0.5 dB, while at the threshold of hearing
about 3 dB is needed.
An audio signals amplitude is not the only characteristic that is per-
ceived approximately logarithmically. Humans can hear from about 20 Hz
(lower than that is felt rather than heard) to over 20 KHz (the precise upper
limit depending on age). This corresponds to about 10 octaves, each octave
being a doubling of frequency. Sinusoids separated by whole octaves sound
similar to us, this fact being the principle behind the musical scale. Inside
each octave the conventional western (well-tempered) division is into twelve
chromatic keys, each having frequency $6 higher than the previous one.
These keys sound to us approximately equally spaced, pointing once again
to a logarithmic perception scale.
434 BIOLOGICAL SIGNAL PROCESSING
M = lOOOlog,(f,,, + 1)
The Bark (named after the acoustician H.G. Barkhausen) scale approxi-
mates the natural frequency scale of the auditory system. Psychophysically,
signals heard simultaneously are perceived as separate sounds when sep-
arated by one Bark or more since they excite different basilar membrane
regions. A Bark is about 100 Hz for frequencies under 500 Hz, is about 150
Hz at 1 KHz, and a full KHz at about 5 KHz.
If we divide the entire audio range into nonoverlapping regions of one Bark
bandwidth we get 24 critical bands. Both the me1 and Bark scales are
approximately logarithmic in frequency.
EXERCISES
11.2.1 Derive equation (11.3). What is the meaning of A and B? What should be
the base of the logarithm?
11.2.2 How long does a tone have to be on for its frequency to be identifiable?
Experiment!
11.2.3 The well-tempered scale is a relatively recent invention, having become pop-
ular with the invention of keyboard-based instruments such as the piano.
Using a computer with a programmable sound generator, test the difference
between a linearly divided scale and a well-tempered one. Play a series of
notes each higher than the previous one by 50 Hz. Do the differences sound
to same? Play a simple tune on the well-tempered scale and on a linearly
divided octave scale. Can you hear the difference? Can you describe it?
11.2.4 Since we perceive sound amplitudes logarithmically, we should quantize them
on a logarithmic scale as well. Compare the p-law and A-law quantizations
prevalent in the public telephone system (equations (19.3) and (19.4)) with
logarithmic response. How are negative values handled? Can you guess why
these particular forms are used?
11.3. SPEECH PRODUCTION 435
= 7 sinh-(f&/0.65)
while the Mel warping was given in the text. Compare these three empirical
formulas with true logarithmic behavior cyln( 1 + z) in the range from 50 Hz
to 5 KHz.
11.2.6 Recent research has shown that Fechners law is only correct over a certain
range, failing when the stimuli are either very weak or very strong. Stevens
proposed a power law Y = ICI where k and n are parameters dependent
on the sense being described. Research Stevens law. For what cases does
Stevens law fit the empirical data better than Fechners law?
11.2.7 Toward the end of his life Fechner studied aesthetically pleasing shapes. Write
a program that allows the user to vary the ratio of the sides of a rectangle
and allow a large number of people to find the nicest rectangle that is
not a square. What ratio do people like? (This ratio has been employed in
architecture since the Greeks.)
To understand this risk we have to compare our mouth and throat re-
gions with those of the other primates. Comparing the profile of a human
with that of a chimpanzee reveals that the chimpanzees muzzle protrudes
much further, while the human has a longer pharynx (throat) and a lower
larynx (voice box). These changes make it easy for the human to change
the resonances of the vocal cavity, but at the expense of causing the respi-
ratory and alimentary tracts to overlap. Thus food can go down the wrong
way, impeding breathing and possibly even leading to death by choking.
However, despite this importance of spoken communication, the speech gen-
eration mechanism is still basically an adapted breathing and eating appa-
ratus, and the speech acquisition mechanism is still essentially the acoustic
predator/prey detection apparatus.
It is convenient to think of speech as being composed of a sequence of ba-
sic units called phonemes. A phoneme is supposed to be the smallest unit of
speech that has independent meaning, and thus can be operationally defined
as the minimal amount of speech that if replaced could change the mean-
ing of what has been said. Thus b and k are distinct phonemes in English
(e.g., book and cook have different meanings), while 1 and r are indis-
tinguishable to speakers of many oriental languages, b and p are the same
in Arabic, and various gutturals and clicks are not recognized by speakers
of Latin-based languages. English speakers replace the French or Spanish r
with their own because the originals do not exist in English and are thus
not properly distinguished. Different sources claim that there are between 42
and 64 phonemes in spoken English, with other languages having typically
between 25 and 100. Although the concept of a phoneme is an approxima-
tion to the whole story, we will posit speech generation and perception to
be the production and detection of sequences of phonemes.
Speech generation commences with air being exhaled from the lungs
through the trachea (windpipe) to the larynx (voice box). The vocal
cords are situated in the larynx. While simply breathing these folds of
tissue are held open and air passes through them unimpeded, but when
the laryngeal muscles stretch them taut air must pass through the narrow
opening between the cords known as the glottis. The air flow is interrupted
by the opening and closing of the glottis, producing a periodic series of
pulses, the basic pulse rate being between 2.5 and 20 milliseconds. The
frequency corresponding to this pulse interval is called the pitch. The tighter
the cords are stretched, the faster the cycle of opening the cords, releasing the
air, and reclosing, and so the higher the pitch. Voice intensities result from
the pressure with which the expelled air is forced through the vocal cords.
The roughly triangular-shaped pulses of air then pass into the vocal tract
11.3. SPEECH PRODUCTION 437
consisting of the pharynx (throat), mouth cavity, tongue, lips, and nose
and are finally expelled. There are two door-like mechanisms that prohibit
or allow air to flow. Two passages proceed downward from the pharynx,
the esophagus (food pipe) and trachea. The epiglottis separates the two
by closing the air passage during swallowing. In addition, air can enter the
nasal tract only when the velum is open.
The air pulses exiting the vibrating vocal cords can be thought of as a
signal with a basic periodicity of between 50 and 400 Hz (typically 50-250
for men, 150-400 for women) but rich in harmonics. Thus the spectrum
of this signal consists of a set of equally spaced lines, typically decreasing
in amplitude between 6 and 12 dB per octave. Because of its physical di-
mensions, the vocal tract resonates at various frequencies called formants,
corresponding to the length of the throat (between 200 and 800 Hz), length
of the nasal passage (500-1500 Hz), and size of the mouth between throat
and teeth (1000-3000 Hz). These resonances enhance applicable frequen-
cies in the glottal signal, in the manner of a set of filters. The result is the
complex waveform that carries the speech information. The spectrum thus
consists of a set of lines at harmonics of the pitch frequency, with amplitudes
dependent on the phoneme being spoken.
The vocal cords do not vibrate for all speech sounds. We call phonemes
for which they vibrate voiced while the others are unvoiced. Vowels (e.g., a, e,
i, o, u) are always voiced unless spoken in a whisper, while some consonants
are voiced while others are not. You can tell when a sound is voiced by
placing your fingers on your larynx and feeling the vibration. For example,
the sound s is unvoiced while the sound z is voiced. The vocal tract is the
same in both cases, and thus the formant frequencies are identical, but z
has a pitch frequency while s doesnt. Similarly the sounds t and d share
vocal tract positions and hence formants, but the former is unvoiced and
the latter voiced. When there is no voicing the excitation of the vocal tract
is created by restricting the air flow at some point. Such an excitation is
noise-like, and hence the spectrum of unvoiced sounds is continuous rather
than discrete. The filtering of a noise-like signal by vocal tract resonances
results in a continuous spectrum with peaks at the formant frequencies.
The unvoiced fricatives f, s, and h are good examples of this; f is gener-
ated by constricting the air flow between the teeth and lip, s by constricting
the air flow between the tongue and back of the teeth, and h results from a
glottal constriction. The h spectrum contains all formants since the excita-
tion is at the beginning of the vocal tract, while other fricatives only excite
part of the tract and thus do not exhibit all the formants.
438 BIOLOGICAL SIGNAL PROCESSING
EXERCISES
11.3.1 What are the main differences between normal speaking on the one hand and
whispering, singing, and shouting on the other?
11.3.2 Why do some boys voices change during adolescence?
11.3.3 Match the following unvoiced consonants with their voiced counterparts: t,
s, k, p, f, ch, sh, th (as in think), wh.
11.3.4 Simulate the speech production mechanism by creating a triangle pulse train
of variable pitch and filtering with a 3-4 pole AR filter. Can you produce
signals that sound natural?
11.3.5 Experiment with a more sophisticated software speech synthesizer (source
code may be found on the Internet). How difficult is it to produce natural-
sounding sentences?
11.4. SPEECH PERCEPTION 439
What about the auditory cortex itself? We started the previous section
by contrasting the vocal tracts of the human with those of other primates,
yet the difference in our brain structure between ourselves and the apes is
even more remarkable. The human brain is not the most massive of any
animals, but our brain mass divided by body mass is truly extraordinary,
and our neocortex is much larger than that of any other animal. There
are two cortical regions that deal specifically with speech, Brocas area and
Wernickes area, and these areas are much more highly developed in humans
than in other species. Brocas area is connected with motor control of speech
production apparatus, while Wernickes area is somehow involved in speech
comprehension.
To summarize, the early stages of the biological auditory system perform
a highly overlapped bank of filters spectral analysis, and it is this represen-
tation that is passed on to the auditory cortex. This seems to be a rather
general-purpose system, and is not necessarily the optimal match to the
speech generation mechanism. For example, there is no low-level extraction
of pitch or formants, and these features have to be derived based on the
spectral representation. While the biology of speech generation has histor-
ically had a profound influence on speech synthesis systems, we are only
now beginning to explore how to exploit knowledge of the hearing system in
speech recognition systems.
EXERCISES
11.4.1 Experiment to find if the ear is sensitive to phase. Generate combinations of
evenly spaced sines with different phase differences. Do they sound the same?
11.4.2 Masking in the context of hearing refers to the psychophysical phenomenon
whereby weak sounds are covered up by stronger ones at nearby frequencies.
Generate a strong tone at 1 KHz and a weaker one with variable frequency.
How far removed in frequency does the tone have to be for detection? Atten-
uate the weaker signal further and repeat the experiment.
11.4.3 Sit in a room with a constant background noise (e.g., an air-conditioner) and
perform some simple task (e.g., read this book). How much time elapses until
you no longer notice the noise?
11.4.4 Go to a (cocktail or non-drinking) party and listen to people speaking around
the room. What effects your ability to separate different voices (e.g. physical
separation, pitch, gender, topic discussed)?
11.4.5 Have someone who speaks a language with which you are unfamiliar speak
a few sentences. Listen carefully and try to transcribe what is being said as
accurately as you can. How well did you do?
442 BIOLOGICAL SIGNAL PROCESSING
11.4.6 Talk to someone about speech recognition and then quickly ask Do you know
how to wreck a nice peach?. Ask your partner to repeat what you just said.
What does this prove?
11.4.7 Most of the time and energy of the speech signal is spent in the vowels, and
hence the speech perception mechanism performs best in them. But do vowels
carry most of the information? You can find out by performing the following
experiment. Select several sentences at random from this book. From each
sentence create two character strings, one in which all consonants are replaced
by question marks, and one in which all vowels are. Now present these strings
to subjects and ask them to fill in the blanks. What are your findings?
11.4.8 Explain the possible mechanisms for acoustic source location. Take into ac-
count the width of the head, the fact that localization is most accurate for
high-frequency sounds with sharp attack times, and the idea that the head
will absorb some sounds casting an acoustic shadow (about 3 dB at 500
Hz, 20 dB at 6 KHz). How is height determined? Devise a neurobiologically
plausible model for a time-of-arrival crosscorrelator.
11.4.9 Simulate the sound perception mechanism by building a bank of overlapping
band-pass filters (at least 100 are needed) and graphically displaying the
output power with time as horizontal axis and filter number as vertical axis.
Test by inputting a sinusoid with slowly varying frequency. Input speech and
try to segment the words on the graphic display.
axon
Figure 11.1: A highly schematic diagram of the classic higher processing neuron. The
dendrites at the left are the inputs, the axon at the right is the output, and the processing
is performed by the soma.
EXERCISES
11.5.1 The central nervous system is composedof the spinal cord, the brain stem, the
cerebellum, the midbrain, and the left and right hemispheresof the neocortex.
What are the functions of these different components? How do we know?
What creatures (reptiles, mammals, primates) have each component?
446 BIOLOGICAL SIGNAL PROCESSING
11.5.2 In addition to neurons the brain also contains glia cells; in fact there are
more glia than neurons. Why did we focus on neurons and neglect glia cells
in our explanation?
11.5.3 There are many morphologies other than the classical model described in
this section. For example, there are neurons with no axon or many axons,
synapses may occur between two axons, or between two dendrites, or even
on the cell body. Research several such variations. What is the function of
these cells?
11.5.4 Research the Nobel prize-winning contribution of Hubel and Wiesel to the
understanding of the neurons in the mammalian visual system. What are
simple cells, complex cells, and hypercomplex cells? What is a hypercolumn
and how are the cells arranged geometrically?
EXERCISES
11.6.1 Which of the following are neural networks according to the definition given
in the text?
1. transistors on the substrate of an integrated circuit
2. arithmetic registers in a microprocessor
3. CPUs in a parallel processing environment
4. cells in the spinal column
5. neurons in an aplysia
11.6.2 By introspection, make a rough order-of-magnitude estimate of the amount
of information (in bits) passedto the brain by the various senses.For vision,
for example, estimate the size of the main field of vision, the pixel density,
the dynamic range, and the number of pictures transferred per second. Based
on the above estimates, how much information must the brain store in a day?
A year? A lifetime? The brain contains about lOlo neurons. Does the above
estimate make sense?
(there is no synapse). If 1lVj 1is large then the effect of this input is significant,
while small IVV1means the input only marginally affects the output.
The linear combination of the McCulloch-Pitts neuron means that the
cells potential is assumed to be
N
h=CWjIj (11.6)
j=l
and the neuron will fire when this potential surpasses the threshold potential
required to excite an action potential 8. Using the 0,l representation we can
write
O=O(h-8)=@ (11.7)
0= sgn(h - 0) = I (11.8)
where the Signum function sgn returns the sign of its argument. Note that
the meaning of this latter equation is somewhat different from the previous
one; here neurons that are not firing also contribute to the sum.
The McCulloch-Pitts neuron is the simplest possible neuron model. The
inputs are combined in a simple deterministic way. The decision is made
based on a simple deterministic comparison. The summation and comparison
are instantaneous and time-independent. Yet this completely nonbiological
formal neuron is already strong enough to perform nontrivial computations.
For example, consider the following image processing problem. We are pre-
sented with a black-and-white image, such as a fax, that has been degraded
by noise. The classic DSP approach would be to filter this binary-valued
image with a two-dimensional low-pass filter, which could be implemented
by averaging neighboring pixels with appropriate coefficients. This would
leave a gray-scale image that could be converted back to a black-and-white
image by thresholding. This combination of the linear combination of input
pixels in neighborhoods followed by thresholding can be implemented by a
two-dimensional array of McCulloch-Pitts neurons. This same architecture
can implement a large variety of other image processing operators.
What is the most general operation a single linear threshold function
can implement? For every possible input vector the McCulloch-Pitts neuron
outputs 0 or 1 (or &l). Such a function is called a decision function or
11.7. THE SIMPLEST MODEL NEURON 451
EXERCISES
11.7.1 One can convert 0,l neurons into fl neurons by using the transformation
S -+ 2s - 1. Show that equations (11.7) and (11.8) are equivalent by thus
transforming the Ij and finding transformations for Wj and 8.
11.7.2 Show that AND and OR gates of in-degree N can be implemented using
McCulloch-Pitts neurons. That is, specify Wj and 8 such that the neuron
will fire only if all inputs fire, or if at least one input fires.
11.7.3 Draw and label the possible input configurations of a three-input linear
threshold function as vertices of a cube. Show graphically which dichotomies
can be implemented with zero threshold.
11.7.4 Extend the results of the previous problem to general McCulloch-Pitts neu-
rons (nonzero threshold) .
11.7.5 The McCulloch-Pitts neuron can be used as a signal detection mechanism.
Assume we wish to detect a signal s, of N samples and report the time that it
appears. One builds a neuron with N continuous inputs and weights precisely
Wj = sj. We then have the input flow past the neuron, so that at any time
the neuron sees N consecutive inputs. Consider the fl representation and
show that the PSP will be maximal when the required signal is precisely
lined up with the neural inputs. How should the threshold be set (take noise
and false alarms into account)? To what signal processing technique does this
correspond?
11.7.6 Extend the results of the previous exercise to image recognition. Would such
an approach be useful for recognition of printed characters on a page? If not
why not? What about cursive handwriting?
11.7.7 Discuss the use of McCulloch-Pitts neurons for associative memory.
11.7.8 What is the difference between Hebbs principle for 0,l neurons and f 1
neurons?
Surprisingly, by making this one processing element faster and more pow-
erful, this computer evolved into a completely different architecture, quite
powerful in its own right. Only now, with computer speeds approaching the
absolute limits that physics imposes, is parallel processing being once again
seriously considered; but even now when computer engineers talk about par-
allel processing they are referring to small numbers of CPUs, such as two,
four, or eight. A comparison of the human brain to a conventional com-
puter, based on the information of the last few sections, is to be found in
Table 11.1.
Brain Computer
x 10 billion neurons 1 CPU
processors (massively parallel) (intrinsically serial)
processor simple complex
complexity inaccurate accurate
processor slow fast
speed (millisec) (nanosec)
inter-processor fast slow
communications bsf4 (millisec)
learning mode learn from experience manual programming
failure many neurons die single fault often
robustness without drastic effect leads to svstem failure
memory content addressable location addressable
organization (CAM) (LAM)
Table 11.1: A quantitative and functional comparison of the human brain and a serial
processing computer.
wrong one may make the solution extremely inefficient or even unsolvable in
practice. For example, were we required to calculate the 137th root of a SO-
digit number, I believe that most readers would agree that the conventional
number-crunching computer is more suited to the task than the human (or
chimpanzee). However, when the problem is the understanding of spoken
language, the reading of cursive handwriting, or the recognition of faces, the
architecture of the brain has proved the more viable. Indeed for these tasks
there is at least an existence proof for a neural network solution, but none
has yet been proffered for the serial computer!
Despite the above argument for neural computation, the neural network
approach has had only limited success so far. Optical Character Recognition
(OCR) engines based on neural networks have indeed eclipsed other tech-
nologies, yet progress on speech recognition has been slow. At least part of
the fault lies in the size of network we can presently build, see Figure 11.2.
Our largest systems seem to be on the level of a mentally retarded mosquito!
We are forced to conclude that our previous existence proof for neural so-
lutions to ASR, OCR, face recognition, and other problems is contrived at
best. The only way our present-day artificial neural networks will be able
to solve practical problems is by being more efficient than biology by many
orders of magnitude.
connections
set
I I I I I I I I
1020 -
0 human
0 rabbit
105 -
0 bee
1010 - * chip
* emulation0 fly
* simulation
0 worm
I I I I I I I I
loo I
loo lo2 lo4 lo6 lo8 lOlo 1012 1014 1016 connections
Figure 11.2: The speed and complexity of various neural networks. The horizontal axis
is the number of synapses in the network, a number that determines both the information
capacity and the complexity of processing attainable. The vertical axis is the number of
synaptic calculations that must be performed per second to emulate the network on a
serial computer, and is an estimate of the networks speed.
456 BIOLOGICAL SIGNAL PROCESSING
EXERCISES
11.8.1 The expert system approach to artificial intelligence is based on the tenet
that anything a human (or other rational agent) can do a standard computer
can be programmed to do. Discuss the veracity and relevance of this hypoth-
esis. What types of problems have been solved using expert systems? What
AI problems have eluded prolonged attempts at solution?
11.8.2 In the Hopfield model there are N neurons {si}i=r...~ each of which can take
on the values fl, where si = +l means that the 2th neurons is firing. The
synapse from presynaptic neuron j to postsynaptic neuron i is called Wij,
and the matrix of synaptic efficacies has zeros on the diagonal Wii = 0 and is
symmetric Wij = Wji. At any given time only one neuron updates its state;
the updating of the ith neuron is according to
after which some other neuron updates. Write a program that generates a ran-
dom symmetric zero diagonal synaptic matrix, starts at random initial states,
and implements this dynamics. Display the state of the network graphically
as a rectangular image, with si = fl represented as different colored pixels.
What can you say about the behavior of the network after a long enough
time? What happens if you update all the neurons simultaneously based on
the previous values of all the neurons? What happens in both of these cases
if the synaptic matrix is antisymmetric? General asymmetric?
11.8.3 Given P N-bit memories {~~}~~ir::;v t o b e stored, the Hebbian synaptic
matrix is defined as
which is the sum of outer products of the memories. Enhance the program
written for the previous exercise by adding a routine that inputs desired
memory images and computes the Hebbian matrix. Store P < O.lN memories
and run the dynamics starting near one of the memories. What happens?
What happens if you start far from any of the memories? Store P > 0.2N
memories and run the dynamics again. What happens now?
BIBLIOGRAPHICAL NOTES 457
Bibliographical Notes
Good general introductions to psychophysics can be found in [50, 711.
Alexander Graham Bells original vocation was speech production and much of
the early research on speech generation mechanisms was performed at Bell Labs
[54, 551. Th e c1assic formant tables [193] and the ear sensitivity curves [62] also
originated there.
Speech production mechanisms are presented in many books on speech pro-
cessing, such as [211] and in more depth in [253]. Speech perception is covered in
[253, 195, 1291. The Bark scale is formally presented in [290, 232, 61 and the me1
scale was defined in [254]. Cochlear modeling is reviewed in [5]. The application of
the psychophysical principle of masking to speech compression is discussed in [232].
The McCulloch-Pitts neuron was introduced in [171]. In 1957 a team led by
Frank Rosenblatt and Charles Wightman built an electronic neural network, which
they called the Murk I Perceptron. This device was designed to perform charac-
ter recognition. It was Rosenblatt who discovered and popularized the perceptron
learning algorithm [224]. Minsky and Paperts charming book [174] both thoroughly
analyzed the algorithm and dampened all interest in neural network research by
its gloomy predictions regarding the possibility of algorithms for more capable net-
works.
For a light introduction to the functioning of the brain, I suggest [32], while a
more complete treatment can be found in [180]. [8] is a thorough introduction to
neuron modeling from a physicists point of view. [168] is a seminal work on neural
network modeling and [169] is the companion book of computer exercises.
Part III
Architectures and
Algorithms
12
Graphical Techniques
461
462 GRAPHICAL TECHNIQUES
In addition, many common DSP tasks are highly structured in time and/or
frequency; this results in DSP block diagrams that have easily perceived
geometric structure. Finally, an algebra of graphic transformations can be
developed that allow one to simplify DSP block diagrams. Such transforma-
tions often result in reductions in computational complexity that would be
hard to derive without the graphical manipulations.
In this chapter we will consider DSP graphical techniques. The word
graphical is not used here as in computer graphics (although we advocate
the use of graphical displays for depicting DSP graphs), rather as in graph
theory. The term graph refers to a collection of points and lines between these
points. We start with a historical introduction to graph theory. Thereafter
we learn about DSP flow graphs and how to manipulate them. RAX is a
visual programming block diagram system. We describe the operation and
internals of RAX in order to acquaint the reader with this important class
of DSP tools.
G; .
Figure 12.1: All graphs with up to four nodes. We only allow arcs connecting distinct
nodes and disallow self connections.
Many of the most interesting problems in graph theory involve the num-
ber of graphs of a certain kind. A graph in which there is a path from
any node to every other node (e.g., Gi, G& Gi, G& G& G;f, G$, Gf,, G&) is
called connected, while one that has all nodes connected to all others (e.g.,
Gi , Gi, Gi, G&) is called complete.
The beginnings of graph theory are usually traced back to 1736. In that
year the famous mathematician, Leonhard Euler, considered the father of
analysis, published his solution to a puzzle that he had been working on.
Euler, who was born in Switzerland, was professor of mathematics at the
academy of St. Petersburg (founded in 1725 by Catherine, the wife of Peter
the Great). The cold weather so adversely affected his eyesight that in 1736
we find him living in the capital of East Prussia, Kiinigsberg (German for
the Kings city). This city, founded in 1255 by Teutonic knights, was the
seat of the dukes of Prussia from 1525 through 1618. After World War II
the city was annexed to the USSR and renamed Kaliningrad (Russian for
Kalinins city) after the Soviet leader M.I. Kalinin. Today it is the capital
of the Kaliningrad Oblast and is Russias sole port that does not freeze-over
in winter.
12.1. GRAPH THEORY 465
EXERCISES
12.1.1 Special types of graphs are used in electronics (schematic diagrams), physics
(Feynman diagrams), computer science (search trees), and many other fields.
Research and explain at least three such uses.
12.1.2 Why isnt K on the list of graphs with four nodes? What about w ?
12.1.3 How many different kinds of graphs are there for five nodes?
12.1.4 Draw all digraphs for 3 nodes.
12.1.5 Explain Eulers rule intuitively.
12.1.6 A graph is called nonplanar if it cannot be drawn on a piece of paper without
arcs crossing each other. Draw a nonplanar graph.
12.2. DSP FLOW GRAPHS 467
12.1.7 An Euler path is similar to an Euler cycle except that one neednt return to
the same node. Similarly we can define a Hamiltonian path. Draw Euler and
Hamilton path for points on a two-dimensional grid. Which paths are cycles?
Find Euler paths that are not Hamilton paths and vice versa.
12.1.8 A trellis is a digraph created by mapping possible transitions between N
states as a function of time. Conventionally the time axis is from left to right
and the N states are drawn vertically. There is an arc between each state
at time t and a several possible states at time t + 1. Assume a trellis with
four states (called 0, 1, 2 and 3) with states 0 and 1 at time n being able to
transition to even states at time n + 1, while 2 and 3 can only transition to
odd states. Draw this trellis from time n = 0 through 4. How many different
trellises of length L are there? How may a trellis be stored in a file? What
data structure may be used in a program?
Y
x = - 9 *= - f -= x
X+-+-Y
Yl
--c Y2
This means that the same signal is delivered to both nodes, yr = y2 = x.
Of course it is meaningless to connect more than one pre-arc node to-
gether; but we can add two signals x1 and x2 to get y = x1 + x2. This is
depicted in the standard notation using an addition node.
Xl Y
+
-T-
x2
Xl Y
-r
x2
We can combine these basic elements in many ways. For example, the
basic N = 2 DFT of equation (4.33) is depicted
X0 X0
Xl Xl
XL
470 GRAPHICAL TECHNIQUES
This diagram means that the signal y is the same as the signal x delayed one
digital unit of time, that is, yn = xn-r for all n. Often we loosely think of
the signal value at some time n as entering the delay, and its previous value
exiting. Since we only represent time-invariant systems with flow graphs
this interpretation is acceptable, as long as it is remembered that this same
operation is performed for every unit of time. Also note that we shall never
see a z in a signal flow diagram. We represent only causal, realizable systems.
Using the unit delay we can easily represent the simple difference ap-
proximation to the derivative yn = A xn = xn - x,+1
You will notice that we have drawn a small filled circle in each of these
diagrams. This circle does not represent a processing system, rather it is a
reminder that a memory location must be set aside to store a signal value.
In order to continuously calculate the simple difference or single delay con-
volution, we must store xn at every time n so that at the next time step it
will be available as ~~-1. We do not usually explicitly mark these memory
locations, only stressing them when the additional emphasis is desired.
12.2. DSP FLOW GRAPHS 471
a0 n
x = / fiY
+
17 x :
AL a0
t
A 1r
al
a0 f7
x = t-Y
+v
Al
Figure 12.3: Four other ways of drawing a basic MA (FIR) block as a DSP flow graph.
The beauty of using graphs is that we can redraw them any way we
consider Esthetic or useful, as long as the topology stays the same. Thus,
the basic single delay FIR block also can be drawn in the ways depicted in
Figure 12.3 and in many other ways.
All the different DSP graphs that depict the same DSP process are
called implementations of this process. Note that the implementation of Fig-
ure 12.3.A is topologically identical to the previous graph, but has the gains
appearing more symmetrically. In Figure 12.3.B we interchanged the order
of the gain and the delay. Thus this implementation is not identical from the
pure graph-theoretic point of view, but is an identical DSP process since the
gain and delay operators commute. Figure 12.3.C looks different but one can
easily convince oneself that it too represents the same block. Figure 12.3.D
is similar to Figure 12.3.C but with the gains positioned symmetrically.
How can we implement the more general convolution?
L
?)n = c wh-2
z=o
472 GRAPHICAL TECHNIQUES
x =
a0lr
Y
Figure 12.4: A straightforward implementation of the FIR filter using an adder of in-
degree L+ 1.
Figure 12.5: A straightforward implementation of the FIR filter using standard two-
input adders.
Once you understand the graph at this level you can return to the world
of DSP programming. The delays are now single sample delays, the pro-
cessing nodes are computations that may all be carried out by the same
processor, and the filled circles are memory locations. When thinking in this
mode we often think of typical values, such as xn and ~~-1, rather than
entire signals such as x. We implicitly allow the same computation to be
carried out over and over by a single processor. The basic computation to
be performed repeatedly consists of multiplication of a delayed input value
by a filter coefficient and adding; this combination is called a Multiply-And-
Accumulate (MAC) operation.
Looking closely at Figure 12.5 we see that this FIR implementation is
based on the block from Figure 12.3.D. Several of these blocks are con-
catenated in order to form the entire convolution. This is a widely used
technique-after perfecting an implementation we replicate it and use it
again and again. You can think of an implementation as being similar in
this regard to a subroutine in standard procedural programming languages.
P
-1
aL
Of course this is not the only way to draw an FIR filter. A particularly
interesting way to depict the same graph is shown in Figure 12.6. In this
implementation we replicate the FIR block of Figure 12.3.A. It is easy to
see that this graph is topologically identical to the previous one.
Up to now we have only seen graphs without cycles, graphs that corre-
spond to feedforward systems. The simple feedback system, yn = i&+i+zn,
is depicted as
Yn = Xn - e bmyn-m
m=l
Once again it is worthwhile to carefully mark all the arcs to be sure that
you understand how this implementation works. Dont be concerned that
signal values are transported backward in time and then influence their own
values like characters in science fiction time-travel stories. This is precisely
the purpose of using feedback (remember Section 7.4).
Of course this is not the only way to draw this AR filter. A particu-
larly interesting implementation is depicted in Figure 12.8.A. We purposely
made this implementation a mirror reflection of the FIR implementation of
Figure 12.6. Now by concatenating the MA and AR portions we can at last
implement
L
Yn = c al%-l - 5 bmYn-n-l
I=0 m=l
12.2. DSP FLOW GRAPHS 475
X-
B
P
-1
EXERCISES
12.2.1 The following examples demonstrate simplification of diagrams with gains.
In all cases identify the gain(s) that appear in the right diagram in terms of
those that appear in the left diagram.
b A
X-L-WY x Y
A
X+f--J-+YX 4 Y
C D
12.2.2 Draw an ARMA filter with MA order 3 and AR order 4. How many memory
locations are required? Label all the signals.
12.2.3 A filter implementation with minimal number of memory allocations is called
canonical. What is the number of memory locations in the canonical ARMA
filter with MA order p and AR order q?
12.2.4 The transposition theorem states that reversing all the arc directions, chang-
ing adders to tee connections and vice-versa, and interchanging the input and
output, does not alter the systems transfer function. Prove this theorem for
the simple case of yn = xn + by,-1.
a memory location
G
X Y a gain y = Gx
Xl Y
+ adding signals y = x1 + x2
where we have identified the signal 4x,-r on two different arcs. We can
consolidate everything between the input and these two signals.
12.3. DSP GRAPH MANIPULATION 479
Two signal processing systems f and g are said to commute if one can
interchange their order without changing the overall system.
In particular any two linear systems, and thus any two filters commute. We
often exploit commutation of filters to simplify DSP flow graphs.
7,
-bM
.
Figure 12.10: An alternative way to depict the ARMA filter. Here we perform the au-
toregressive (all-pole) filter before the moving average (FIR) filter.
As an example, lets simplify Figure 12.8.B for the full ARMA filter.
Note that there we actually performed the MA portion before the AR, which
would make this an MAAR filter (were this possible to pronounce). Since
the MA and AR subsystems are filters and thus commute we can place the
AR portion before the MA part, and obtain Figure 12.10.
This diagram is just as symmetric as the first but seems to portray a
different story. The signal first enters the infinite loop, cycling around and
around inside the left subsystem, and only the signal that manages to leak
480 GRAPHICAL TECHNIQUES
\,
\+ -bM-1
J+
7,
-bM aL
Figure 12.11: Yet another way to depict the ARMA filter. This graph requires the min-
imal number of internal memory locations.
EXERCISES
12.3.1 Draw the basic all-pole block in four different ways.
12.3.2 Recall that the main entity of the state-space description is the systems
internal state. One way to encode the internal state involves specifying hidden
signals. Which hidden signals are required to be identified? How does the state
thus specified evolve with time?
12.3.3 Give an example of two systems that do not commute.
12.3.4 Draw high-level hierarchical flow graphs for the following systems, and then
decompose the high-level description into primitives.
l a filter composed of several copies of the same FIR filter in series
l a band-pass filter that mixes a signal down to zero frequency and then
low-pass filters it
l a circuit that computes the instantaneous frequency
The RAX model of the world is that of racks of equipment, which are
vertical rectangular arrays. These vertical arrays are called racks; racks are
made up of slots, and each slot can hold a single piece of equipment. The piece
of equipment in a specific rack and slot position is called a box, while what a
type of equipment does is called its function. Each box has input connectors,
output connectors, and buttons. Cables can be connected between a single
output connector and any number of input connectors. Buttons are used
to set internal parameters of the different boxes (e.g., input filenames and
amplification gains).
0.000
>
GENERATOR
. . . I : SCOPE
) l +omoe
>
>
SYNTHESIZER 3 ;AMPLI FI ER
. . . . .....
1
Figure 12.12: The graphics screen of RAX for a simple setup. Note that this figures
resolution and fonts are representative of 1983 computer graphic displays (the original
had 16 colors).
never actually be used in this way since the synthesizer has adjustable am-
plitude) can take up to five inputs. Its output is the sum of these inputs,
each multiplied by its own gain (selected by appropriate buttons). The scope
has input channels for one or two signals to be displayed. Its other inputs
are for external trigger and clock. The scope has no output, but buttons
that set the volts per division, timebase, trigger mode and level, clock rate,
number of sweeps. The scope also has a display, called CRT for Cathode
Ray Tube. The output of the function generator is connected to channel B
of the scope, while the synthesizer feeds the amplifier which in turn is con-
nected to channel A of the scope. A sample scope display from this setup is
depicted in Figure 12.13.
IlS.940
Figure 12.13: The graphics screen of the scope for the simple RAX setup depicted in
the previous figure. The small + represents the position of the pointing device, and this
position is indicated in the message window.
RAX can be integrated into the real world in two ways. For non-real-time
use there are input file and output file boxes. These boxes are stream-
oriented, reading and writing as required. The input file box has one output
and a button with which the user specifies the name of the file to read.
Similarly, the output file box has a single input and a button to specify the
file to write. For simple processing, or when DSP processors are used, there
are also A/D and D/A boxes that stream to and from the true devices.
In RAX the same piece of equipment can be placed into many differ-
ent rack-slot positions, which is interpreted as different boxes that happen
to have the function. Boxes are identified by giving the function, the rack
number, and the slot number (e.g., SCOPE[2,1] ). Connectors and buttons
have their own notations. When the pointing device enters a rack-slot that
houses a piece of equipment its identifier is displayed in the message area at
the upper right. When the pointer is close enough to a button or connector,
its identifier is displayed as well. Pressing the pointers actuator (similar
to clicking a mouse) over a button causes a pop-up menu to appear where
the user can edit the corresponding parameter. Pressing the actuator near
a connector causes a rubber band line to be drawn from that connector to
the pointer, which can then be placed near another connector and pressed
again. If the connection is valid the rubber band line disappears and in its
place a connection route is drawn. Valid connections connect a single output
to any number of inputs. The connection route is drawn to avoid existing
routes, and is color coded for optimal distinguishability.
After bringing up the application, the user specifies the number of racks
and the number of slots per rack. These numbers can be changed at any time,
with the restriction that no mounted equipment should fall onto the floor.
Next the user opens the store room and drags pieces of equipment from there,
placing them into rack-slots. The user can then connect output connectors
to input connectors and set parameters using the buttons. When satisfied
the user points and depresses the run button. At that point time starts to
run and the signal processing begins. The user may at any time select any
display (e.g., from a scope, spectrum analyzer, or voltmeter) and view the
graphic results. When such a display is active, the message area continuously
displays the pointers coordinates, for example, volts and time for the scope.
To return to the racks display the user can then press the return button,
and equipment buttons can be adjusted while time is running. To stop time
from running there is a stop button. If the user considers the setup to be
useful it can be saved to a netlist file, from which it can be loaded at some
later date. The netlist for the simple setup of Figure 12.12 is printed out as
Figure 12.14.
486 GRAPHICAL TECHNIQUES
EXERCISES
12.4.1 RAX is a clock-driven system, meaning that some external concept of
time causesthe scheduler to operate. Alternatives include data-driven and
control-driven systems. In the former, external inputs are the trigger for
everything to happen; each input is followed through causing box after box
to operate in turn. In the latter, whenever a box cannot run due to an in-
put not being ready, the box connected to it is run in order to generate
that input (and recursively all boxes before it). Discuss the advantages and
disadvantages of these three techniques.
12.5. RAX INTERNALS 487
12.4.2 Observe in Figure 12.12 that different cables never overlap, at most they
cross at a point. Give a simple algorithm for accomplishing this.
12.4.3 Find some visual programming language to experiment with. How is it similar
to and how is it different from RAX?
12.4.4 What are RAXs main functional deficiencies?
initializations
main loop
handle user events
update graphics
if RunMode
schedule tasks
increment time
finalizations
This is the responsibility of the task scheduler. The scheduler runs through
every piece of equipment in the racks, and decides whether it is ready to
run. This decision is based on continuity of time and an assumption of
causality. Each box remembers when it last ran, and each input to each box
contains a time-stamped value. A box can run only when the present time
is strictly after the time it last ran, and only when the time is not before
the time of its inputs. Assuming a box can run, the scheduler is responsible
for loading the box state, connecting input and output cables, calling the
proper task (perhaps running on a DSP), and storing the new state. These
duties determine the context switch time, the minimum time it takes to
switch from running one box to another. RAX was somewhat wasteful in
this regard, having originally been designed for simulation purposes and only
later being retrofitted with DSP boards for real-time use. An alternative
strategy (one that was employed for the DSP code) is for each box to keep
its state information internally. This cannot be done using static arrays for
host code since one equipment type can be used multiple times in a single
setup.
Finally, when all boxes have run for the specified time, the time is incre-
mented according to the present sampling rate. One of the major limitations
of RAX is the use of a single sampling rate. Although the sampling rate can
be changed, there cannot be simultaneously more than one rate, and all
boxes must use the same clock. This is both an efficiency problem (some
processes might only need updating very infrequently, but must be sched-
uled every time) and a real constraint (resampling processes, such as those
required for modems, cannot be implemented). This problem could be fixed
by simulating real time using a common multiple of all desired sampling rates
and dividing as required. Hardware systems commonly implement this same
policy by using a high-frequency crystal oscillator and various frequency
dividers.
Behind the simple description of the handling of user events and sched-
uler are a plethora of infrastructure functions. For example, there is a func-
tion that given the rack and slot numbers determines whether the rack-slot
is occupied, and if so retrieves the type of equipment. Another, given a type
of equipment, finds the meaning of the inputs or buttons. Given a cable
identifier, the output number, origin rack-slot that feeds it, and all inputs it
feeds can be found. Given an entire configuration, the minimum number of
racks and slots may be calculated. Of course there are functions to place a
piece of equipment in a given position, to remove a piece of equipment, to
connect a cable between connectors, etc.
12.5. RAX INTERNALS 489
the present value or a few past ones, and we might be wasting memory
by storing the maximum number. Of course we could configure each cable
differently, choosing the number of values to store according to the maximum
needed by boxes to which the cable is connected.
Another possibility is to allow only one box to be fed by a given cable
and to make it that boxs responsibility to clear unneeded values from the
cable. In such an implementation the precable box writes to the cable, and
the post-cable box reads from it deleting the value; thus we call this method
write-read. This is similar to the mechanisms of pipes and sockets that
are provided by many operating systems, and these mechanisms can be
exploited in implementing such cables. In a system with cables that are
written and read we could allow several boxes to write to a single cable,
but only one box can read a cable since once a value has been read it is no
longer available for other boxes to use. This is not really an insurmountable
limitation since we could easily create a tee connector equipment type, which
takes in a value from one input and makes it available on two or more
outputs.
We have still not completely specified what happens in systems with
cables of the latter type. One possibility is for the read attempt to fail if
the desired values are not ready, or equivalently, to give the reading box the
ability to test for readiness before attempting to read from a cable. Such
write-test-read systems can seem to act nondeterministically, even without
explicit randomness built in. For example, consider a piece of equipment
built to merge two inputs into a single output. This equipment checks its
inputs until it finds one which is ready and writes its value to the output.
It then reads and discards all other inputs that might be simultaneously
available. Even if it always sweeps through its inputs in the same order, its
output depends on the detailed operation of the scheduler, and thus seems
unpredictable. A second possibility is for the reading box to become blocked
until the desired input value is ready. Indeed the blocking mechanism can be
used as the heart of the system instead of an explicitly encoded scheduler. All
boxes in a blocked-read system simply run in parallel, with the unblocked
boxes preparing outputs that in time unblock other processes. Finally, the
writing process may become blocked until the cable is ready to receive the
value to be sent. Although this blocked-write method seems strange at first,
it shares with blocked-read the advantage of automatically synchronizing
truly parallel processes.
Since RAX was an overwrite system, DSP processors could only be em-
ployed for their relative speed as compared with the host processor available
at that time. The potential for parallel processing could not be exploited
12.5. RAX INTERNALS 493
since the scheduler was responsible for sending data to each processor, wait-
ing for it to complete its computation, and then collecting its output. In
order to allow the processors to truly run in parallel some method of syn-
chronization, either that inherent in blocked-read and blocked-write or an
explicit interprocess communications method, must be employed. One model
that has been exploited for the parallelization of DSP tasks is Hoares com-
municating sequential processes. In this model a collection of computational
processes, each of which separately runs sequentially, run truly in parallel
and communicate via unidirectional blocked-write channels.
This completes our description of RAX internals. While some of the
details are specific to this system, many of the concepts are applicable to
any visual programming system. When using such a system for simple tasks
the analogy with analog equipment is enough to get you started, but for
more complex problems a basic understanding of the internals may mean
the difference between success and frustration.
EXERCISES
12.5.1 Write a package to implement graphics display lists as singly linked lists
of commands. At minimum there must be MOVE (x ,y> and DRAW (x, y)
commands, while more elaborate implementations will have other opcodes
such as SETCOLOR c, DRAWRECTANGLE (left, right, bottom, top), and
CIRCLE (x, y ,r> . Remember to include routines to construct a new display
list, clear the display list (as, for example, when a scope retriggers), add a
command to the display list (take into account that the display may or not be
currently showing), show the display on screen, translate between real world
coordinates and integer screen coordinates, and free the list.
12.5.2 Write a package to handle netlist files. You will need at least one routine to
read a netlist file and translate it into an internal representation of boxes and
parameters and one routine to write a netlist file.
12.5.3 Write a RAX-like GUI.
12.5.4 Write a RAX-like scheduler.
12.5.5 Implement a basic RAX system for functions that all run on the host. You
should supply at least a sine wave generator, a filter, and a scope.
494 BIBLIOGRAPHICAL NOTES
Bibliographical Notes
A good general reference on graph theory is [91], which is a newer version of a
classic text.
The present author has not found any mention of the use of flow graphs for
signal processing before the 1953 article of Mason [160, 1611.
One of the earliest uses of visual programming in DSP was BLODI (the BLOck
Diagram compiler), which was developed Bell Labs in late 1960. Although without
a true graphic interface, it was said to be easier to learn than FORTRAN, and at
times easier to use even for the experienced programmer. BLODI had blocks for IO,
signal and noise generation, arithmetic operations between signals, delay and FIR
filtering, sampling and quantization, and even a flip-flop. Other documented flow
languages for signal processing include SIGNAL [88] and LUSTRE [89]. Probably
the most popular visual programming environment for signal processing is what
was once called BOSS (Block-Oriented System Simulator) but was later renamed
SPW (Signal Processing WorkSystem).
Hoares communicating sequential processes, presented in 11031, also motivated
several DSP systems. For a good discussion of implementational issues for data-
flow-oriented languages consult [2].
13
Spectral Analysis
495
496 SPECTRAL ANALYSIS
Figure 13.1: Zero crossing detector for clean sinusoid with no DC offset. The sampling
rate is about double Nyquist (four samples per period). Note that the linear approximation
is reasonable but not perfect.
Given the signal we first compute the sequence of interpolated zero cross-
ing instants to, tl, tf~,t3 . . . and then compute the zero crossing intervals by
subtraction of successive times (the finite difference sequence) A, =tl - to,
A2= t2 - tl, A3= t3 - t2 and so on. Next we find the zero crossing derivative
as the second finite difference A2[21=A2-Al,A~1=A3-A2,A~=A4-A3
etc. If the underlying frequency is truly constant the A sequence averages
to the true frequency reciprocal and the A['] sequence is close to zero. FYe-
quency variations show up in the derivative sequence.
This is about as far as it is worth going in this direction. If the zero
crossing derivatives are not sufficient then we probably have to do some-
thing completely different. Actually zero crossings and their derivatives are
frequently used to derive features for pattern recognition purposes but al-
most never used as frequency estimators. As feature extractors they are
relatively robust, fast to calculate, and contain a lot of information about
the signal. As frequency estimators they are not reliable in noise, not par-
ticularly computationally efficient, and cannot compete with the optimal
methods we will present later on in this chapter.
EXERCISES
13.1.1 What is the condition for two signal values sn and sn+i to straddle a rising
zero crossing? A falling zero crossing? Any zero crossing?
13.1.2 Assume that we have located a rising zero crossing between times n and n+ 1.
Derive an expression for St, the fractional correction to be added to t = n,
assuming that the signal traces a straight line between sn and sn+r. Extend
to an arbitrary (rising or falling) zero crossing.
498 SPECTRAL ANALYSIS
ii0 5 sN- 1
A-
= BPF -s
-- . . . -
s =
. ..
.. .
BPF -+--iN-
Figure 13.2: A bank of filters dividing the frequency band from 0 to F into N subbands,
each containing a band-pass signal. On the left the spectrum is depicted, while the right
shows the bank of filters that accomplishes this division.
At this point we could simply add the filter outputs s together and
reconstruct the original signal s; thus the set of signals 5 contains all the
information contained in the original signal. Such an equivalent way of en-
coding the information in a signal is called a representation. The original
signal s is the time domain representation, the spectrum is the frequency
domain representation, and this new set of signals is the subband represen-
tation. Subband representations are useful in many contexts, but for now we
will only compute the energies of all the subband signals 9, obtaining an
estimate of the power spectrum. The precision of this estimate is improved
when using a larger number of subbands, but the computational burden goes
up as well.
The bank of filters approach to the PSD does not differentiate between
a clean sinusoid and narrow-band noise, as long as both are contained in the
13.2. BANK OF FILTERS 499
same subband. Even if the signal is a clean sinusoid this approach cannot
provide an estimate of its frequency more precise than the bandwidth of the
subband.
We have taken the subbands to be equal in size (i.e., we have divided
the total spectral domain into N equal parts), but this need not be the case.
For instance, speech spectra are often divided equally on a logarithmic scale,
such that lower frequencies are determined more precisely than higher ones.
This is no more difficult to do, since it only requires proper design of the
filters. In fact it is computationally lighter if we build up the representation
recursively. First we divide the entire domain in two using one low-pass
and one high-pass filter. The energy at the output of the high-pass filter is
measured, while the signal at the output of the low-pass filter is decimated
by two and then input to a low-pass and a high-pass filter. This process is
repeated until the desired precision of the lowest-frequency bin is attained.
Returning to the case of equal size subbands, we note that although all
the signals go through SN-1 have equal bandwidth, there is nonetheless a
striking lack of equality. The lowest subband so is a low-pass signal, exist-
ing in the range from 0 to 5. It can be easily sampled and stored using
the low-pass sampling theorem. All the other Sc are band-pass signals and
hence require special treatment. For example, were we required to store the
signal in the subband representation rather than merely compute its power
spectrum, it would be worthwhile to downmix all the band-pass signals to
the frequency range of 0 to $. Doing this we obtain a new set of signals we
now call simply Sk; so is exactly go, while all the other sk are obtained from
the respective s by mixing down by F. This new set of signals also con-
tains all the information of the original signal, and is thus a representation
as well. We cm call it the low-pass subband representation to be contrasted
with the previous band-pass subband representation. The original signal s is
reconstructed by mixing up each subband to its proper position and then
summing as before. The power spectrum is computed exactly as before since
the operation of mixing does not affect the energy of the subband signals.
The low-pass subband representation of a signal can be found without
designing and running N different band-pass filters. Rather than filtering
with band-pass filters and then downmixing, one can downmix first and then
low-pass filter the resulting signals (see Figure 13.3). In sequential computa-
tion this reduces to a single mixer-filter routine called N times on the same
input with different downmix frequencies. This is the digital counterpart of
the swept-frequency spectral analyzer that continuously sweeps in sawtooth
fashion the local oscillator of a mixer, plotting the energy at the output of
a low-pass filter as a function of this frequency.
500 SPECTRAL ANALYSIS
Figure 13.3: Two equivalent implementations of a bank of filters dividing the frequency
range into N low-pass signals. In (A) the band-pass signals are band-pass filtered and then
downmixed using a real mixer, while in (B) the input signal is downmixed by a complex
mixer and then low-pass filtered.
where WN is the Nth root of unity (see equation (4.30)). These products
need to be low-pass filtered in order to build the sk. If we choose to imple-
ment the low-pass filter as a causal FIR filter, what should its length be?
From an information theoretic point of view it is most satisfying to choose
length N, since then N input samples are used to determine N subband
representation values. Thus we find that the kth low-pass signal is given by
N-l N-l
s,k = c h s ,-i$kn = hnsnWEk (13.1)
nn c
n=O n=O
13.2. BANK OF FILTERS 501
which looks somewhat familiar. In fact we can decide to use as our low-
pass filter a simple moving average with all coefficients equal to one (see
Section 6.6). Recall that this is a low-pass filter; perhaps not a very good
one (its frequency response is a sine), but a low-pass filter all the same. Now
we can write
N-l N-l
(13.2)
n=O n=O
which is, of course, precisely the DFT. However, instead of thinking of the
DFT as providing the frequency domain representation of a signal, here
we consider it as calculating the low-pass subband representation. In this
fashion the DFT becomes a tool for efficiently simultaneously downmixing
and filtering the signal. The mixers are easily seen in the definition of the
DFT; the filtering is implicit in the sum over N input values.
We have to acclimate ourselves to this new interpretation of the DFT.
Rather than understanding Sk to be a frequency component, we interpret sk
as a time domain sample of a subband signal. For instance, an input signal
consisting of a few sinusoids corresponds to a spectrum with a few discrete
lines. All subband signals corresponding to empty DFT bins are correctly
zero, while sinusoids at bin centers lead to constant (DC) subband signals.
So the interpretation is consistent for this case, and we may readily convince
ourselves that it is consistent in general.
We have seen that in our bank of filters approach to computing the power
spectrum we actually indirectly compute the DFT. In the next section we
take up using the DFT to directly estimate the power spectrum.
EXERCISES
13.2.1 The low-pass subband representation can be useful in other contexts as well.
Can you think of any? (Hint: FDM.)
13.2.2 Why does the bank of filters approach become unattractive when a large
number of filters must be used?
13.2.3 Compare the following three similar spectral analysis systems: (1) a bank of
N + 1 very steep skirted analog band-pass filters spaced at Af from 0 to
F = NAf; (2) a similar bank of N + 1 digital filters; (3) a single DFT with
bin size Af. We inject a single sinusoid of arbitrary frequency into each of
the three systems and observe the output signal (note that we do not observe
only the energy). Do the three give identical results? If not, why not?
13.2.4 Compare the computational complexity of the recursive method of finding
the logarithmic spectrum with the straightforward method.
502 SPECTRAL ANALYSIS
13.2.5 Prove that the energy of a band-pass signal is unchanged when it is mixed
to a new frequency range.
13.2.6 We saw that the DFT downmixes the subbands before filtering, and we know
that a mixer is not a filter. In what senseis the DFT equivalent to a bank
of filters? How can we empirically measure the frequency responseof these
filters?
13.2.7 Build a bank of filters spectrum analyzer using available filter design or FFT
software. Inject static combinations of a small number of sinusoids. Can you
always determine the correct number of signals?Plot the outputs of the filters
(before taking the energy). Do you get what you expect? Experiment with
different numbers of bins. Inject a sinusoid of slowly varying frequency. Can
you reconstruct the frequency response of the filters? What happens when
the frequency is close to the border between two subbands?
-50dB
0 16 32 46 64 0 f $2
Figure 13.4: Leakage in the spectrum of a single sinusoid. In (A) precisely 15 cycles of
the sinusoid fit into the buffer of length 64 samples and thus its periodogram contains a
single line. In (B) 15.04 cycles fit into the buffer and thus there is a small discontinuity
at the edge. The periodogram displays leakage into neighboring bins. In (C) 14; cycles fit
and thus the discontinuity and leakage are maximal. Note also that the two equal bins are
almost 4 dB lower than the single maximal bin in the first case, since the Parseval energy
is distributed among many bins.
Figure 13.5: The effect of windowing with a noninteger number of cycles in the DFT
buffer. Here we see a signal with 43 cycles in the buffer. After replication to the left and
right the signal has the maximum possible discontinuity.
13.3. THE PERIODOGRAM 505
EXERCISES
13.3.1 Show directly, by expressing the sample s~N+~ outside the buffer in terms
of the complex DFT coefficients sk, that computing the N-point DFT corre-
sponds to replicating the signal in the time domain.
13.3.2 Plot the energy in a far bin as a function of the size of the discontinuity. (Its
enough to use a cosine of digital frequency a and observethe DC.) Why isnt
it practical to use a variable-length rectangular window to reduce leakage?
13.3.3 Is signal discontinuity really a necessarycondition for leakage?If not, what
is the exact requirement? (Hint: Try the sinusoid sin(27r(lc+ i)/N).)
13.3.4 As the length of the buffer grows the number of discontinuities per time
decreases,and thus we expect the spectral SNR to improve. Is this the case?
13.3.5 In the text we discussedthe half-sine window function. Trying it for a fre-
quency right on a bin boundary (i.e., maximal discontinuity) we find that it
works like a charm, but not for other frequencies. Can you explain why?
13.4 Windows
In Sections 4.6 and 13.3 we saw the general requirements for window func-
tions, but the only explicit examples given were the rectangular window and
the somewhat unusual half-sine window. In this section we will become ac-
quainted with many more window functions and learn how to window shop,
that is, to choose the window function appropriate to the task at hand.
Windows are needed for periodograms, but not only for periodograms.
Windows are needed any time we chop up a signal into buffers and the signal
is taken to be periodic (rather than zero) outside the observation buffer. This
is a very frequent occurrence in DSP! When calculating autocorrelations (see
Chapter 9) the use of windows is almost universal; a popular technique of
designing FIR filters is based on truncating the desired impulse response
13.4. WINDOWS 507
by a window (see Section 7.8); sample buffers are windowed before LPC
analysis (see Section 9.9); and the list goes on and on. Yet windowing as
a preprocessing stage for the periodogram is probably the best known use,
and we will concentrate on it here. Recalling the interpretation of the FT
as a bank of FIR band-pass filters, we will see that the frequency response
of these filters is directly determined by the window function used.
We must, once again, return to the issue of buffer indexing. The com-
puter programming convention that the buffer index runs from 0 to N - 1
is usually used with a window that obeys w. = 0 and 20~ = 0. In this fash-
ion the first point in the output buffer is set to zero but the last point is
not (the N th point, which is zero, belongs to the next buffer). Some people
cannot tolerate such asymmetry and make either both w. = 0, wNwl = 0 or
w-1 = 0,WN = 0. These conventions should be avoided! The former implies
two zeroed samples in the replicated signal, the latter none. In theoretical
treatments the symmetric buffer indexation 44. . . M with M E g is com-
mon, and here only one of the endpoints is to be considered as belonging to
the present buffer. To make things worse the buffer length may be even or
odd, although FFT buffers will usually be of even length. As a consequence
you should always check your window carefully before looking through it.
We will present expressions in two formats, the practical 0. . . N - 1 with
even N and w. = 0, WN = 0 and the symmetric odd length -M . . . M with
w&m = 0 and thus N = 2M + 1. To differentiate we will use an index n for
the former case and m for the latter.
The rectangular window is really not a window function at all, but we
consider it first for reference. Measuring analog time in units of our sampling
interval, we can define an analog window function w(t) that is one between
t = -M and t = +M and zero elsewhere. We know that its FT is
W(w) = M sinc(Mw)
and its main lobe (defined between the first zeros) is of width g. As M
increases the main lobe becomes narrower and taller, but if we increase the
frequency resolution, as allowed by the uncertainty theorem, we find that the
number of frequency bins remains the same. In fact in the digital domain
the N = 2M point FFT has a frequency resolution of J$ (the sampling
frequency is one), and thus the main lobe is two frequency bins in width for
all M. It isnt hard to do all the mathematics in the digital domain either.
The digital window is 20~ = 1 for -M 5 m 5 +M and 20~ = 0 elsewhere.
508 SPECTRAL ANALYSIS
where we have used formula (A.46) for the sum of a finite geometric series,
and substituted N = 2M + 1.
Prom this expression we can derive everything there is to know about the
rectangular window. Its main lobe is two bins in width, and it has an infinite
number of sidelobes, each one bin in width. Its highest sidelobe is attenuated
13.3 dB with respect to the main lobe, and the sidelobes decay by 6 dB per
octave, as expected of a window with a discontinuity (see Section 4.2).
Before we continue we need some consistent quantities with which to
compare windows. One commonly used measure is the noise bandwidth de-
fined as the bandwidth of an ideal filter with the same maximum gain that
would pass the same amount of power from a white noise source. The noise
bandwidth of the rectangular window is precisely one, but is larger than
one for all other windows. Larger main lobes imply larger noise bandwidths.
Another important parameter is the ripple of the frequency response in the
pass-band. The rectangular window has almost 4 dB pass-band ripple, while
many other windows have much smaller ripple. We are now ready to see some
nontrivial windows.
Perhaps the simplest function that is zero at the buffer ends and rises
smoothly to one in the middle is the triangular window
wn = Wm = 1-2L I I (13.4)
M+l
which is also variously known as the Bartlett window, the Fejer window, the
Parzen window, and probably a few dozen more names. This window rises
linearly from zero to unity and then falls linearly back to zero. If the buffer
is of odd length there is a point in the middle for which the window function
is precisely unity, for even length buffers all values are less than one. The
highest sidelobe of the triangular window is 26 dB below the main lobe, and
the sidelobes decay by 12 dB per octave, as expected of a window with a
first derivative discontinuity. However, the noise bandwidth is 1.33, because
the main lobe has increased in width.
The Harming window is named after the meteorologist Julius von Hann.
Apparently the verb form to Hann the data was used first; afterward people
started to speak of Hanning the signal, and in the end the analogy with the
Hamming window (see below) caused the adoption of the misnomer Hanning
window. The Hanning window is also sometimes called the cosine squared,
or raised cosine window (use the m index to see why). The Hanning
windows main lobe is twice as wide as that of the rectangular window, and
at least three spectral lines will always be excited, even for the best case.
The noise bandwidth is 1.5, the highest sidelobe is 32 dB down, and the
sidelobes drop off by 18 dB per octave.
The Hamming window is named in honor of the applied mathematician
Richard Wesley Hamming, inventor of the Hamming error-correcting codes,
creator of one of the first programming languages, and author of texts on
numerical analysis and digital filter design.
Wn =,-~,cos(2~~)+.,cos(2++z~cos(2*~)... (13.7)
h
cl 16 32 43 84
Figure 13.6: Various window functions. Depicted are 64-point rectangular, triangular,
Hanning, Hamming, Blackman, and Kaiser wkdows. -
OdB OdB
II r
A B
0 v 0 P
OdB OdB
r
E F
-5OdB I I
0 P
Figure 13.7: Periodogram of worst-case single sinusoids using various window functions,
namely (A) rectangular, (B) triangular, (C) Harming, (D) Hamming, (E) Blackman-Harris,
and (F) Kaiser, Each periodogram is normalized such that its maximum height corresponds
to 0 dB.
13.4. WINDOWS 511
Afterward the choice is not clear cut. The Blackman and Kaiser windows
reduce the sidelobe height, but cannot simultaneously further reduce the
main lobe width. The Hamming window attempts to narrow the main lobe,
but ends up with higher distant sidelobes. Not shown is a representative of
the Dolph-Chebyshev family, which as can be assumed for anything bearing
the name Chebyshev, has constant-height sidelobes.
Which window function is best? It all depends on what you are trying to
do. Rectangular weighting could be used for sinusoids of precisely the right
frequencies, but dont expect that to ever happen accidentally. If you are
reasonably sure that you have a single clean sinusoid, this may be verified
and its frequency accurately determined by using a mixer and a rectangular
window STFT; just remember that the signals frequency is the combination
of the bins frequency and the mixer frequency. An even trickier use of the
rectangular window is for the probing of linear systems using synthetically
generated pseudorandom noise inputs (see Section 5.4). By using a buffer
length precisely equal to the periodicity of the pseudorandom signal we
can ensure that all frequencies are just right and the rectangular weighted
STFT spectra are beautiful. Finally, rectangular windows should be used
when studying transients (signals that are nonxero only for a short time).
We can then safely place the entire signal inside the buffer and guarantee
zero signal values at the buffer edges. In such cases the rectangular window
causes the least distortion and requires the least computation.
For general-purpose frequency displays the Hanning and Hamming win-
dows are often employed. They have lower sidebands and lower pass-band
ripple than the rectangular window. The coefficients of the Hanning window
neednt be stored, since they are derivable from the FFTs twiddle factor
tables. Another trick is to overlap and average adjacent buffers in such a
way that the time weighting becomes constant.
A problem we havent mentioned so far is twc>-tone separability. We
sometimes need to separate two closely spaced tones, with one much stronger
than the other. Because of main lobe width and sidelobe height, the weaker
tone will be covered up and not noticeable unless we choose our window
carefully. For such cases the Blackman, Dolph-Chebyshev, or Kaiser windows
should be used, but we will see stronger methods in the following sections.
512 SPECTRAL ANALYSIS
EXERCISES
13.4.1 Convert the Hanning and Hamming windows to symmetric m notation and
explain the names cosine squared and raised cosine often applied to the for-
mer. Express the Hanning window as a convolution in the frequency domain.
What are the advantages of this approach?
13.4.2 Plot the periodograms for the same window functions as in Figure 13.7, but
for a best-case sinusoid (e.g., for N = 64, a sinusoid of frequency 15/64).
13.4.3 Plot periodograms of the logistics signal for various 1 5 X < 3.57, as was
done in Section 5.5. Which window is best? Now use X that give for 3, 5, and
6 cycles. Which window should be used now?
13.4.4 Try to separate two close sinusoids, both placed in worst-case positions, and
one much stronger than the other. Experiment with different windows.
where u is the uncorrelated white noise. Its autocorrelations are easily de-
rived
A2
Cs(m) = (SnSn+m) = 2 cos(wm) + 0$5~,0 (13.9)
and the first few lags are given by the following.
A2
G(O) = T+o;
A2
G(l) = -yj- cos(w)
A2 A2
G(2) = 2 cos(2w) = 2 (2co4w) - 1)
The noise only influences the lag zero term (energy) due to the assumption
of white noise. Any deviation from whiteness causes the other lags to acquire
noise-related terms as well.
Were the noise to be zero, we could simply calculate
W = cos-l cm
-
( cs (0) >
but this fails miserably when noise is present. Can we find an expression that
uses only nonzero lags, and is thus uninfluenced by the noise? Pisarenkos
method uses only the two lags m = 1 and m = 2. Using the trigonometric
identity cos(2w) = 2cos2(w) - 1 it is easy to show that
W) &L cm +2
=EJi) 24c,2(1)
\i (13.10)
only one of which leads to the correct solution (see exercise 13.5.2). We thus
find
w = cos-l -cs (2) + &gn (G(l)) (13.11)
4c, (1)
514 SPECTRAL ANALYSIS
which is the PHD estimate for the digital angular frequency (the analog
frequency is obtained by dividing by 27r and multiplying by the sampling
frequency).
The PHD expression we have just found is not a frequency estimate at
all. Assuming the noise is perfectly white, it is an infinitely precise mea-
surement of the frequency. Of course there is no problem with this infinite
precision since we assumed that we have exact values for the two autocorre-
lation lags C9(1) and Cs (2). Obtaining these exact values requires knowing
the signal over all time, and therefore the uncertainty theorem does allow
infinitely precise predictions. However, even when we use empirical autocor-
relations (equation (9.11)) calculated using only N samples the prediction
still seems to be perfectly precise. Unlike periodogram methods there is no
obvious precision reduction with decreasing N; but the accuracy of the pre-
diction decreases. It is straightforward, but somewhat messy, to show that
the variance of the PHD estimator is inversely proportional to the size of
the buffer and the square of the SNR (SNR = $).
Y
co424 + cos2(w) 1
&UI = sin2(w)(cos(2ti) + 2)2 N SNR2
EXERCISES
13.5.1 Derive the expression (13.9) for the autocorrelation (use exercise 9.2.12).
13.5.2 Exploiting the fact that we want 0 5 w < x show that the proper solution of
the quadratic has the sign of C8(1).
13.5.3 In the text we quoted the variance of the error of the PHD estimation. What
about its bias? Find this numerically for various buffer sizes.
13.5.4 The PHD is a second-orderfrequency estimator in the sensethat the highest
autocorrelation lag it utilizes is m = 2. Using the trigonometric identity
cos(w) + cos(3w) = 2 cos(clr)cos(2w) prove that Cs(l) - 2C,(2)c + C8(3) = 0.
Show that this leads to the following third-order estimator.
w = cos-l
(G(l) + G(3)
2G (2) >
13.5.5 Compare the third-order estimator of the previous exercisewith the PHD by
generating sinusoids in various amonnts of white noise and estimating their
frequencies. Which is better for low SNR? High SNR? Small buffer size?
Large buffer size?
Ul = 2cos(w)
Xn = al%--1 + C12Xn-2 where (13.12)
u2 = -1
516 SPECTRAL ANALYSIS
(we have simply defined al = -cl and a2 = -0~). We will call al and
a2 recursion coeficients. Given the recursion coefficients we can write the
equation
2 =
- ( %,%--1,%-2 >
a =
(1 9 -al, -a2)
-x-a=0
- (13.14)
i.e., the clean signal vector and the recursion coefficient vector are orthog-
onal. Now the noisy signal is s = z + u. This signal has mean zero (since
we assume the noise to be zer,me&) &d its covariance matrix is thus the
signal autocorrelation matrix
thus see that the frequency of a sinusoid in additive noise can be deter-
mined by diagonalizing its autocorrelation matrix. This is a specific case of
the desired formulation of the PHD.
We can now understand the term decomposition that appears in the name
PHD. The decomposition is that of the covariance matrix into signal-related
and noise-related parts (see equation (13.15)) which implies the splitting
of the (2p + 1)-dimensional space of signal vectors into orthogonal signal
(equation (13.14)) and noise subspaces.
The proof of the general p case is similar to that of the p = 1 case. The
key idea is that signals consisting of p real exponentials or sinusoids obey
difference equations; those of real exponentials involve p previous values,
while p sinusoids obey recursions involving 2p previous values. Its easier to
understand this by first considering the exponential case.
P
xn = Ai eQin
c
i=l
and thus
P
ake-qik = 1
c
k=l
518 SPECTRAL ANALYSIS
we see that
1- 5 akzvk
k=l
has roots z = eqi. Similarly for the sum of sinusoids
Xn =
p
a . Aielwn + Are -iwn
>
i=l
we leave it as an exercise to show that
2P
Xn =
c akxn-k (13.18)
k=l
where
2P
1 - c a&z- (13.19)
k=l
and for N = 2p the matrix is square Toeplitz and the equations can be
readily solved for the ak. Once we have the al, we can find the roots of the
polynomial, and retrieve the qi . Thereafter the Ai can be found as explained
above. Thus Pronys method reduces the solution of a very nonlinear prob-
lem to the solution of two linear problems and the (nonlinear) operation of
finding the roots of a polynomial.
EXERCISES
13.6.1 Why are the eigenvaluesof CZ nonnegative?
13.6.2 Complete the proof of the PHT for general p. To do this prove equation (13.18)
and the claim about the roots.
13.6.3 Specialize the PHD back to p = 1 and show that we obtain our previous
PHD equation (13.11).
13.6.4 What is the computational complexity of the PHD for general p?
13.6.5 Pronys method as described works only for noiselesssignals. How can it be
extended to the noisy case?
13.6.6 Extend Pronys method to p sinusoids.
520 SPECTRAL ANALYSIS
44 = zT(s,) = 5 s,z-~
n=--00
calculated on the unit circle. Thus corresponding to the moving STFT there
is a STzT
Sm(Z) = 2 SnZBn (13.20)
n=m-N+l
where we have not explicitly shown a window function and have chosen the
causal indexing convention. At time n = 0 this reduces to
0 N-l
SO(%) = C Sn%-n = C S-nZn (13.21)
n=-N+l n=O
(13.22)
13.7. IIR METHODS 521
which is not a bad assumption if the autocorrelations die off rapidly enough.
This is easily seen to be an all-zero approximation, and leads to the blur-
ring of sharp spectral lines. In 1967, John Burg introduced an alternative
assumption, that the spectral estimate should be the most random spectrum
consistent with the lags we do have. By most random we mean the spec-
trum with the highest entropy, and thus this technique is called maximum
entropy spectral analysis.
The reasoning behind the maximum entropy principle is easy to under-
stand. DFT methods assume that all data that has not been observed either
consist of periodic repetition of the data we have seen or are identically zero.
There is usually little physical evidence for such assumptions! Maximum en-
tropy means that we should remain as open minded as possible regarding
unseen data. Indeed Burgs method actually tells us to use the most unpre-
dictable extrapolation of the data possible. There are many possible spectra
that are consistent with the data we have collected, each corresponding to a
different extrapolation of the data; maximum entropy insists that the most
likely spectrum is that corresponding to the least constraints on the un-
known data. In other words we should assume that the uncollected data is
as random as possible.
What type of approximation corresponds to the maximum entropy as-
sumption? In Section 18.6 we will see, and if you have studied thermody-
namics you already know, that maximum randomness means maximization
of the entropy. We assume that the entropy
OH dlnS as(w) dw 1
0
= X,(m) = J dS XYJm) = J
qqe
-imw h
13.8. WALSH FUNCTIONS 523
We see that the Fourier coefficients of the reciprocal of S(w) are zero for
Im( > M (i.e., the inverse spectrum is a finite Fourier series). Accordingly,
the maximum entropy power spectrum can be written as the reciprocal of a
finite Fourier series, that is, is all-pole.
EXERCISES
13.7.1 Generate a signal that is the sum of a small number of sinusoids and noise.
Find the PSD via the periodogram. Solve the LPC equations and derive the
spectrum using equation (13.24). Compare the results. Experiment by placing
weak spectral lines close to strong ones (recall exercise 13.4.4).
13.7.2 Record some speech and compute its all-pole spectrum. What features do
you observe? Can you recognize different sounds from the PSD?
wall01(t) = 1 (13.28)
callkl (t) = wa1L2l(t) = waP1 2t + (-1) wall1 (2(t - a))
( >
sal[+ll(t) = wal[2k+11(t) = wall] (2t) - (-1) wal[l (2(t - i))
and assume all of the functions to be zero to t < 0 and t < 1. After thus
defining the functions on the unit interval we extend the definitions periodi-
cally to the entire t axis. Note that the wal functions are a single family like
the complex exponentials, while the Sal and Cal functions are analogous
524 SPECTRAL ANALYSIS
waP
3
salr31
NTqb
waP1 q
sa1141
Figure 13.8: The first eight Walsh functions in order of increasing sequency. The cal
functions are on the left and the sal functions on the right.
to sine and cosine. The label k equals the number of transitions in the unit
interval and is called the sequency.
The value-by-value product of two Walsh functions is always a Walsh
function
wal[l (t ) wal bl (t ) = walIkl (t )
where k is obtained by bit-by-bit xor of i and j. From this and the fact that
all Walsh functions except wal[l are DC-free it is easy to prove the required
orthonormality property.
1
walIil (t) walli] (t) dt = 6.213
.
s0
N-l
& =
c
n=O
where the normalization was chosen according to our usual convention. This
looks very similar to the DFT. The two-point transform is identical to that
of the DFT
x0 = x0 + Xl
Xl = x()-x1
EXERCISES
13.8.1 Plot Sal(t) and overlay it with sin(2?rt); similarly plot Cal(t) and overlay
cos(2nt). What is the relationship?
13.8.2 What is the connection between the Walsh functions and the Hadamard
matrices defined in exercise 14.5.3?
13.8.3 Find a nonrecursive formula for wallkl(t).
13.8.4 Write a program that computes the decimation in sequencyfast Walsh trans-
form (see Section 14.3).
13.8.5 Since the DWT can be computed without multiplications and using only real
arithmetic, it would be useful to be able to obtain the DFT from the DWT.
How can this be done?
13.8.6 In the Fourier case multiplication is related to convolution. What is the
analogous result for Walsh transforms?
526 SPECTRAL ANALYSIS
where we defined cas(t) = cos(t) + sin(t). Note that the Hartley transform is
its own inverse (to within normalization). Similarly we can define the discrete
Hartley transform.
N-l
1
& = - c
N n=O
N-l
2, = c
k=O
How do you retrieve the power spectrum from the Hartley transform? Ob-
tain the DFT from the discrete Hartley transform. Develop a fast Hartley
transform (see Section 14.3).
13.9 Wavelets
No modern treatment of spectral analysis could be complete without men-
tioning wavelets. Although there were early precursors, wavelet theory orig-
inated in the 1980s when several researchers realized that spectral analysis
based on basis functions that are localized in both frequency and time could
be useful and efficient in image and signal processing.
What exactly is a wavelet? A basic wavelet is a signal $(t) of finite time
duration. For example, a commonly used basic wavelet is the sinusoidal pulse
where the prefactor normalizes the energy. The time translation to is simple
to understand, it simply moves the wavelet along the time axis. The time
duration of the wavelet is proportional to r; conversely, you can think of the
scaling transformation compressing the time scale for r > 1 and stretching
it for r < 1. The center frequency is inversely proportional to r (i.e., the
frequency axis scales in the opposite way to the time axis).
What about the wavelets bandwidth? Since the nonzero bandwidth re-
sults from the finite time duration via the uncertainty theorem, the band-
width must scale inversely to r. This last statement can be made more
explicit by borrowing the filter design concept of the Q.
Af
Q=x (13.32)
Since the center frequency f = 27rwo and the bandwidth both scale inversely
with r, all the wavelets $(7(t), to, t) have the same Q.
We can now build a transform based on these wavelets by replacing the
infinite-duration sinusoids of the FT by finite-duration wavelets.
(13.33)
-9
f0 2fO VO 4fo
.f
fo 2fo 4fO 8fO
Figure 13.9: Comparison of Fourier and wavelet transforms. The top figure depicts a
four bin DFT while the bottom is a four-bin wavelet transform. Note that the FT bins
have constant bandwidth while those of the wavelet transform have constant Q. For the
purposes of illustration we have taken the basic wavelet to be rectangular in the frequency
domain.
time resolution gets better with higher frequency, while the frequency reso-
lution becomes better at low frequencies (longer time durations). So using
a single wavelet transform we can perform both measurements.
Digital wavelet transforms can be computed efficiently (a fast wavelet
transform) using the pyramid algorithm, which extends the recursive com-
putation of the logarithmic spectrum discussed in Section 13.2. We employ
a pair of filters called Quadrature Mirror Filters (QMFs). The QMF pair
consists of a low-pass FIR filter that passes the lower half of the spectrum
and a high-pass FIR filter that passes the upper half. The two filters are
required to be mirror images of each other in the spectral domain, and in
addition they must guarantee that the original signal may be recovered.
The simplest QMF pair is (a, 4) and (3, -&, the first being low-pass, the
second high-pass, and their sum obviously the original signal. The pyramid
algorithm works as follows. First we apply the QMF filters to the incoming
signal, creating two new signals of half the original bandwidth. Since these
signals are half bandwidth, we can decimate them by a factor of two with-
13.9, WAVELETS 529
EXERCISES
13.9.1 Use a raised cosine times a sine as a basic wavelet and draw the scaled
wavelets for various r. Compare these with the basis functions for the STFT.
139.2 A digital QMF pair obeys IIOpl(f)l = II-W(~ - f)l, where H[~P]I(~) and
I#131 (f) are the frequency responses of the low-pass and high-pass filters.
Show that /$I = (-l)nhpl or &?I = (-1) n h,,[IPI for odd length filters
and similar statements for even length ones. Show that the latter form is
consistent with the wavelets being an orthogonal basis for even length filters.
13.9.3 Can the original signal really be recovered after QMF filtering and decimation
have been applied?
139.4 Derive an efficient procedure for computing the inverse digital wavelet trans-
form.
13.9.5 Build a signal consisting of two close sinusoids that pulse on and off. Similarly
build a signal that consists of a single sinusoid that appears as two close
pulses. Try to simultaneously measure frequency and time phenomena using
the STFT and the wavelet transform.
13.9.6 Compare the wavelet transform with the time-frequency distributions dis-
cussed in Section 4.6.
530 BIBLIOGRAPHICAL NOTES
Bibliographical Notes
Kay and Marple have written a good tutorial of modern approaches to spectral
analysis [128], and each has written a book as well [127, 1231.
The periodogram waa first introduced by Schuster [233] and became an even
more indispensable tool after the introduction of the FFT. Blackman and Tukey
wrote early book on the practical calculation of power spectra [19]. The bible of
windows was written by Harris [93].
As noted in the bibliographical notes to Chapter 6, Yule [289] formulated the
Yule-Walker equations for signals containing one or two sinusoidal components in
the late 192Os,in an attempt to explain the 11-year periodicity of sunspot num-
bers. Walker, calling Yules earlier work an important extension of our ideas re-
garding periodicity, expanded on this work, discovering that the autocorrelations
were much smoother than the noisy signal itself, and suggesting using the corre-
lation periodogram as a substitute for the Schuster periodogram. He applied this
technique to the analysis of air pressuredata and could rule out as spurious various
claimed periodicities.
Wiener was instrumental in explaining why the Schuster periodogram did not
work well for noisy signals [276, 2771,but this was not widely appreciated at the
time. A highly interesting historical account is given by Robinson [223].
Pisarenkos original article is [196]. Various authors have analyzed the perfor-
mance of the PHD [227, 255, 2851.
Officer of the U.S. Navy and Harvard mathematician Joseph Leonard Walsh
presentedhis functions in 1923 [268]. The standard text is [12] and a short introduc-
tion can be found in [13]. The conversion between DWT and DFT was expounded
in [256, 2571.Hartley, who was in charge of telephone research at Bell Labs and
responsible for an early analog oscillator, presented his transform in 1942 [94]. The
DHT and FHT were published in [24, 251.The standard text is [27].
Wavelets already have a rich literature. For DSP purposes we recommend the
review article by Rioul and Vetterli [221].
14
531
532 THE FAST FOURIER TRANSFORM
n=O
-i2
WNze N =cos($) - isin
How many calculations must we perform to find one XI, from a set of N
time domain values x *? Assume that the complex constant WN and its pow-
ers W$ have all been precalculated and stored for our use. Looking closely
at (4.32) we see N complex multiplications and N - 1 complex additions are
performed in the loop on n. Now to find the entire spectrum we need to do
N such calculations, one for each value of k. So we expect to have to carry
out N2 complex multiplications, and N(N - 1) complex additions.
This is actually a slight overestimate. By somewhat trickier programming
we can take advantage of the fact that Wk = 1, so that each of the Xk>O
takes N - 1 multiplications and additions, while X0 doesnt require any
multiplications. We thus really need only (N - 1) 2 complex multiplications.
A complex addition requires the addition of real and imaginary parts,
and is thus equivalent to two real additions. A complex multiplication can
be performed as four real multiplications and two additions (a + ib)(c +
id) = (ac - bd) + i(bc + ad) or as three multiplications and five additions
(a+ib)(c+id) = u(c+d)-d(u+b)+i (u(c+d) +c(b-a)). Thelatter form
may be preferred when multiplication takes much more time than addition,
but can be less stable numerically. Other combinations are possible, but it
can be shown that there is no general formula for the complex product with
less than three multiplications. Using the former, more common form, we
find that the computation of the entire spectrum requires 4(N - 1)2 real
multiplications and 2(N - 1)(2N - 1) real additions.
Actually, the calculation of a single XI, can be performed more efficiently
than we have presented so far. For example, Goertzel discovered a method of
transforming the iteration in equation (4.32) into a recursion. This has the
effect of somewhat reducing the computational complexity and also saves
the precomputation and storage of the W table. Goertzels algorithm, to be
presented in Section 14.8, still has asymptotic complexity of order O(N) per
calculated XI,, although with a somewhat smaller constant than the direct
14.1. COMPLEXITY OF THE DFT 533
buffer size is purely a memory issue, and does not impact the ability to keep
up with real-time. However, if the complexity is superlinear (for example,
T processing = qNp with p > 1) , then as N increases we have less and less
time to process each sample, until eventually some N is reached where we
can no longer keep up, and data-overrun is inevitable.
Lets clarify this by plugging in some numbers. Assume we are acquiring
input at a sample rate of 1000 samples per second (i.e., we obtain a new
sample every millisecond) and are attempting to process blocks of length
250. We start our processor, and for one-quarter of a second, we cannot do
any processing, until the first acquisition buffer fills. When the buffer is full
we quickly switch buffers, start processing the 250 samples collected, while
the second buffer of length 250 fills. We must finish the processing within
a quarter of second, in order to be able to switch buffers back when the
acquisition buffer is full. When the dependence of the processing time on
buffer length is strictly linear, Tpro,-essing = qN, then if we can process a
buffer of N = 250 samples in 250 milliseconds or less, we can equally well
process a buffer of 500 samples in 500 milliseconds, or a buffer of N = 1000
samples in a second. Effectively we can say that when the single sample
processing time is no more than q = 1 millisecond per sample, we can
maintain real-t ime processing.
What would happen if the buffer processing time depended quadratically
on the buffer siz~Tproce+n~ = qN2? Lets take q to be 0.1 millisecond per
sample squared. Then for a small lo-millisecond buffer (length N = lo),
we will finish processing in Tprocessing = 0.1 lo2 = 10 milliseconds,
l just
in time! However, a lOO-millisecond buffer of size N = 100 will require
T processing = 0.1 1002 milliseconds,
l or one second, to process. Only by
increasing our computational power by a factor of ten would we be able to
maintain real-time! However, even were we to increase the CPU power to
accommodate this buffer-size, our 250-point buffer would still be out of our
reach.
As we have mentioned before, the FFT is an algorithm for calculating
the DFT more efficiently than quadratically, at least for certain values of N.
For example, for powers of two, N = 2, its complexity is 0( N log2 N). This
is only very slightly superlinear, and thus while technically the FFT is not
suitable for real-time calculation in the asymptotic N --) 00 limit, in practice
it is computable in real-time even for relatively large N. To grasp the speed-
up provided by the FFT over direct calculation of (4.29)) consider that the
ratio between the complexities is proportional to ,*N. For N = 24 = 16 the
FFT is already four times faster than the direct DFT, for N = 21 = 1024
14.1. COMPLEXITY OF THE DFT 535
it is over one hundred times faster, and for N = 216 the ratio is 4096! It
is common practice to compute lK- or 64.Kpoint FFTs in real-time, and
even much larger sizes are not unusual.
The basic idea behind the FFT is the very exploitation of the N2 com-
plexity of the direct DFT calculation. Due to this second-order complexity,
it is faster to calculate a lot of small DFTs than one big one. For example, to
calculate a DFT of length N will take N2 multiplications, while the calcula-
tion of two DFTs of length $ will take 2($)2 = $, or half that time. Thus
if we can somehow piece the two partial results together to one spectrum in
less than T time then we have found a way to save time. In Sections 14.3
and 14.4 we will see several ways to do just that.
EXERCISES
14.1.1 Finding the maximum of an N-by-N array of numbers can be accomplished in
O(N2) time. Can this be improved by partitioning the matrix and exploiting
the quadratic complexity as above?
14.1.2 In exercise 4.7.4 you found explicit
equations for the N = 4 DFT for N = 4.
Count up the number of complex multiplications and additions needed to
compute X0, Xi, X2, and Xs. How many real multiplications and additions
are required?
14,1.3 Define temporary variables that are used more than once in the above equa-
tions. How much can you save? How much memory do you need to set aside?
(Hint: Compare the equations for X0 and X2.)
14.1.4 Up to now we have not taken into account the task of finding the trigonomet-
ric W factors themselves, which can be computationally intensive. Suggest at
least two solutions, one that requires a large amount of auxiliary memory but
practically no CPU, and one that requires little memory but is more CPU
intensive.
.4.1.5 A computational system is said to be real-time-oriented when the time
it takes to perform a task can be guaranteed. Often systems rely on the
weaker criterion of statistical real-time, which simply means that on-the-
average enough computational resources are available. In such cases double
buffering can be used in the acquisition hardware, in order to compensate
for peak MIPS demands. Can hardware buffering truly make an arbitrary
system as reliable as a real-time-oriented one?
14.1.6 Explain how double-buffering can be implemented using a single circular
buffer.
536 THE FAST FOURIER TRANSFORM
into two subsequences of half the original size (assume for simplicitys sake
that N is even)? One way is to consider pairs of adjacent elements, such as
xi, x2 or x3, x4, and place the smaller of each pair into the first subsequence
and the larger into the second. For example, assuming x0 < x1, 22 > x3 and
XN-2 < x&r, we obtain
four? The splitting would then cost four comparisons per quartet, or N
comparison altogether, and then two $ searches must be carried out. Thus
we require N + 2: and a factor of l$ is still needed. Indeed, after a little
reflection, the reader will reach the conclusion that no further improvement
is possible. This is because the new problems of finding only the minimum
or maximum of a subsequence are simpler than the original problem.
When a problem can be reduced recursively to subproblems similar to the
original, the process may be repeated to attain yet further improvement. We
now discuss an example where such recursive repetition is possible. Consider
multiplying two (N+ 1)-digit numbers A and B to get a product C using long
multiplication (which from Section 6.8 we already know to be a convolution).
AN AN-~ *a* Al A0
BN BN-~ *** B1 Bo
BOAN BoAN-~ -0* BoAl BoAo
&AN &AN-I . . . &Ao
Since we must multiply every digit in the top number by every digit in
the bottom number, the number of one-digit multiplications is N2. You are
probably used to doing this for decimal digits, but the same multiplication
algorithm can be utilized for N-bit binary numbers. The hardware-level
complexity of straightforward multiplication of two N-bit numbers is pro-
portional to N2.
Now assume N is even and consider the left $ digits and the right G
digits of A and B separately. It does not require much algebraic prowess to
convince oneself that
A = AL2+ +AR
B = BL2+ +BR (14.1)
C = ALBUMS + (ALBR + ARBL)2+ + ARBR
= ALB~(~~ + 23) + (AL - AR)(B~ - BL)2+ + ARBR(2+ + 1)
just begun! Unlike for the previous example, we have reduced the original
multiplication problem to three similar but simpler multiplication problems!
We can now carry out the three $-bit multiplications in equation (14.1)
similarly (assuming that $ is still even) and continue recursively. Assum-
ing N to have been a power of two, we can continue until we multiply
individual bits. This leads to an algorithm for multiplication of two N-bit
numbers, whose asymptotic complexity is O(NO~Z(~)) M 0( N1*585). The
slightly more sophisticated Toom-Cook algorithm divides the N-bit num-
bers into more than two groups, and its complexity can be shown to be
O(N log(N)2-). Th is is still not the most efficient way to multiply
numbers. Realizing that each column sum of the long multiplication in equa-
tion (14.1) can be cast into the form of a convolution, it turns out that the
best way to multiply large numbers is to exploit the FFT!
EXERCISES
14.2.1 The reader who has implemented the Shell sort may have used a different
method of choosing the pairs of elements to be compared. Rather than com-
paring adjacent elements x21 and x21+1, it is more conventional to consider
elements in the same position in the first and second halves the sequence,
21, and x++k Write down a general form for the new sequence. How do we
find the minimum and maximum elements now? These two ways of dividing
a sequence into two subsequences are called decimation and partition.
14.2.2 Devise an algorithm for finding the median of N numbers in O(N log N).
14.2.3 The product of two two-digit numbers, ab and cd, can be written ab * cd =
(10 * a + b) * (10 * c + d) = 1OOac + lO(ad + bc) + bd. Practice multiplying
two-digit numbers in your head using this rule. Try multiplying a three-digit
number by a two-digit one in similar fashion.
14.2.4 We often deal with complex-valued signals. Such signals can be represented
as vectors in two ways, interleaved
or separated
Put another way, partition divides the sequence into two groups according
to the MSB of the index, while decimation checks the LSB.
Either partition and decimation may be employed to separate the orig-
inal signal into half-sized signals for the purpose of computation reduction.
Decimation in time implies partition in frequency (e.g., doubling the time
duration doubles the frequency resohtion), while partition in time signifies
decimation in frequency. These two methods of division lead to somewhat
different FFT algorithms, known conventionally as the Decimation In Time
(DIT) and Decimation In Frequency (DIF) FFT algorithms. We will here
consider radix-2 partition and decimation, that is, division into two equal-
length subsequences. Other partition and decimation radixes are possible,
leading to yet further FFT algorithms.
We will now algebraically derive the radix-2 DIT algorithm. We will need
the following trigonometric identities.
w,N = 1
w; = -1 (14.2)
W2N = WN
v
540 THE FAST FOURIER TRANSFORM
The FFTs efficiency results from the fact that in the complete DFT
many identical multiplications are performed multiple times. As a matter of
fact, in XI, and Xk+TN all the multiplications are the same, although every
other addition has to be changed to a subtraction.
N-l
n=O
N-l
N-l
= C Xn WEk (-l)n
n=O
which, not accidentally, is very similar to the two-point DFT diagram pre-
sented in Section 12.2 (actually, remembering the second identity of (14.2)
we recognize the two-point FFT as a special case of the DIT butterfly). Ro-
tating the diagram by 90 and using some imagination clarifies the source
of the name butterfly. We will soon see that the butterfly is the only oper-
ation needed in order to perform the FFT, other than calculation (or table
lookups) of the twiddle factors.
Now, is this method of computation more efficient than the straightfor-
ward one? Instead of (N - 1)2 multiplications and N(N - 1) additions for
the simultaneous computation of XI, for all k, we now have to compute two
g-point DFTs, one additional multiplication (by the twiddle factor), and
two new additions, for a grand total of 2(+ - 1)2 + 1 = g - 2N + 3 multi-
plications and 2 (6 ($ - 1)) + 2 = y - N + 2 additions. The savings may
already be significant for large enough N!
But why stop here? We are assuming that XF and Xf were computed
by the straightforward DFT formula! We can certainly save on their compu-
tation by using the recursion as well! For example, we can find Xf by the
following butterfly.
xp--y+&-xf
542 THE FAST FOURIER TRANSFORM
x0- -X0
x1- -x1
x2- -x2
x3-
8-point -x3
x4-
DFT ---x4
x5- --x5
a- -X3
x7- -x7
53 f
4-point
DFT
55 b
a-point
DFT
2-point
DFT
t - w-
x0 -
x3 =
Bit
Reversal
x4 :
26 =
Figure 14.5: The full eight-point radix-2 DIT DFT, with bit reversal on inputs.
imation uses the LSB of the indices to decide how to divide the sequence
into subsequences, so it is only natural to investigate the effect of in-place
decimation on the binary representation of the indices. For example, for 24
element sequences, there are four stages, the indices of which are permuted
as follows.
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111
0 2 4 6 8 10 12 14 1 3 5 7 9 11 13 15
0000 0010 0100 0110 1000 1010 1100 1110 0001 0011 0101 0111 1001 1011 1101 1111
0 4 8 12 2 6 10 14 1 5 9 13 3 7 11 15
0000 0100 1000 1100 0010 0110 1010 1110 0001 0101 1001 1101 0011 0111 1011 1111
0 8 4 12 2 10 6 14 1 9 5 13 3 11 7 15
0000 1000 0100 1100 0010 1010 0110 1110 0001 1001 0101 1101 0011 1011 0111 1111
Looking carefully we observe that the elements of the second row can be
obtained from the matching ones of the first by a circular left shift. Why is
that? The first half of the second row, 0246. . ., is obtained by an arithmetic
left shift of the first row elements, while the second half is identical except
for having the LSB set. Since the second half of the first row has the MSB
set the net effect is the observed circular left shift.
The transition from second to third row is a bit more complex. Elements
from the first half and second half of the second row are not intermingled,
rather are separately decimated. This corresponds to clamping the LSB and
circularly left shifting the more significant bits, as can be readily verified
in the example above. Similarly to go from the third row to the fourth we
546 THE FAST FOURIER TRANSFORM
clamp the two least significant bits and circularly shift the rest. The net
effect of the m stages of in-place decimation of a sequence of length 2m is
bit reversal of the indices.
In order to unshuffle the output of the DIT FFT we just need to perform
an initial stage of bit reversal on the xn, as depicted in Figure 14.5. Although
this stage contains no computations, it may paradoxically consume a lot of
computation time because of its strange indexing. For this reason many
DSP processors contain special addressing modes that facilitate efficient
bit-reversed access to vectors.
EXERCISES
14.3.1 Draw the flow diagram for the 16-point DIT FFT including bit reversal.
(Hint: Prepare a large piece of paper.)
14.3.2 Write an explicitly recursive program for computation of the FFT. The main
routine FFT(N ,X> should first check if N equals 2, in which case it replaces
the two elements of X with their DFT. If not, it should call FFT(N/2, Y) as
needed.
14.3.3 Write a nonrecursive DIT FFT routine. The main loop should run m =
log, N times, each time computing J!L
2 butterflies. Test the routine on sums
of sinusoids. Compare the run time of this routine with that of N straight-
forward DFT computations.
14.3.4 Rather than performing bit reversal as the first stage, we may leave the inputs
and shuffle the outputs into the proper order. Show how to do this for an
eight-point signal. Are there any advantages to this method?
14.3.5 Write an efficient high-level-language routine that performs bit reversal on
a sequence of length N = 2m. The routine should perform no more than N
element interchanges, and use only integer addition, subtraction, comparison,
and single bit shifts.
N-l
--Y l N-l
kE
XI, = c XnWEk = c XnWEk + C XnWzkWN2 (14.5)
n=O n=O n=- N2
--: l --
= c x;wgk+ c x,R
WEk
T l
n=O n=O
1
Now lets compare the even and odd XI, (decimation in the frequency ao-
main). Using the fact that Wi = W+
--i l
x2k = x,L wtk + x,R w;k W,)
CC
n=O
--
T l
x2k+l = x,L W$k + xf wtk Wjyk) WE
C(
n=O
which is different from the DIT butterfly, mainly in the position of the
twiddle factor.
x0 x0 = . , x0
x4 = = Xl
- x2
x6
t-
I
I--
I
x7
Figure 14.6: Full eight-point radix-2 DIF DFT, with bit reversal on outputs.
x(-j = x0 + Xl + x2 + 23
X1 = 20 - izr - x2 + ix3
(14.7)
x2 = x0 - Xl + x2 - 23
x3 = X0 + izl - x2 - ix3
which graphically is
X0
Xl
x2
x3
X0
Xl
x2
x3
we are surprised to see that only eight complex additions and no multipli-
cations are needed. Thus it is more efficient to compute a four-point DFT
using radix-2 butterflies than radix-4! However, this does not mean that for
large N the radix-2 FFT is really better. Recall that when connecting stages
of DIT butterflies we need to multiply half the lines with twiddle factors,
leading to 0( g log2 N) multiplications. Using radix-4 before every stage we
550 THE FAST FOURIER TRANSFORM
EXERCISES
14.4.1 Complete the derivation of the radix-2 DIF FFT both algebraically and
graphically. Explain the origin of the bit reversal.
14.4.2 Redraw the diagram of the eight-point radix-2 DIT so that its inputs are
in standard order and its outputs bit reversed. This is Cooley and Tukeys
original FFT! How is this diagram different from the DIF?
14.4.3 Can a radix-2 algorithm with both input and output in standard order be
performed in-place?
14.4.4 In addition to checking the LSB (decimation) and checking the MSB (par-
tition) we can divide sequencesin two by checking other bits of the binary
representation of the indices. Why are only DIT and DIF FFT algorithms
popular? Design an eight-point FFT based on checking the middle bit.
14.4.5 A radix-2 DIT FFT requires a final stage of bit reversal. What is required
for a radix-4 DIT FFT? Demonstrate this operation on the sequence0. . .63.
14.4.6 Write the equations for the radix-8 DFT butterfly. Explain how the FFT842
algorithm works.
14.4.7 Filtering can be performed in the frequency domain by an FFT, followed by
multiplying the spectrum by the desired frequency response, and finally an
IFFT. Do we need the bit-reversal stage in this application?
14.4.8 Show how to compute a 15-point FFT by decimating the sequenceinto five
subsequencesof length three. First express the time index n = 0. . .14 as
n = 3ni + n2 with n1 = 0, 1,2,3,4 and n2 = 0, 1,2 and the frequency index
in the opposite manner k = ICI + 5k2 with ICI = 0, 1,2 and Cc2= 0, 1,2 (these
are called inclez maps). Next rewrite the FFT substituting these expressions
for the indices. Finally rearrange in order to obtain the desired form. What is
the computational complexity? Compare with the straightforward DFT and
with the 16-point radix-2 FFT.
552 THE FAST FOURIER TRANSFORM
14.4.9 Research the prime factor FFT. How can the 15-point FFT be computed
now? How much complexity reduction is obtained?
14.4.10 Show that if N = IIni then the number of operations required to perform
the DFT is about (Eni)(IIn,).
(g)=(/; -j -j(;z)
which is a factoring of the DFT matrix into the product of two sparser
matrices. So far we have not gained anything since the original matrix mul-
tiplication -X = --W4x took 42 = 16 multiplications, while the rightmost
matrix times x takes eight and then the left matrix times the resulting vec-
tor requires a-further eight. However, in reality there were in the original
only six nontrivial multiplications and only four in the new representation.
Now for the trick. Reversing the middle two columns of the rightmost
matrix we find that we can factor the matrix in a more sophisticated way
14.5. THE MATRIX INTERPRETATION OF THE FFT 553
(14.8)
(14.9)
where Tm is the diagonal m-by-m matrix with elements IVim, the twiddle
factors for the 2m-dimensional DFT.
Looking carefully at the matrices we recognize the first step in the de-
composition of the DFT that leads to the radix-2 DIT algorithm. Reading
from right to left (the way the multiplications are carried out) the column
permutation Cm is the in-place decimation that moves the even-numbered
elements up front; the two Wm are the half size DFTs, and the leftmost
matrix contains the twiddle factors. Of course, we can repeat the process for
the blocks of the middle matrix in order to recurse down to -W2.
EXERCISES
14.5.1 Show that by normalizing WN by -& we obtain a unitary matrix. What are
Z
its eigenvalues?
14.5.2 Define pi to be the DFT matrix after bit-reversal permutation of its rows.
Write d= mz and m.. Show that md can be written as follows.
C C C
H2=
C ( >
1 1
1 -1
)
Build the Hadamard matrices for m = 2 and 3. Show that they are symmetric
and orthogonal. What is the inverse of Hsm?
14.5.4 Use the Hadamard matrix instead of W to define a transform. How many
additions and multiplications are needed0 compute the transform? What are
the twiddle factors? What about bit reversal? Can we compute the Hadamard
transform faster by using higher radix algorithms?
Where do the tables come from? For most applications the size N of the
FFT is decided early on in the code design, and the twiddle factor tables can
be precomputed and stored as constants. Many DSP processors have special
table memory that is ideal for this purpose. For general-purpose (library)
routines the twiddle factor tables are usually initialized upon the call to the
FFT routine. On general-purpose computers one can usually get away with
calling the library trigonometric functions to fill up the tables; but on DSPs
one either stores only entire table in program code memory, or stores only
WN itself and derives the rest of the required twiddle factors using
FFT input buffer. The FFT output will be nonzero only in the lath bin, and
so all the energy of the input signal will be concentrated into a single large
number. For example, if the signal was pure DC zn = 1, the only nonzero
bin is Xe = C xn = N, and similarly for all other single-bin cases. It is thus
clear that if the input signal almost filled the dynamic range of the fixed
point word, then the single output bin will most certainly overflow it.
The above argument may lead you to believe that overflow is only of
concern in special cases, such as when only a single bin or a small number of
bins are nonzero. We will now show that it happens even for the opposite case
of white noise, when all the output bins are equal in size. Prom Parsevals
relation for the DFT (4.42) we know that (using our usual normalization), if
the sum of the input squared is E2, then the sum of the output squared will
be NE2. Hence the rms value of the output is greater by a factor fl than
the input rms. For white noise this implies that the typical output value is
greater than a typical input value by this factor!
Summarizing, narrow-band FFT components scale like N while wide-
band, noise-like components scale like fl. For large N both types of out-
put bins are considerably larger than typical input bins, and hence there is
a serious danger of overflow. The simplest way to combat this threat is to
restrict the size of the inputs in order to ensure that no overflows can occur.
In order to guarantee that the output be smaller than the largest allowed
number, the input must be limited to k of this maximum. Were the input
originally approximately full scale, we would need to divide it by N; result-
ing in a scaling of the output spectrum by k as well. The problem with this
prescaling is that crudely dividing the input signal by a factor of N increases
the numeric error-to-signal ratio. The relative error, which for floating point
processing was proportional to log2 N, becomes approximately proportional
to N. This is unacceptably high.
In order to confine the numeric error we must find a more sophisticated
way to avoid overflows; this necessitates intervening with the individual com-
putations that may overflow, namely the butterflies. Assume that we store
complex numbers in two memory locations, one containing the real part and
the other the imaginary part, and that each of these memories can only store
real numbers between -1 and +l. Consider butterflies in the first stage of
a radix-2 DIT. These involve only addition and subtraction of pairs of such
complex numbers. The worst case is when adding complex numbers both of
which are equal to +l, -1, +i or -i, where the absolute value is doubled.
Were this worst case to transpire at every stage, the overall gain after log2 N
stages would be 2@ = N, corresponding to the case of a single spectral
14.6. PRACTICAL MATTERS 557
line. For white noise xn we see from the DIT butterfly of equation (14.4)
XI, = x,E+ w;x,o
xk++ = x,E - w;x,o
Due to the butterflies working with maximal inputs the round-off error is
significantly reduced as compared with that of the input prescaling method.
Some numeric error is introduced by each butterfly, but this noise is itself
reduced by a factor of two by the failsafe butterfly; the overall error-to-signal
ratio is proportional to fi. A further trick can reduce the numeric error
still more. Rather than using the failsafe butterfly throughout the FFT, we
can (at the expense of further computation) first check if any overflows will
occur in the present stage. If yes, we use the failsafe butterfly (and save the
appropriate scaling factor), but if not, we can use the regular butterfly. We
leave as an exercise to show that this data-dependent prescaling, does not
require double computation of the overflowing stages.
EXERCISES
14.6.1 We can reduce the storage requirements of twiddle factor tables by using
trigonometric symmetries (A.22). What is the minimum size table needed
for N = 8? In general? Why are such economies of space rarely used?
14.6.2 What is the numeric error-to-signal ratio for the straightforward computation
of the DFT, for floating and fixed point processors?
14.6.3 In the text we discussed the failsafe prescaling butterfly. An alternative is
the failsafe postscaling butterfly, which divides by two after the butterfly is
computed. What special computational feature is required of the processor
for postscaling to work? Explain data-dependent postscaling. How does it
solve the problem of double computation?
14.6.4 In the text we ignored the problem of the finite resolution of the twiddle
factors. What do you expect the effect of this quantization to be?
signals often arise in practice. These most common applications cases are
zero-padding, interpolation, zoom-FFT and real-valued signals.
We have mentioned the use of zero-padding (adding zeros at the end of
the signal) to force a signal to some useful length (e.g., a power of two) or
to increase spectral resolution. It is obvious that some savings are obtain-
able in the first FFT stage, since we can avoid multiplications with zero
inputs. Unfortunately, the savings do not carry over to the second stage
of either standard DIT or DIF, since the first-stage butterflies mix signal
values from widely differing places. Only were the zero elements to be close
in bit-reversed order would the task of pruning unnecessary operations be
simple.
However, recalling the time partitioning of equation (14.5) we can per-
form the FFT of the fully populated left half and that of the sparse right half
separately, and then combine them with a single stage of DIF butterflies. Of
course the right half is probably not all zeros, and hence we cant realize all
the savings we would wish; however, its right half may be all-zero and thus
trivial, and the combining of its two halves can also be accomplished in a
single stage.
Another application that may benefit from this same ploy is interpo-
lation. Zero-padding in the time domain increased spectral resolution; the
dual to this is that zero-padding in the frequency domain can increase time
resolution (i.e., perform interpolation). To see how the technique works, as-
sume we want to double the sampling rate, adding a new signal value in
between every two values. Assume further that the signal has no DC com-
ponent. We take the FFT of the signal to be interpolated (with no savings),
double the number of points in the spectrum by zero-padding, and finally
take the IFFT. This final IFFT can benefit from heeding of zeros; and were
we to desire a quadrupling of the sampling rate, the IFFTs argument would
have fully three-quarters of its elements zero.
Using a similar ploy, but basing ourselves in the time decimation of
equation (14.3), we can save time if a large fraction of either the even-
or odd-indexed signal values are zero. This would seem to be an unusual
situation, but once again it has its applications.
Probably the most common special case is that of real-valued signals.
The straightforward way of finding their FFT is to simply use a complex
FFT routine, but then many complex multiplications and additions are per-
formed with one component real. In addition the output has to be Hermi-
tian symmetric (in the usual indexation this means XN-~ = Xz) and so
computation of half of the outputs is redundant. We could try pruning the
computations, both from the input side (eliminating all operations involv-
560 THE FAST FOURIER TRANSFORM
ing zeros) and from the output side (eliminating all operations leading to
unneeded results), but once again the standard algorithms dont lend them-
selves to simplification of the intermediate stages. Suppose we were to make
the mistake of inputting the vector of 2N real signal values Ri into a com-
plex N-point FFT routine that expects interleaved input (Rc, la, RI, 11, . . .
where xi = Ri + i&). Is there any way we could recover? The FFT thinks
that the signal is x0 = & + iRr, x1 = RQ+ iR3, etc. and computes a single
spectrum XI,. If the FFT routine is a radix-2 DIT the even and odd halves
are not mixed until the last stage, but for any FFT we can unmix the FFTs
of the even and odd subsequences by the inverse of that last stage of DIT
butterflies.
EXERCISES
14.7.1 How can pruning be used to reduce the complexity of zero-padded signals?
Start from the diagram of the eight-point DIF FFT and assume that only
the first two points are nonzero. Draw the resulting diagram. Repeat with
the first four points nonzero, and again with the first six points.
14.7.2 How can the FFT of two real signals of length N be calculated using a single
complex FFT of length N?
14.8. GOERTZELS ALGORITHM 561
14.7.3 How can the FFT of four real symmetric (2~~~ = s,) signals of length N
be calculated using a single complex FFT of length N?
14.7.4 We are interested only in the first two spectral values of an eight-point FFT.
Show how pruning can reduce the complexity. Repeat with the first four and
six spectral values.
Hl H2 H3 H4
L2- ;I
L3-
L4-
1 H4 1 1633 1
Figure 14.7: The DTMF telephony signalingmethod. (Note: The A, B, C, and D tones
are not available on the standard telephone,being reservedfor specialuses.)
N-l
xk = xnvN-n = x()vN + xlvN-l + . . . + xN-2v2 + XN-lv
c
n=O
xi =xOvN-l
+xlvN-2
+...+32N4V +LEN-1
which translates to the following recursion.
and will now show that the Q are indeed real-valued. Substituting into the
recursion step
pn + Xn + Pn-1V
Qn - W&n-l + xn + (Qn-1 - W&n-2) V
Qn + xn + (V + W)Qn-1 + (WV)Qn-2
Qn + xn + A&n-l - Qn-2
must be computed at the end. This recursion requires only a single frequency-
dependent coefficient A and requires keeping two lags of Q. Computationally
there is only a single real multiplication and two real additions per iteration,
for a total of N - 1 multiplications and 2(N - 1) additions.
There is basically only one design parameter to be determined before
using Goertzels algorithm, namely the number of points N. Goertzels al-
gorithm can only be set up to detect frequencies of the form f = fifs where
fs is the sampling frequency; thus selecting larger N allows finer resolution
in center frequencies. In addition, as we shall see in the exercises, larger N
implies narrower bandwidth as well. However, larger N also entails longer
computation time and delay.
14.9. FIFO FOURIER TRANSFORM 565
EXERCISES
14.8.1 Since Goertzels algorithm is equivalent to the DFT, the power spectrum
response (for w = 2&/N) is P(k) = Show that the half
power point is at 0.44, i.e., the half power bandwidth is 0.88 bin, where each
bin is simply $. What is the trade-off between time accuracy and frequency
accuracy when using Goertzels algorithm as a tone detector?
14.8.2 DTMF tones are allowed to be inaccurate in frequency by 1.5%. What would
be the size of an FFT that has bins of about this size? How much computation
is saved by using Goertzels algorithm?
14.8.3 DTMF tones are used mainly by customers, while telephone companies use
different multitone systems for their own communications. In North America,
telephone central offices communicate using MF trunk tones 700, 900, 1100,
1300, 1500, 1700, 2600 and 3700 Hz, according to the following table.
Tone 1 2 3 I 4 5 6
Frequencies 700+900 700+1100 900+1100 700+1300 900+1300 1100+1300
Tone 7 8 9 0 KP ST
Frequencies 700+ 1500 900+ 1500 1100+1500 1300+1500 1 1100+1700 1500+1700
All messages start with KP and end with ST. Assuming a sampling rate of 8
KHz, what is the minimum N that exactly matches these frequencies? What
will the accuracy (bandwidth) be for this N? Assuming N is required to be
a power of two, what error will be introduced?
14.8.4 Repeat the previous question for DTMF tones, which are purposely chosen to
be nonharmonically related. The standard requires detection if the frequen-
cies are accurate to within 51.5% and nonoperation for deviation of 53.5%
or more. Also the minimal on-time is 40 milliseconds, but tones can be trans-
mitted at a rate of 10 per second. What are the factors to be considered when
choosing N?
EXERCISES
14.9.1 For what types of MA filter coefficients are there FIFO algorithms with
complexity independent of N?
14.9.2 Sometimes we dont actually need the recomputation for every input sample,
but only for every T samples. For what r does it become more efficient to use
the FFT rather than the FIFOFT?
14.9.3 Derive a FIFOFT that uses N complex additions and multiplications per
desired frequency component, for the case of resetting the clock.
N-l
xk, = c
xm + nWEk (14.13)
n=O
14.9.4 The FIFOFT as derived above does not allow for windowing of the input
signal before transforming. For what types of windows can we define a moving
average FT with complexity independent of N?
568 BIBLIOGRAPHICAL NOTES
Bibliographical Notes
More detail on the use of FFT like algorithms for multiplication can be found in
Chapter 4 of the second volume of Knuth (1361.
An early reference to computation of Fourier transforms is the 1958 paper by
Blackman and Tukey that was later reprinted as a book [19].
The radix-2 DIT FFT was popularized in 1965 by James Cooley of IBM and
John Tukey of Princeton [45]. Cooley recounts in [43] that the complexity reduction
idea was due to Tukey, and that the compelling applications were military, including
seismic verification of Russian compliance with a nuclear test ban and long-range
acoustic detection of submarines. Once Cooley finished his implementation, IBM
was interested in publishing the paper in order to ensure that such algorithms did
not become patented. The first known full application of the newly published FFT
was by an IBM geophysicist named Lee Alsop who was studying seismographic
records of an earthquake that had recently taken place in Alaska. Using 2048 data
points, the FFT reduced the lengthy computation to seconds.
Gordon Sande, a student of Tukeys at Princeton, heard about the complexity
reduction and worked out the DIF algorithm. After Cooley sent his draft paper
to Tukey and asked the latter to be a co-author, Sande decided not to publish his
work.
Actually, radix-2 FFT-like algorithms have a long history. In about 1805 the
great mathematician Gauss [177] used, but did not publish, an algorithm essentially
the same as Cooley and Tukeys two years before Fouriers presentation of his
theorem at the Paris Institute! Although eventually published posthumously in
1866, the idea did not attract a lot of attention. Further historical information is
available in [44].
The classic reference for special real-valued FFT algorithms is [248]. The split-
radix algorithm is discussed in [57, 247, 561.
The prime factor FFT was introduced in [137], based on earlier ideas (e.g.
[240, 291) an d an in-place algorithm given in [30]. The extension to real-valued
signals is given in [98].
Winograds prime factor FFT [283, 2841 is based on a reduction of a DFT of
prime length N into a circular convolution of length N - 1 first published as a
letter by Rader [214]. A good account is found in the McClellan and Rader book
on number theory in DSP [166].
Goertzels original article is [77]. The MAFT is treated in [7, 2491. The zoom
FFT can be found in [288].
A somewhat dated but still relevant review of the FFT and its applications can
be found in [16] and much useful material including FORTRAN language sources
came out of the 1968 Arden House workshop on the FFT, reprinted in the June
1969 edition of the IEEE Transactions on Audio and Electroacoustics (AU-17(2)
pp. 66-169). Many original papers are reprinted in [209, 1661). Modern books on
the FFT include [26, 28,31, 2461 and Chapter 8 of [241]. Actual code can be found
in the last reference, as well as in (311, [30, 247, 171, [41, 1981 etc.
15
In this chapter we will delve more deeply into the practical task of using
digital filters. We will discuss how to accurately and efficiently implement
FIR and IIR filters.
You may be asking yourself why this chapter is important. We already
know what a digital filter is, and we have (or can find) a program to find the
coefficients that satisfy design specifications. We can inexpensively acquire
a DSP processor that is so fast that computational efficiency isnt a concern,
and accuracy problems can be eliminated by using floating point processors.
Arent we ready to start programming without this chapter?
Not quite. You should think of a DSP processor as being similar to a jet
plane; when flown by a qualified pilot it can transport you very quickly to
your desired destination, but small navigation errors bring you to unexpected
places and even the slightest handling mistake may be fatal. This chapter is
a crash course in digital filter piloting.
In the first section of this chapter we discuss technicalities relating to
computing convolutions in the time domain. The second section discusses
the circular convolution and how it can be used to filter in the frequency
domain; this is frequently the most efficient way to filter a signal. Hard
real-time constraints often force us to filter in the time domain, and so we
devote the rest of the chapter to more advanced time domain techniques.
We will exploit the graphical techniques developed in Chapter 12 in order
to manipulate filters. The basic building blocks we will derive are called
structures, and we will study several FIR and IIR structures. More complex
filters can be built by combining these basic structures.
Changing sampling rate is an important application for which special
filter structures known as polyphuse filters have been developed. Polyphase
filters are more efficient for this application than general purpose structures.
We also deal with the effect of finite precision on the accuracy of filter
computation and on the stability of IIR filters.
569
570 DIGITAL FILTER IMPLEMENTATION
where the index i runs over the inputs, assuming these exist. This variation
is still causal in nature, but describes inputs that have already been placed in
an array by the calling application. Rather than dedicating further memory
inside our convolution routine for the FIFO buffer, we utilize the existing
buffering and its indexation. This variation is directly suitable for off-line
15.1. COMPUTATION OF CONVOLUTIONS 571
assuming that the requisite inputs exist. This symmetry comes at a price;
when we get the n th input, we can compute only the (n- X) th output. This
form makes explicit the buffer delay of X between input and output.
In all the above procedures, we assumed that the input signal existed
for all times. Infinite extent signals pose no special challenge to real-time
systems but cannot really be processed off-line since they cannot be placed
into finite-length vectors. When the input signal is of finite time duration and
has only a finite number N of nonzero values, some of the filter coefficients
will overlap zero inputs. Assume that we desire the same number of outputs
as there are inputs (i.e., if there are N inputs, n = 0,. . . N - 1, we expect N
outputs). Since the input signal is identically zero for n < 0 and n 2 N, the
first output, yo, actually requires only X + 1 multiplications, namely uoxo,
~1x1, through U-XXX, since al through a~ overlap zeros.
Likewise the last X outputs have the filter overlapping zeros as well.
... aA ax-1 ... a2 al a0 a-1 a-2 . . . a-x+1 a-A
. . . XN-1 XN-2 ... 2N-2 XN-1 XN 0 0 ... 0 0
The programming of such convolutions can take the finite extent into ac-
count and not perform the multiplications by zero (at the expense of more
complex code). For example, if the input is nonzero only for N samples
starting at zero, and the entire input array is available, we can save some
computation by using the following sums.
min(N-1,n) min(N-l,n+X)
Yn = c an-iXi = c an-iXi (15.5)
i=max(O,n-(l-l)) i=max(O,n-A)
The improvement is insignificant for N >> L.
We have seen how to compute convolutions both for real-time-oriented
cases and for off-line applications. We will see in the next section that these
straightforward computations are not the most efficient ways to compute
convolutions. It is almost always more efficient to perform convolution by
going to the frequency domain, and only harsh real-time constraints should
prevent one from doing so.
EXERCISES
15.1.1 Write two routines for array-based noncausal convolution of an input signal
x by an odd length filter a that does not perform multiplications by zero.
The routine convolve (N, L, x, a, y> should return an output vector y of
the same length N as the input vector. The filter should be indexed from 0
to L- 1 and stored in reverse order (i.e., a0 is stored in a [L-II ) . The output
yi should correspond to the middle of the filter being above xi (e.g., the first
and last outputs have about half the filter overlapping nonzero input signal
values). The first routine should have the input vectors index as the running
index, while the second should use the filters index.
15.1.2 Assume that a noncausal odd-order FIR filter is symmetric and rewrite the
above routines in order to save multiplications. Is such a procedure useful for
real-time applications?
15.1.3 Assume that we only want to compute output values for which all the filter
coefficients overlap observed inputs. How many output values will there be?
Write a routine that implements this procedure. Repeat for when we want
all outputs for which any inputs are overlapped.
15.2. FIR FILTERING IN THE FREQUENCYDOMAIN 573
XN- x2
il
a0
Figure 15.1: Circular convolution for a three-coefficient filter. For shifts where the index
is outside the range 0.. . N - 1 we assume it wraps around periodically, as if the signal
were on a circle.
where mod is the integer modulus operation (see appendix A.2) that always
returns an integer between 0 and L - 1. Basically this means that when the
filter is outside the signal range rather than overlapping zeros we wrap the
signal around, as depicted in Figure 15.1.
Linear and circular convolution agree for all those output values for which
the filter coefficients overlap true signal values; the discrepancies appear
only at the edges where some of the coefficients jut out. Assuming we have
a method for efficiently computing the circular convolution (e.g., based on
the FFT), can it somehow be used to compute a linear convolution? Its not
hard to see that the answer is yes, for example, by zero-padding the signal
to force the filter to overlap zeros. To see how this is accomplished, lets take
a length-l signal x0 . . . XL- 1, a length M filter a0 . . . aM- 1, and assume that
M < L. We want to compute the L linear convolution outputs ye . . . y~-i.
The L - M + 1 outputs YM-1 through y~-r are the same for circular and
linear convolution, since the filter coefficients all overlap true inputs. The
other M - 1 outputs yo through PM-2 would normally be different, but if we
artificially extend the signal by x-M+1 = 0, through x-r = 0 they end up
being the same. The augmented input signal is now of length N = L+ M - 1,
and to exploit the FFT we may desire this N to be a power of two.
15.2. FIR FILTERING IN THE FREQUENCY DOMAIN 575
Figure 15.2: Overlap save method of filtering in the frequency domain. The input signal
zn is divided into blocks of length 15, which are augmented with M - 1 values saved from
the previous block, to fill a buffer of length N = L + M - 1. Viewed another way, the
input buffers of length N overlap. The buffer is converted to the frequency domain and
multiplied there by N frequency domain filter values. The result is converted back into
the time domain, M - 1 incorrect values discarded, and L values output.
15.2. FIR FILTERING IN THE FREQUENCYDOMAIN 577
Figure 15.3: Overlap add method of filtering in the frequency domain. The input signal
x,, is divided into blocks of length L, to which are added M - 1 zeros to fill a buffer of
length N = L + M - 1. This buffer is converted to the frequency domain and multiplied
there by N frequency domain filter values. The result is converted back into the time
domain, M - 1 partial values at the beginning of the buffer are overlapped and then added
to the M - 1 last values from the previous buffer.
578 DIGITAL FILTER IMPLEMENTATION
niques this delay is composed of two parts. First we have to fill up the signal
buffer (and true gains in efficiency require the use of large buffers), resulting
in bufer delay, and then we have to perform the entire computation (FFT,
block multiplication, iFFT), resulting in algorithmic delay. Only after all
this computation is completed can we start to output the yn. While the
input sample that corresponds to the last value in a buffer suffers only the
algorithmic delay, the first sample suffers the sum of both delays. For appli-
cations with strict limitations on the allowed delay, we must use techniques
where the computation is spread evenly over time, even if they require more
computation overall.
EXERCISES
15.2.1 Explain why circular convolution requires specification of the buffer size while
linear convolution doesnt. Explain why linear convolution can be considered
circular convolution with an infinite buffer.
15.2.2 The circular convolution yc = aeze + alzl, yi = aizo + aczl implies four
multiplications and two additions. Show that it can be computed with two
multiplications and four additions by precomputing Go = 3 (a0 + ai), G1 =
$<a0- al), and for each 20, ~1 computing zo = ~0 + ~1 and ~1 = ~0 - ~1.
15.2.3 Convince yourself that overlap save and overlap add really work by coding
routines for straightforward linear convolution, for OA and for OS. Run all
three and compare the output signals.
15.2.4 Do you expect OA/OS to be more or lessnumerically accurate than straight-
forward convolution in the time domain?
15.2.5 Compare the number of operations per time required for filtering an infinite
signal by a filter of length M, using straightforward time domain convolution
with that using the FFT. What length FFT is best? When is the FFT method
worthwhile?
15.2.6 One can compute circular convolution using an algorithm designed for linear
convolution, by replicating parts of the signal. By copying the L - 2 last
values before ~0 (the cyclic prefix) and the L - 2 first values after ZN- 1 (the
cyclic sufix), we obtain a signal that looks like this.
070, xN-L+l,XN-L+2,*.*XN-2,XN-l,
x0,x1, *. * XN-2,xN-1,
Figure 15.4: Direct form implementation of the FIR filter. This form used to be known
as the tapped delay line, as it is a direct implementation of the weighted sum of delayed
taps of the input signal.
Figure 15.5: Thnsposed form implementation of the FIR filter. Here the present input
zn is multiplied simultaneously by all L filter coefficients, and the intermediate products
are delayed and summed.
580 DIGITAL FILTER IMPLEMENTATION
W
x = - a,b = = E . c,& + y = x.-b--p&q-WY
L
Figure 15.6: Cascading simple filters. On the left the output y is created by filtering WI,
itself the output of filtering x. On the right is the equivalent single filter system.
Substituting, we see that the two in series are equivalent to a single filter
that depends on the present and two past inputs.
15.3. FIR STRUCTURES 581
Wn = (u+bz-l)x,
Yn = (c+dz-)wn
and the resultant filter is given by the product.
Yn = (c + dz-)(a + bz-) xn
= UC+ (ad + bc)z- + bdze2) xn
(
= A + Bz- + CzB2 xn
( >
We see that the A, B, C parameters derived here by formal multiplication
of polynomials in z-l are exactly those derived above by substitution of the
intermediate variable wn. It is suggested that the reader experiment with
more complex subfilters and become convinced that this is always the case.
Not only is the multiplication of polynomials simpler than the substitu-
tion, the zT formalism has further benefits as well. For example, it is hard
to see from the substitution method that the subfilters commute, that is,
had we cascaded
vn = cxn + dxn-1 c+d=l
Vn = awn + bwn-1 u+b=l
582 DIGITAL FILTER IMPLEMENTATION
Figure 15.7: Cascade form implementation of the FIR filter. Here the input is filtered
successively by M second-order sections, that is, simple FIR filters that depend on the
present input and two past inputs. The term second-order refers to the highest power of
z - being two, and section is synonymous with what we have been calling subfilter. If
C m = 0 the section is first order.
we would have obtained the same filter. However, this is immediately obvious
in the zT formalism, from the commutativity of multiplication of polynomi-
als.
(c + cEz-l)(a + bz-l) = (a + bz-)(c + dz-l)
Even more importantly, in the zT formalism it is clear that arbitrary filters
can be decomposed into cascades of simple subfilters, called sections, by
factoring the polynomial in zT. The fundamental theorem of algebra (see
Appendix A.6) guarantees that all polynomials can be factored into linear
factors (or linear and quadratic if we use only real arithmetic); so any filter
can be decomposed into cascades of first-order and second-order sections.
...
...
Figure 15.8: Lattice form implementation of the FIR filter. Here the input is filtered
successivelyby M lattice stages, every two of which is equivalent to a direct form second-
order section.
Yn = Xn + hlxn-1 + h2xn-2
A single lattice stage has only a single free parameter, so well need two
stages to emulate the second-order section. Following the graphic imple-
mentation for two stages we find
and comparing this with the previous expression leads to the connection
between the two sets of coefficients (assuming h2 # -1).
h = kl(l+ka) kl = &
h2 = k2 k2 = h2
EXERCISES
15.3.1 Consider the L-tap FIR filter hu = 1, hi = X, h2 = X2,. , . hL-1 = XL-l.
Graph the direct form implementation. How many delays and how many
MACS are required? Find an equivalent filter that utilizes feedback. How
many delays and arithmetic operations are required now?
15.3.2 Why did we discuss series connection of simple FIR filter sections but not
parallel connection?
584 DIGITAL FILTER IMPLEMENTATION
15.3.3 We saw in Section 7.2 that FIR filters are linear-phase if they are either
symmetric h-, = h, or antisymmetric h-, = -h,. Devise a graphic imple-
mentation that exploits these symmetries. What can be done if there are an
even number of coefficients (half sample delay)? What are the advantages of
such a implementation? What are the disadvantages?
15.3.4 Obtain a routine for factoring polynomials (these are often called polynomial
root finding routines) and write a program that decomposes a general FIR
filter specified by its impulse response h, into first- and second-ordersections.
Write a program to filter arbitrary inputs using the direct and cascade forms
and compare the numeric results.
Of course
Actually, just as bad since we have been neglecting aliasing. The original
signal x can have energy up to $, while the new signal y must not have
appreciable energy higher than A. In order to eliminate the illegal compo-
nents we are required to low-pass filter the original signal before decimating.
For definiteness assume once again that we wish to decimate by 4, and to
use a causal FIR antialiasing filter h of length 16. Then
wo = hoxo + hlxel + h2xv2 + h3xe3 + . . . + h15xs15
Wl = hoxl + hlxo + f-w-1 + h3x-2 + . . . + h15x-14
w2 = km + hm + h2xo + hzx-1 + . . . + h15x-13 (15.8)
w3 = hox3 + hm + h2xl + h3xo + + hisx-12
l . l
In both cases we should compute only a single row of the above equations
during each time interval, thus evenly distributing the computation over the
M time intervals.
Now we come to a subtle point. In a real-time system the input signal
x, will be placed into a buffer E. In order to conserve memory this buffer
will usually be taken to be of length L, the length of the low-pass filter. The
convolution is performed between two buffers of length L, the input buffer
and the filter coefficient table; the coefficient table is constant, but a new
input xn is appended to the input buffer every sampling time.
In the above equations for computing wa the subscripts of xn are absolute
time indices; lets try to rephrase them using input buffer indices instead.
We immediately run into a problem with the partitioned form. The input
values in the last row are no longer available by the time we get around to
wanting them. But this obstacle is easily avoided by reversing the order.
With the understanding that the input buffer updates from row to row, and
using a rather uncommon indexing notation for the input buffer, we can now
rewrite the partitioned computation as
decimate @I
decimate /$I * = (
decimate =
Figure 15.9: The polyphase decimation filter. We depict the decimation of an input signal
xn by a factor of four, using a polyphase filter. Each decimator extracts only inputs with
index divisible by 4, so that the combination of delays and decimators results in all the
possible decimation phases. hIk] for k = 0, 1,2,3 are the subfilters; h[l = (ho, hl , hz, ha),
h[l = (hd, hg, he, h7), etc.
Once again this is a polyphase filter, with the input fixed but the subfil-
ters being changed; but this time the absolute time indices of the signal
are fixed, not the buffer-relative ones! Moreover, we do not need to add the
subfilter outputs; rather each contributes a different output phase. In actual
implementations we simply interleave these outputs to obtain the desired
15.4. POLYPHASEFILTERS 589
Figure 15.10: The polyphase interpolation filter. We depict the interpolation of an input
signal xn by a factor of four, using a polyphase filter. Each subfilter operates on the same
inputs but with different subfilters, and the outputs are interleaved by zero insertion and
delay.
EXERCISES
15.4.1 A commutator is a diagrammatic element that chooses between M inputs
1 . . . M in order. Draw diagrams of the polyphase decimator and interpolator
using the commutator.
15.4.2 Both 32 KHz and 48 KHz are common sampling frequencies for music, while
CDs uses the unusual sampling frequency of 44.1 KHz. How can we convert
between all these rates?
15.4.3 The simple decimator that extracts inputs with index divisible by M is not
a time-invariant system, but rather periodically time varying. Is the entire
decimation system of Figure 15.9 time-invariant?
15.4.4 Can the polyphase technique be used for IIR filters?
15.4.5 When the decimation or interpolation factor M is large, it may be worthwhile
to carry out the filtering in stages. For example, assume M = Ml M2, and that
we decimate by Ml and then by M2. Explain how to specify filter responses.
15.4.6 A half-band filter is a filter whose frequency response obeys the symmetry
H(w) = 1 - H(w,id - w) around the middle of the band wrnid = 4. For
every low-pass half-band filter there is a high-pass half-band filter called its
mirror filter. Explain how mirror half-band filters can be used to efficiently
compute a bank of filters with 2m bands.
where x,,, is the maximal absolute value the input signal takes. In order
to ensure that y, never overflows in an accumulator of b bits, we need to
ensure that the maximal x value does not exceed the following bound.
2b
(15.9)
xmax Cl lhll
This worst-case analysis of the possibility of overflow is often too ex-
treme. The input scaling implied for even modest filter lengths would so
drastically reduce the SNR that we are usually willing to risk possible but
improbable overflows. Such riskier scaling methods are obtained by replacing
the sum of absolute values in equation (15.9) with different combinations of
the hl coefficients. One commonly used criterion is
592 DIGITAL FILTER IMPLEMENTATION
which results from requiring the output energy to be sufficiently low; another
is
2b
Xmax
Hmax
where Hmax is the maximum value of the filters frequency response, re-
sulting from requiring that the output doesnt overflow in the frequency
domain.
When a result overflow does occur, its effect is hardware dependent.
Standard computers usually set an overflow flag to announce that the re-
sult is meaningless, and return the meaningless least significant bits. Thus
the product of two positive numbers may be negative and the product of
two large numbers may be small. Many DSP processors have a saturation
arithmetic mode, where calculations that overflow return the largest avail-
able number of the appropriate sign. Although noise is still added in such
cases, its effect is much less drastic. However, saturation introduces clipping
nonlinearity, which can give rise to harmonic distortion.
Even when no overflow takes place, digital filters (especially IIR filters)
may act quite differently from their analog counterparts. As an example,
take the simple AR filter
Yn = Xn - 0.9yn-1 (15.10)
0.6
0.4
0.2
a.2
0.4
Figure 15.11: The behavior of a simple AR filter using fixed point arithmetic. The
decaying plot depicts the desired behavior, while the second plot is the behavior that
results from rounding to a single digit after the decimal point.
the quantization of the filter coefficients causes the poles to stray from their
original positions, and in particular a pole may wander outside the unit
circle. Once excited, such a pole causes oscillating outputs even when the
input vanishes.
This idea leads us to investigate the effect of coefficient quantization on
the position of the filters poles and zeros, and hence on its transfer function.
Lets express the transfer function
-1
H( ) z-z AW1> cko a1 2 l-g& - Cd
z (15.11)
B(z-l) 1 - CEcl bmrm = nf$,(~ - n,)
and consider the effect of quantizing the bm coefficients on the pole positions
n,. The quantization introduces round-off error, so that the effective coeffi-
cient is bm+Sbm, and assuming that this round-off error is small, its effect on
the position of pole k may be approximated by the first-order contributions.
(15.12)
In particular, if the original filter has poles that are close together (i.e.,
for which rrl, - 7~j is small), small coefficient round-off errors can cause signif-
icant movement of these poles. Since close poles are a common occurrence,
straightforward implementation of IIR filters as difference equations often
lead to instability when fixed point arithmetic is employed. The most com-
mon solution to this problem is to implement IIR filters as cascades of sub-
filters with poles as far apart as possible. Since each subfilter is separately
computed, the round-off errors cannot directly interact, and pole movement
can be minimized. Carrying this idea to the extreme we can implement IIR
filters as cascades of second-order sections, each with a single pair of conju-
gate poles and a single pair of conjugate zeros (if there are real poles or zeros
we use first-order structures). In order to minimize strong gains that may
cause overflow we strive to group together zeros and poles that are as close
together as possible. This still leaves considerable freedom in the placement
order of the sections. Empirically, it seems that the best strategy is to order
sections monotonically in the radius of their poles, either from smallest to
largest (those nearest the unit circle) or vice versa. The reasoning is not
hard to follow. Assume there are poles with very small radius. We wouldnt
want to place them first since this would reduce the number of effective
bits in the signal early on in the processing, leading to enhanced round-off
error. Ordering the poles in a sequence with progressively decreasing radius
ameliorates this problem. When there are poles very close to the unit circle
placing them first would increase the chance of overflow, or require reducing
the dynamic range in order to avoid overflow. Ordering the poles in a se-
quence with progressively increasing radius is best in this case. When there
are both small and large poles it is hard to know which way is better, and
it is prudent to directly compare the two alternative orders. Filter design
programs that include fixed point optimization routines take such pairing
and ordering considerations into account.
EXERCISES
15.5.1 A pair of conjugate poles with radius r < 1 and angles 33 contribute a
second-order section
15.5.2 As we discussed in Section 14.6, fixed point FFTs are vulnerable to numer-
ical problems as well. Compare the accuracy and overflow characteristics of
frequency domain and time domain filtering.
15.5.3 Develop a strategy to eliminate limit cycles, taking into account that limit
cycles can be caused by round-off or overflow errors.
15.5.4 Complete the derivation of the dependenceof nk on &,.
15.5.5 What can you say about the dependence of zero position cl on small changes
in numerator coefficients al ? Why do you think fixed point FIR filters are so
often computed in direct form rather than cascade form?
15.5.6 We saw that it is possible to prescale the input in order to ensure that an
FIR filter will never overflow. Is it possible to guarantee that an IIR filter
will not overflow?
15.5.7 In the text we saw a system whose impulse response should have decayed to
zero, but due to quantization was a a-cycle. Find a system whose impulse
response is a nonzero constant. Find a system with a 4-cycle. Find a system
that goes into oscillation because of overflow.
$1 = $1 $1 = #
...
l-biK -1~,-l~b[K-lz-2
2
Figure 15.12: General cascade implementation of an IIR filter. Each section implements
an independent (first- or) second-order section symbolized by the transfer function ap-
pearing in the rectangle. Note that a zero in any of these subfilters results in a zero of the
filter as a whole.
Figure 15.13: Direct form implementation of a second-order IIR section. This structure
is derived by placing the MA (all-zero) filter before the AR (all-pole) one.
15.6. IIR STRUCTURES 597
Figure 15.14: Canonical form implementation of a second-order IIR section. This struc-
ture is derived by placing the AR (all-pole) filter before the MA (all-zero) one and com-
bining common elements. (Why didnt we draw a filled circle for w!!,?)
Figure 15.15: Transposed form implementation of a second-order IIR section. Here only
the intermediate variables are delayed. Although only three adders are shown the center
one has three inputs, and so there are actually four additions.
598 DIGITAL FILTER IMPLEMENTATION
In real-time applications the loop over time will normally be an infinite loop.
Each new input sample is first MA filtered to give the intermediate signal
w!]
w!l = #xn + a1PIx+1+ upxn-~
and then this signal is AR filtered to give the sections output
Xi] t y!.y
and the process repeats until all K stages are completed. The output of the
final stage is the desired result.
Yn = YnW-11
Each direct form stage requires five multiplications, four additions, and four
delays. In the diagrams we have emphasized memory locations that have to
be stored (static memory) by a circle. Note that wn is generated each time
and does not need to be stored, so that there are only two saved memory
locations.
As we saw in Section 12.3 we can reverse the order of the MA and AR
portions of the second-order section, and then regroup to save memory loca-
tions. This results in the structure known as canonical (meaning accepted
or simplest) form, an appellation well deserved because of its use of the least
number of delay elements. While the direct form requires delayed versions
of both xn and yn, the canonical form only requires storage of wn.
15.6. IIR STRUCTURES 599
and once again we can either stored negative b coefficients or perform sub-
traction(s). Each canonical form stage requires five multiplications, four ad-
ditions, two delays, and two intermediate memory locations.
The transposed form is so designated because it can be derived from the
canonical form using the transposition theorem, which states that reversing
all the arc directions, changing adders to tee connections and vice-versa,
and interchanging the input and output does not alter the systems transfer
function. It is also canonical in the sense that it also uses only two delays,
but we need to save a single value of two different signals (which we call
u, and vn), rather than two lags of a single intermediate signal. The full
computation is
loop over time n
3$ + z
loop onnsection number k t 0 to K - 1
?$I + apzi$ _ @&+I
Un + U~Ll$ - Jpy2l + p1 1
n-
yn t y!f-l
Dont be fooled by Figure 15.15 into thinking that there are only three ad-
ditions in the transposed form. The center adder is a three-input adder,
which has to be implemented as two separate additions. Hence the trans-
posed form requires five multiplications, four additions, two delays, and two
intermediate memory locations, just like the canonical form.
The cascade forms we have just studied are numerically superior to direct
implementation of the difference equation, especially when pole-zero pairing
and ordering are properly carried out. However, the very fact that the signal
has to travel through section after section in series means that round-off
600 DIGITAL FILTER IMPLEMENTATION
aP1+,Iol,-1+,Iol,-2
Xn =
1-bpz--b$-2
a111+,[11,-1+,Pl,-2
; Yn
l-b+-I--b$-2
a[K-I+,[K-11Z-1+,(K-11,-2
l-b~K-]z-l-b~K-],-2
2
Figure 15.16: Parallel form implementation of the IIR filter. In this form the subfilters
are placed in parallel, and so round-off errors do not accumulate. Note that a pole in any
of these subfilters results in a pole of the filter as a whole.
(15.13)
with all coefficients real. If there are more zeros than poles in the system
function, we need an additional FIR filter in parallel with the ARMA sec-
tions.
The decomposition is performed in practice by factoring the denominator
of the system function into real first- and second-order factors, writing the
partial fraction expansion, and comparing. For example, assume that the
system function is
1+ az-l + bz-2
H(z) =
(1 + cz-l)(l + dz-l + ezv2)
then we write
A B + Cz-l
H(z) = +
1 + cz-l 1 + dz-l + eze2
= (A + B) + (Ad + Bc + C)z- + (Ae + Cc)ze2
Cl+ cz-l>( 1 + dz-l + ez-2)
and compare. This results in three equations for the three variables A, B,
and C.
EXERCISES
15.6.1 An arbitrary IIR filter can always be factored into cascaded first-order sec-
tions, if we allow complex-valued coefficients. Compare real-valued second-
order sections with complex-valued first-order sections from the points of
view of computational complexity and numerical stability.
15.6.2 A second-order all-pass filter section has the following transfer function.
c + dz-l + z-~
1 + dz-l + cz-2
Diagram it in direct form. How many multiplications are needed? Redraw
the section emphasizing this.
15.6.3 Apply the transposition theorem to the direct form to derive a noncanonical
transposed section.
15.6.4 The lattice structure presented for the FIR filter in Section 15.3 can be
used for IIR filters as well. Diagram a two-pole AR filter. How can lattice
techniques be used for ARMA filters?
602 DIGITAL FILTER IMPLEMENTATION
Assuming that both FIR and IIR filters are still in the running (e.g.,
only the amplitude of the frequency response is of interest), the issue of
computational complexity is usually the next to be considered. IIR filters
with a relatively small number of coefficients can be designed to have very
sharp frequency response transitions (with the phase being extremely non-
linear near these transitions) and very strong stop-band attenuation, For a
given specification elliptical IIR filters will usually have dramatically lower
computational complexity than FIR filters, with the computational require-
ments ratio sometimes in the thousands. Only if the filters are relatively
mild and when a large amount of pass-band ripple can be tolerated will the
computational requirements be similar or even in favor of the FIR. Cheby-
shev IIR filters are less efficient than elliptical designs but still usually better
performers than FIR filters. Butterworth designs are the least flexible and
hence require the highest order and the highest computational effort. If phase
linearity compensation is attempted for a Butterworth IIR filter the total
computational effort may be comparable to that of an FIR filter.
The next consideration is often numerical accuracy. It is relatively simple
to determine the worst-case number of bits required for overflow-free FIR
computation, and if sufficient bits are available in the accumulator and the
quantized coefficients optimized, the round-off error will be small. Of course
long filters and small registers will force us to prescale down filter coefficients
or input signals causing 6 dB of SNR degradation for each lost bit. For IIR
filters determining the required number of bits is much more complex, de-
pending on the filter characteristics and input signal frequency components.
FIR filters are inherently stable, while IIR filters may be unstable or may
become unstable due to numerical problems. This is of overriding impor-
tance for filters that must be varied as time goes on; an IIR filter must be
continuously monitored for stability (possibly a computationally intensive
task in itself) while FIR filters may be used with impunity.
Finally, all things being equal, personal taste and experience comes into
play. Each DSP professional accumulates over time a bag of fully honed and
well-oiled tools. It is perfectly legitimate that the particular tool that feels
right to one practitioner may not even be considered by another. The main
problem is that when you have only a hammer every problem looks like a
nail. We thus advise that you work on as many different applications as
possible, collecting a tool or two from each.
604 BIBLIOGRAPHICAL NOTES
Bibliographical Notes
Most general DSP texts, e.g., [186, 2001and Chapters 6 and 7 of [241], cover dig-
ital filter structures to some degree. Also valuable are libraries and manuals that
accompany specific DSP processors.
The idea of using the DFT to compute linear convolutions appears to have been
invented simultaneously at MIT [124], at Bell Labs [loo] and by Sandeat Princeton.
Polyphase filtering was developed extensively at Bell Labs, and a good review
of polyphase filters for interpolation and decimation is (481.
The effect of numerical error on filters has an extensive bibliography, e.g., [151,
215, 114, 1151.
16
605
606 FUNCTION EVALUATION ALGORITHMS
The digital frequency of the sine wave, f / fs( is required to have resolution
&, which means that the physical frequency is quantized to f = gfs. Thus
the functions to be calculated are all of the following form:
13 15 17
sin(x) = x - 3x + 3x - ;iix + .a. (16.1)
. . .
l2 l4 6
cos(x) = 1- TX + TX - 3 + ***
. . .
which converge rather slowly. For any given place of truncation, we can im-
prove the approximation (that is, reduce the error made) by slightly chang-
ing the coefficients of the expansion. Tables of such corrected coefficients are
available in the literature. There are also techniques for actually speeding up
the convergence of these polynomial expansions, as well as alternative ratio-
nal approximations. These approximations tend to be difficult to implement
on DSP processors, although (using Horners rule) polynomial calculation
can be pipelined on MAC machines.
For the special case (prevalent in DSP) of equally spaced samples of a
sinusoidal oscillator of fixed frequency, several other techniques are possi-
ble. One technique that we studied in Section 6.11 exploits the fact that
sinusoidal oscillations are solutions of second-order differential or difference
equations, and thus a new sine value may be calculated recursively based on
two previous values. Thus one need only precompute two initial values and
thereafter churn out sine values. The problem with any recursive method of
this sort is error accumulation. Our computations only have finite accuracy,
and with time the computation error builds up. This error accumulation
16.1. SINE AND COSINE GENERATION 607
EXERCISES
16.1.1 Evaluate equation (16.2), successively generating further sine and cosine val-
ues (use single precision). Compare these values with those returned by the
built-in functions. What happens to the error?
16.1.2 Try to find limitations or problems with the trigonometric functions as sup-
plied by your compilers library. Can you guess what algorithm is used?
16.1.3 The simple cubic polynomial
approximates sin(s) to within 2% over the range [-i , . . $1. What are the
advantages and disadvantages of using this approximation? How can you
bring the error down to less than l%?
16.1.4 Code the three-table sine and cosine algorithm in your favorite programming
language. Preprepare the required tables. Test your code by generating the
sine and cosine for all whole-degree values from 0 to 360 and comparing with
your library routines.
16.1.5 The signal supplied to a signal processing system turns out to be inverted in
spectrum (that is, f ---) fS - f) due to an analog mixer. You are very much
worried since you have practically no spare processing power, but suddenly
realize the inversion can be carried out with practically no computation. How
do you do it?
16.1.6 You are given the task of designing a mixer-filter, a device that band-pass
filters a narrow bandwidth signal and at the same time translates it from
one frequency to another. You must take undesired mixer by-products into
account, and should not require designing a filter in real-time. Code your
mixer filter using the three-table sine and cosine algorithm. Generate a sig-
nal composed of a small number of sines, mix it using the mixer filter, and
perform an FFT on the result. Did you get what you expect?
16.2. ARCTANGENT 609
16.2 Arctangent
The floating point arctangent is often required in DSP calculations. Most
often this is in the context of a rectangular to polar coordinate transform&
tion, in which case the CORDIC-based algorithm given in Section 16.5 is
usually preferable. For other cases simple approximations may be of use.
First one can always reduce the argument range to 0 5 x 5 1, by ex-
ploiting the antisymmetry of the function for negative arguments, and the
symmetry
1
tail-1(x) = f - tan-l a;
0
for x > 1.
For arguments in this range, we can approximate by using the Taylor
expansion around zero.
tan-yx) = x - ix3 + 6x5 - 3x7 + 0 l l
(16.3)
As for the sine and cosine functions equations (16.1), the approximation can
be improved by slightly changing the coefficients.
EXERCISES
16.2.1 Code the arctangent approximation of equation (16.3), summing up N terms.
What is the maximum error as a function of N?
16.2.2 How can improved approximation coefficients be found?
16.2.3 Look up the improved coefficients for expansion up to fifth order. How much
better is the improved formula than the straight Taylor expansion? Plot the
two approximations and compare their global behavior.
16.2.4 For positive 2 there is an alternative expansion:
x- 1
tan-l(z) = % + sly + a3y3 + a5y5 + . . . where y E -
x+1
Find the coefficients and compare the accuracy with that of equation (16.3).
16.2.5 Make a phase detector, i.e., a program that inputs a complex exponential
Sn = xn + iy, = A&(wn+dn), computes, and outputs its instantaneous phase
$71= tan-l(yn, xn) - wn using one of the arctangent approximations and
correcting for the four-quadrant arctangent. How can you find w? Is the
phase always accurately recovered?
610 FUNCTION EVALUATION ALGORITHMS
16.3 Logarithm
This function is required mainly for logarithmic AM detection, conversion
of power ratios and power spectra to decibels, as well as for various musical
effects, such as compression of guitar sounds. The ear responds to both
sound intensities and frequencies in approximately logarithmic fashion, and
so logarithmic transformations are used extensively in many perception-
based feature extraction methods. Considerable effort has also been devoted
to the efficient computation of the natural and decimal logarithms in the
non-DSP world.
Due to its compressive nature, the magnitude of the output of the log
operation is significantly less than that of the input (for large enough inputs).
Thus, relatively large changes in input value may lead to little or no change
in the output. This has persuaded many practitioners to use overly simplistic
approximations, which may lead to overall system precision degradation.
We can concentrate on base-two logarithms without limiting generality
since logarithms of all other bases are simply related.
If only a single bit of a numbers binary representation is set, say the kth
one, then the log is simple to calculate-it is simply k. Otherwise the bits
following the most significant set bit k contribute a fractional part
k k k
X = c xi p = 2 + c x&i 2k-i = 2 1+ c xk-@ = 2 (1 + 2)
i=o i=l i=l
EXERCISES
16.3.1 Code the linear interpolation approximation mentioned above and compare
its output with your library routine. Where is the maximum error and how
much is it?
16.3.2 Use a higher-order approximation (check a good mathematical handbook for
the coefficients) and observe the effect on the error.
16.3.3 Before the advent of electronic calculators, scientists and engineersused slide
rules in order to multiply quickly. How does a slide rule work? What is the
principle behind the circular slide rule? How doesthis relate to the algorithm
discussedabove?
j/z = 22 * x = $log&)
x: $ y M abmax(z, y) + Ic abmin(z, y)
with abmax (abmin) returning the argument with larger (smaller) absolute
value. This approximation is good when 0.25 5 Ic 5 0.31, with Ic = 0.267304
giving exact mean and Ic = 0.300585 minimum variance.
The straightforward method of calculating z @ y requires two multipli-
cations, an addition, and a square root. Even if a square root instruction is
available, one may not want to use this procedure since the squaring oper-
ations may underflow or overflow even when the inputs and output are well
within the range of the DSPs floating point word.
Several techniques have been suggested, the simplest perhaps being that
of Moler and Morrison. In this algorithm x and y are altered by transforma-
tions that keep x $ y invariant while increasing x and decreasing y. When
negligible, x contains the desired output.
In pseudocode form:
P + m=44 IYI)
Q+ min(l47Ivl>
while q> 0
r + (g)2
P
--&
p + p+2*s*p
Q+ S-P
output p
EXERCISES
16.4.1 Practice finding square roots in your head using Newton-Raphson.
16.4.2 Code Moler and Morrisons algorithm for the Pythagorean sum. How many
iterations does it require to obtain a given accuracy?
16.4.3 Devise examples where straightforward evaluation of the Pythagorean sum
overflows. Now find cases where underflow occurs. Test Moler and Morrisons
algorithm on these cases.
16.5. CORDIC ALGORITHMS 613
(16.4)
614 FUNCTION EVALUATION ALGORITHMS
If one knows numerically the R matrix for some angle, the desired functions
are easily obtained by rotating the unit vector along the x direction.
(16.5)
However, how can we obtain the rotation matrix without knowing the values
of sin(e) and cos(e)? We can exploit the sum rule for rotation matrices:
= fiR(oI) (16.6)
i=o
and SO for 8 = Cr=, ai, using equation (16.4), we find:
and the matrix products can be performed using only right shifts. We can
easily generalize this result to angles 8 that can be written as sums of ai =
Z/Ztan-1(2-i). Due to the symmetry cos(-a) = cos(o), the product of cosines
is unchanged, and the M matrices are either the same as those given above,
or have the signs reversed. In either case the products can be performed
by shifts and possibly sign reversals. Now for the surprise-one can show
that any angle 6 inside a certain region of convergence can be expressed
as an infinite sum of &cui = & tan- (2-i)! The region of convergence turns
out to be 0 5 8 5 1.7433 radians M 99.9, conveniently containing the first
quadrant. Thus for any angle 8 in the first quadrant, we can calculate sin(@)
and cos(8) in the following fashion. First we express 8 as the appropriate
sum of ai. We then calculate the product of M matrices using only shift
operations. Next we multiply the product matrix by the universal constant
K E HE0 cos(cq) z 0.607. Finally, we multiply this matrix by the unit
16.5. CORDIC ALGORITHMS 615
x+-K
Yto
xte
for i t 0 to b-l
s + w44
x t x-s.y.2-i
y + y+s*x*2-i
x + x - s tan-1(2-i)
l
cos(8) t x
sin(B) + y
error + 2
Of course only additions, subtractions, and right shifts are utilized, and the
b values tan-l (2-i) are precomputed and stored in a table. Beware that in
the loop the two values x and y are to be calculated simultaneously. Thus
to code this in a high-level language place the snippet
for i + 0 to b- 1
s +x
x t <-s.y+
y +-- y+s+2-i
1 iLO
atani t
rr ilO
tpi +-
12-i i 2 0
{ t4an-1(2-i) i >, 0
16.5. CORDIC ALGORITHMS 617
and K +- q II!=, cos ( tan-(2-i)) and, for example, the backward rota-
tion is now carried out by the following algorithm:
x+X
Y+--Y
n-0
for it-2tob-1
s +- W(Y)
x + x+s*y*tpi
Y+Y - s x tpi
z + x+seatq
j/m +- K*x
error + y
tan-l(Y/X) t 2
Up to now we have dealt only with circular functions. The basic CORDIC
iteration can be generalized to
(16.8)
%+1 = Zi + Siti
where for the circular functions m = +l and ti = tan-1(2-i), for the hyper-
bolic functions m = -1 and ti = tanh-1(2-i), and for the linear functions
m= 0 and ti = 2-i. For the circular and hyperbolic cases one must also
renormalize by the constants K = I/ nyZo dp. For the hyperbolic
case additional iterations are always required.
EXERCISES
16.5.1 Code the forward and backward extended-range CORDIC algorithms. Test
them by comparison with library routines on randomly selected problems.
16.5.2 Recode the mixer filter from the exercisesof Section 16.1 using CORDIC to
generate the complex exponential.
16.5.3 Code a digital receiver that inputs a complex signal s(t) = A(t)ei(t+4(t)),
mixes the signal down to zero frequency s(t) = A(t)e@(Q (using forward
CORDIC), and then extracts both the amplitude and phase (using backward
CORDIC).
618 BIBLIOGRAPHICAL NOTES
Bibliographical Notes
The reader is referred to the mathematical handbook of Abramowitz and Stegun [l]
for properties of functions, and polynomial and rational approximation coefficients.
For a basic introduction to numerical techniques I recommend [216].
Techniques for speeding up the convergence of polynomial and rational expan-
sions are discussed in [1381.
Generation of sinusoids by recursively evaluating a second-order difference equa-
tion is discussed in [58].
Mitchell [120] proposed simple linear interpolation for the evaluation of loga-
rithms, while Marino [158] proposed the quadratic approximation.
Knuths METRFONT typeface design program (which generates the fonts usually
used with Tj$ and IX&$) is an example of a language that has @ as a primitive. Its
manual and entire source code are available in book form [134]. The abmax-abmin
formula for $ was apparently first discussed in [222] but later covered in many
sources, e.g., [184]. The Moler and Morrison algorithm was first present&d in (1751
and was developed for software that evolved into the present MATLAB[go].
The CORDIC algorithm was proposed by Volder [266] in 1959, and refined
mathematically by Walther [269]. Its use in the first full-function scientific calculator
(the HP-35) is documented in [38]. CORDICs approximation error is analyzed
in [107]. Extending CORDIC to a full four-quadrant technique was proposed by
[105], while its use for computation of the inverse trigonometric functions is in
[162]. CORDIC-like architectures can aid in the computation of the FFT [51, 521,
eigenvalues and singular values [60], and many other DSP tasks [106].
Digital Signal Processors
Until now we have assumed that all the computation necessary for DSP ap-
plications could be performed either using pencil and paper or by a general-
purpose computer. Obviously, those that can be handled by human calcu-
lation are either very simplistic or at least very low rate. It might surprise
the uninitiated that general-purpose computers suffer from the same limita-
tions. Being general-purpose, a conventional central processing unit (CPU)
is not optimized for DSP-style number crunching, since much of its time
is devoted to branching, disk access, string manipulation, etc. In addition,
even if a computer is fast enough to perform all the required computation
in time, it may not be able to guarantee doing so.
In the late 197Os, special-purpose processors optimized for DSP appli-
cations were first developed, and such processors are still multiplying to-
day (pun definitely intended). Although correctly termed Digital Signal
Processors, we will somewhat redundantly call them DSP processors, or
simply DSPs. There are small, low-power, inexpensive, relatively weak DSPs
targeted at mass-produced consumer goods such as toys and cars. More capa-
ble fixed point processors are required for cellular phones, digital answering
machines, and modems. The strongest, often floating point, DSPs are used
for image and video processing, and server applications.
DSP processors are characterized by having at least some of the fol-
lowing special features: DSP-specific instructions (most notably the MAC),
special address registers, zero-overhead loops, multiple memory buses and
banks, instruction pipelines, fast interrupt servicing (fast context switch),
specialized ports for input and output, and special addressing modes (e.g.,
bit reversal).
There are also many non-DSP processors of interest to the DSP imple-
mentor. There are convolution processors and FFT processors devoted to
these tasks alone. There are systolic arrays, vector and superscalar proces-
sors, RISC processors for embedded applications, general-purpose processors
with multimedia extensions, CORDIC processors, and many more varieties.
619
620 DIGITAL SIGNAL PROCESSORS
DSP cores are available that can be integrated on a single chip with other
elements such as CPUs, communications processors, and IO devices. Al-
though beyond the scope of our present treatment the reader would be well
advised to learn the basic principles of these alternative architectures.
In this chapter we will study the DSP processor and how it is optimized
for DSP applications. We will discuss general principles, without considering
any specific DSP processor, family of processors, or manufacturer. The first
subject is the MAC operation, and how DSPs can perform it in a single
clock cycle. In order to understand this feat we need to study memory ar-
chitectures and pipelines. We then consider interrupts, ports, and the issue
of numerical representation. Finally, we present a simple, yet typical exam-
ple of a DSP program. The last two sections deal with the practicalities of
industrial DSP programming.
loop
update j , update Ic
a + U-l-XjYk
for i t 1 to N
statements
the calculation the loop index register must be incremented, and checked for
termination. Of course if there are not enough registers the loop index must
be retrieved from memory, incremented and checked, and then stored back
to memory. Except for the last iteration, a branch or jump instruction
must be performed to return execution to the top of the loop.
DSP processors provide a zero-overhead hardware mechanism (often
called repeat or do) that can repeat an instruction or number of instructions
a prespecified number of times. Due to hardware support for this repeat in-
struction no clocks are wasted on branching or incrementing and checking
the loop index. The maximum number of iterations is always limited (64K is
common, although some processors have low limits such as 128) and many
processors limit the number of instructions in the loop (1, 16), but these
limitations fall into the envelope of common DSP operations. Some proces-
sors allow loop nesting (since the FFT requires 3 loops, this is a common
limit), while for others only the innermost loop can be zero overhead.
Now lets concentrate on the computations inside the loop. How would
a general-purpose CPU carry out the desired computation? We assume that
x and y are stored as arrays in memory, so that xj is stored j locations after
x0, and similarly for yk. Furthermore, we assume that the CPU has at least
two pointer registers (that we call j and k) that can be directly updated
(incremented or decremented) and used to retrieve data from memory. Fi-
nally, we assume the CPU has at least two arithmetic (floating point or
fixed point) registers (x and y) that can be used as operands of arithmetic
operations, a double-length register (z) that can receive a product, and an
accumulator (a) for summing up values.
Assuming that the loop has been set up (i.e., the counter loaded, the
base pointers for Xj and yk set, and the automatic updating of these pointers
programmed in), the sequence of operations for computation of the contents
of the loop on a general-purpose CPU will look something like this.
update pointer to Xj
update pointer to yk
load Zj into register x
load ok into register y
fetch operation (multiply)
decode operation (multiply)
multiply x by y storing the result in register z
fetch operation (add)
decode operat i on (add)
add register z to accumulat or a
622 DIGITAL SIGNAL PROCESSORS
We see that even assuming each of the above lines takes the same amount
of time (which is dubious for the multiplication), the computation requires
about 10 instruction times to complete. Of course different CPUs will have
slightly different instruction sets and complements of registers, but similar
principles hold for all CPUs.
A major distinction between a general-purpose CPU and a DSP is that
the latter can perform a MAC in a single instruction time. Indeed this feature
is of such importance that many use it as the definition of a DSP. The
main purpose of this chapter is explain how this miracle is accomplished.
In particular it is not enough to simply add a MAC instruction to the set
of opcodes; such an MAC-augmented CPU would still have to perform the
following steps
update pointer to zj
update pointer to yk
load z~j into register x
load yk into register y
fetch operation (MAC)
decode operation (MAC)
MAC a +- x * y
for a total of seven instruction times. We have managed to save a few clocks
but are still far from our goal. Were the simple addition of a MAC instruction
all a DSP processor had to offer, it would probably not be worth devoting
precious silicon real-estate to the special MAC hardware. In order to build
a DSP we need more imagination than this.
The first step in building a true DSP is to note that the pointers to xj and
$fk are independent and thus their updating can be performed in parallel. To
implement this we need new hardware; we need to add two address updating
units to the hardware complement of our hypothetical DSP processor. Using
the symbol 11to signify two operations that are performed in parallel, the
MAC now looks like this:
We have obviously saved at least the time of one instruction, since the xj and
yk pointers are now updated simultaneously, but even though we no longer
17.2. MEMORY ARCHITECTURE 623
require use of the CPUs own adder it does not seem possible to further
exploit this in order to reduce overall execution time. It is obvious that we
cannot proceed to load values into the x and y registers until the pointers
are ready, and we cannot perform the MAC until the registers are loaded.
The next steps in optimizing our DSP call for more radical change.
EXERCISES
17.1.1 For the CPU it would be clearer to have j and k stored in fixed point registers
and to retrieve zz;j by adding j to the address of ~0. Why didnt we do this?
17.1.2 Explain in more detail why it is difficult for two buses to access the same
memory circuits.
17.1.3 Many DSP processors have on-chip ROM or RAM memory. Why?
17.1.4 Many CPU architectures use memory caching to keep critical data quickly
accessible. Discuss the advantages and disadvantages for DSP processors.
17.1.5 A processor used in personal computers has a set of instructions widely ad-
vertised as being designed for multimedia applications. What instructions are
included in this set? Can this processor be considered a DSP?
17.1.6 Why does the zero-overhead loop only support loops with a prespecified
number of iterations (for loops)? What about while (condition) loops?
The solution here is to leave the usual model of a single linear memory,
and to define multiple memory banks. Different buses service different mem-
ory banks, and placing the zj and yk arrays in separate banks allows their
simultaneous transfer to the appropriate registers. The existence of more
than one memory area for data is a radical departure from the memory
architecture of a standard CPU.
The next step in improving our DSP is to take care of the fetch and
decode steps. Before explaining how to economize on these instructions we
should first explain more fully what these steps do. In modern CPUs and
DSPs instructions are stored sequentially in memory as opcodes, which are
binary entities that uniquely define the operation the processor is to perform.
These opcodes typically contain a group of bits that define the operation
itself (e.g., multiply or branch), individual bit parameters that modify the
meaning of the instruction (multiply immediate or branch relative), and
possibly bits representing numeric fields (multiply immediate by 2 or branch
relative forward by 2). Before the requested function can be performed these
opcodes must first be retrieved from memory and decoded, operations that
typically take a clock cycle each.
We see that a nonnegligible portion of the time it takes to execute an
instruction is actually devoted to retrieving and decoding it. In order to
reduce the time spent on each instruction we must find a way of reducing this
overhead. Standard CPUs use program caches for this purpose. A program
cache is high speed memory inside the CPU into which program instructions
are automatically placed. When a program instruction is required that has
already been fetched and decoded, it can be taken from the program cache
rather than refetched and redecoded. This tends to significantly speed up
the execution of loops. Program caches are typically rather small and can
only remember the last few instructions; so loops containing a large number
of instructions may not benefit from this tactic. Similarly CPUs may have
data caches where the last few memory locations referenced are mirrored,
and redundant data loads avoided.
Caches are usually avoided in DSPs because caching complicates the
calculation of the time required for a program to execute. In a CPU with
17.2. MEMORY ARCHITECTURE 625
EXERCISES
17.2.1 A pure Harvard architecture does not allow any direct connection between
program and data memories, while the modified Harvard architecture contains
copy commands between the memories. Why are these commands useful?
Does the existence of these commands have any drawbacks?
17.2.2 DSPs often have many different types of memory, including ROM, on-chip
RAM, several banks of data RAM, and program memory. Explain the func-
tion of each of these and demonstrate how these would be used in a real-time
FIR filter program.
17.2.3 FIR and IIR filters require a fast MAC instruction, while the FFT needs the
butterfly
X +- x+wy
Y t X-WY
where x and y are complex numbers and W a complex root of unity. Should
we add the butterfly as a basic operation similar to the MAC?
17.2.4 There are two styles of DSP assembly language syntax. The opcode-mnemonic
style usescommands such as MPY A0 , Al, A2, while the programming style
looks more like a conventional high-level language A0 = Al * A2. Research
how the MAC instruction with parallel retrieval and address update is coded
in both these styles. Which notation is better? Take into account both algo-
rithmic transparency and the need to assist the programmer in understanding
the hardware and its limitations.
17.3. PIPELINES 627
17.3 Pipelines
In the previous sections we saw that the secret to a DSP processors speed
is not only special instructions, but the exploitation of parallelism. Address
registers are updated in parallel, memory retrievals are performed in parallel,
and program instructions are fetched in parallel with execution of previous
instructions. The natural extension is to allow parallel execution of any
operations that logically can be performed in parallel.
Figure 17.1: The pipelining of a MAC calculation. Time runs from left to right, while
height corresponds to distinct hardware units, update meaning the updating of the xj
and yk pointers, load the loading into x and y, and MAC the actual computation. At
the left there are three cycles during which the pipeline is filling, while at the right there
are a further three cycles while the pipeline is emptying. The result is available seven
cycles after the first update.
The three steps of the three-clock MAC we obtained in the previous sec-
tion use different processor capabilities, and so should be allowed to operate
simultaneously. The problem is the dependence of each step on the comple-
tion of the previous one, but this can be sidestepped by using a pipeline to
overlap these operations. The operation of the pipeline is clarified in Fig-
ure 17.1. In this figure update 1 refers to the first updating of the pointers
to zj and yk; load 1 to the first loading of Xj and yk into registers x and y;
and MAC 1 means the first multiplication. As can be seen, the first load
takes place only after the first update is complete, and the MAC only after
the loads. However, we do not wait for the MAC to complete before updat-
ing the pointers; rather we immediately start the second update after the
first pointers are handed over to the loading process. Similarly, the second
load takes place in parallel with the first MAC, so that the second MAC
can commence as soon as the first is completed. In this way the MACs are
performed one after the other without waiting, and once the pipeline is filled
each MAC requires only one instruction cycle. Of course there is overhead
due to the pipeline having to fill up at the beginning of the process and
empty out at the end, but for large enough loops this overhead is negligible.
Thus the pipeline allows a DSP to perform one MAC per instruction clock
on the average.
628 DIGITAL SIGNAL PROCESSORS
We see that a total of four clock cycles is required for this single addition,
and our instruction time is actually four clock cycles. There might be
additional subinstruction operations as well, for instance, transfer of a value
from the register to memory. Fixed point DSP processors may include an op-
tional postarithmetic scaling (shift) operation, while for floating point there
is usually a postarithmetic normalization stage that ensures the number is
properly represented.
Using a subinstruction pipeline we neednt count four clock cycles per
instruction. While we are performing the arithmetic portion of an instruc-
tion, we can already be decoding the next instruction, and fetching the one
after that! The number of overlapable operations of which an instruction is
comprised is known as the depth of the pipeline. The minimum depth is three
(fetch, decode, execute), typical values are four or five, but by dividing the
arithmetic operation into stages the maximum depth may be larger. Recent
DSP processors have pipeline depths as high as 11.
17.3. PIPELINES 629
Figure 17.2: The operation of a depth-four pipeline. Time runs from left to right, while
height corresponds to distinct hardware units. At the left there are three cycles during
which the pipeline is filling, while at the right there are three cycles while the pipeline is
emptying. The complete sum is available eight cycles after the first fetch.
EXERCISES
17.3.1 Why do many processors limit the number of instructions in a repeat loop?
17.3.2 What happens to the pipeline at the end of a loop? When a branch is taken?
17.3.3 There are two styles of DSP assembly language syntax regarding the pipeline.
One emphasizes time by listing on one line all operations to be carried out
simultaneously, while the other stresses data that is logical related. Consider
a statement of the first type
where Al, A2, A3, A4 are accumulators and Rl, R2 pointer registers. Explain
the relationship between the contents of the indicated registers. Next consider
a statement of the second type
AO=AO+(*Rl++**R2++)
and explain when the operations are carried out.
17.3.4 It is often said that when the pipeline is not kept filled, a DSP is slower than
a conventional processor, due to having to fill up and empty out the pipeline.
Is this a fair statement?
17.3.5 Your DSP processor has 8 registers RI, R2, R3, R4, R5, R6, R7, R8, and
the following operations
l load register from memory: Rn + location
0 store register to memory: location t Rn
l single cycle no operation: NOP
0 negate: Rn +- - Rn [l cycle latency]
l add: Rn +- Ra + Rb [2 cycle latency]
l subtract: Rn +-- Ra - Rb [2 cycle latency]
l multiply: Rn + Ra . Rb [3 cycle latency]
l MAC: Rn + Rn + Ra . Rb [4 cycle latency]
where the latencies disclose the number of cycles until the result is ready to
be stored to memory. For example,
RI t Rl + R2 . R3
answer + RI
does not have the desired effect of saving the MAC in answer, unless four
NOP operations are interposed. Show how to efficiently multiply two complex
numbers. (Hint: First code operations with enough NOP operations, and then
interchange order to reduce the number of NOPs.)
17.4. INTERRUPTS, PORTS 631
All that is left is to explain how signal values are input to and output
from the DSP. This is done by ports, of which there are several varieties.
Serial ports are typically used for low-rate signals. The input signals bits
are delivered to the DSP one at a time and deposited in an internal shift
register, and outputs are similarly shifted out of the DSP one bit per clock.
Thus when a 16-bit A/D is connected to a serial port it will send the sample
as 16 bits, along with a bit clock signal telling the DSP when each bit is
ready. The bits may be sent MSB first or LSB first depending on the A/D
and DSP involved. These bits are transferred to the DSPs internal serial
port shift register. Each time the A/D signals that a bit is ready, the DSP
serial port shift register shifts over one bit and receives the new one. Once
all 16 bits are input the A/D will assert an interrupt requesting the DSP to
store the sample presently in the shift register to memory.
Parallel ports are faster than serial ports but require more pins on the
DSP chip itself. Parallel ports typically transfer eight or sixteen bits at a
time. In order to further speed up data transfer Direct Memory Access
(DMA) channels are provided that can transfer whole blocks of data to or
from the DSP memory without interfering with the processing. Typically
once a DMA transfer is initiated, only a single interrupt is required at the
end to signal that the transfer is complete.
Finally, communications ports are provided on those DSPs that may be
interconnected with other DSPs. By constructing arrays of DSPs processing
tasks may be divided up between processors and such platforms may attain
processing power far exceeding that available from a single processor.
EXERCISES
17.4.1 How does the CPU know which interrupt handler to call?
17.4.2 Some DSPs have internal peripherals that can generate interrupts. What
can these be used for?
17.4.3 What happens when an interrupt interrupts an interrupt?
17.4.4 When a DSP is on a processing board inside a host computer there may be a
method of input and output other than ports-shared memory. Discuss the
pros and cons of shared memory vs. ports.
17.5. FIXED AND FLOATING POINT 633
When two 16-bit integers are added, the sum can require 17 bits; when
multiplied, the product can require 32 bits. Floating point hardware takes
care of this bit growth by automatically discarding the least significant bits,
but in fixed point arithmetic we must explicitly handle the increase in preci-
sion. CPUs handle addition by assuming that the resultant usually does fit
into 16 bits; if there is an overflow a flag is set or an exception is triggered.
Products are conventionally stored in two registers, and the user must de-
cide what to do next based on the values in the registers. These strategies
are not optimal for DSP since they require extra operations for testing flags
or discarding bits, operations that would break the pipeline.
Fixed point DSPs use one of several strategies for handling the growth of
bits without wasting cycles. The best strategy is for the adder of the MAC
instruction to use an accumulator that is longer than the largest possible
product. For example, if the largest product is 32 bits the accumulator could
have 40 bits, the extra bits allowing eight MACs to be performed without any
possibility of overflow. At the end of the loop a single check and possible
discard can be performed. The second strategy is to provide an optional
scaling operation as part of the MAC instruction itself. This is basically a
right shift of the product before the addition, and is built into the pipeline.
The least satisfactory way out of the problem, but still better than nothing, is
the use of saturation arithmetic. In this case a hard limiter is used whenever
an overflow occurs, the result being replaced by the largest representable
number of the appropriate sign. Although this is definitely incorrect, the
error introduced is smaller than that caused by straight overflow.
Other than these surmountable arithmetic problems, there are other pos-
sible complications that must be taken into account when using a fixed point
processor. As discussed in Section 15.5, after designing a digital filter its co-
efficients should not simply be rounded; rather the best integer coefficients
should be determined using an optimization procedure. Stable IIR filters
may become unstable after quantization, due to poles too close to the unit
circle. Adaptive filters are especially sensitive to quantization. When bits
are discarded, overflows occur, or limiting takes place, the signal processing
system ceases to be linear, and therefore cycles and chaotic behavior become
possible (see Section 5.5).
Floating point DSPs avoid many of the above problems. Floating point
numbers consist of a mantissa and an exponent, both of which are signed
integers. A recognized standard details both sizes for the mantissa and ex-
ponent and rules for the arithmetic, including how exceptions are to be
handled. Not all floating point DSPs conform to this standard, but some
that dont provide opcodes for conversion to the standard format.
17.6. A REAL-TIME FILTER 635
EXERCISES
17.5.1 Real numbers are represented as integers by multiplying by a large number
and rounding. Assuming there is no overflow, how is the integer product
related to the real product? How is a fixed point multiply operation from
two b-bit registers to a b-bit register implemented?
17.5.2 Simulate the simple IIR filter yn = cry,-1 + xn (0 5 o. ,< 1) in floating point
and plot the impulse response for various a. Now repeat the simulation using
8-bit integer arithmetic (1 becomes 256, 0 5 a! 5 256). How do you properly
simulate 8-bit arithmetic on a 32-bit processor?
17.5.3 Design a narrow band-pass FIR filter and plot its empirical frequency re-
sponse. Quantize the coefficients to 16 bits, 8 bits, 4 bits, 2 bits, and finally a
single bit (the coefficients sign); for each case replot the frequency response.
17.5.4 Repeat the previous exercise for an IIR filter.
and output yn in less than t, seconds, before the next sample arrives. This
should take only somewhat more than L processor cycles, the extra cycles
being unavoidable overhead.
636 DIGITAL SIGNAL PROCESSORS
for 1 + 1 to (L-l)
x11-11 + xc11
x[L-11 + input
Y+-O
for 1 t- 0 to (L-1)
y + y + h[l] * x[L-l-l]
output + y
We first made room for the new input and placed it in x [L-l]. We then
computed the convolution and output the result.
There are two main problems with this computation. First we wasted
a lot of time in moving the static data in order to make room for the new
input. We neednt physically move data if we use a circular bufler, but then
the indexation in the convolution loop would be more complex. Second, the
use of explicit indexation is wasteful. Each time we have need x [L-l-l]
we have to compute L-l-l, find the memory location, and finally retrieve
the desired data. A similar set of operations has to be performed for h Cl]
before we are at last ready to multiply, A more efficient implementation
uses pointers; assuming we initialize h and x to point to ho and ~-1-l
respectively, we have the following simpler loop:
Y+-O
repeat L times
y t y + (*h) * C*(x)
h+h+l
x+x-l
Here *h means the contents of the memory location to which the pointer
h points. We can further improve this a little by initializing y to horn and
performing one less pass through the loop.
How much time does this CPU-based program take? In the loop there is
one multiplication, two additions and one subtraction, in addition to assign-
ment statements; and the loop itself requires an additional implicit decre-
ment and comparison operation.
Now we are ready to try doing the same filtering operation on a DSP.
Figure 17.3 is a program in assembly language of an imaginary DSP. The
words starting with dots (such as . table) are directives; they direct the
assembler to place the following data or code in specific memory banks. In
17.6. A REAL-TIME FILTER 637
. table
H: ho
h
HLAST: hLsl
. data
x: (L-1) * 0
XNEW: 0
. program
START:
if (WAIT) goto START
*XNEW + INPUT
h + HLAST
x+x
NOP
OUTPUT t y
goto START
this case the L filter coefficients ho . . . hLsl are placed in table memory; the
static buffer of length L is initialized to all zeros and placed in data memory;
and the code resides in program memory. These placements ensure that the
MAC instructions will be executable in a single cycle. The names followed
by colons (such as HLAST: ) are labels, and are used to reference specific
memory lo cat ions.
The filter coefficients are stored in the following order ho, hl, . . . hL-1
with ho bearing the label H and hL-1 labeled HLAST. The static buffer is in
reversed order z,+-11, . . . xn-.r, x 72with the oldest value bearing the label
X and the present input labeled XNEW.
The program code starts with the label START, and each nonempty line
thereafter corresponds to a single processor cycle. The first line causes the
638 DIGITAL SIGNAL PROCESSORS
processor to loop endlessly until a new input arrives. In a real program such
a tight loop would usually be avoided, but slightly looser do-nothing loops
are commonly used.
Once an input is ready it is immediately copied into the location pointed
to by XNEW,which is the end of the static buffer. Then pointer register h is
set to point to the end of the filter buffer (hi-1) and pointer x is set to point
to the beginning of that buffer (the oldest stored input).
Accumulator y is initialized to h~-r~a, the last term in the convolution.
Note that y is a numeric value, not a pointer like x. The 11notation refers
to operations that are performed in parallel. In this line the filter buffer
pointer is decremented and the static buffer pointer is incremented. These
operations are carried out before they are next required.
The next line contains a zero-overhead loop. This loop is only executed
L-l times, since the last term of the convolution is already in the accu-
mulator. The last iteration multiplies the he coefficient by the new input.
However, something else is happening here as well. The * (x-l) + *x being
executed in parallel is a data-move that shifts the input data that has just
been used one place down; by the time the entire loop has been executed
the static buffer has all been shifted and is ready for the next iteration.
Once the entire convolution has been carried out we are ready to output
the result. However, in some DSP processors this output operation can only
take place once the pipeline has been emptied; for this reason we placed
a NOP (no-operation) command before copying the accumulator into the
output register. Finally, we jump back to the start of the program and wait
for the next input to arrive.
EXERCISES
17.6.1 Taking a specific number of coefficients (e.g., L=5), walk through the program
in Figure 17.3, noting at each line the values of the pointers, the state of the
static buffer, and the algebraic value in the accumulator.
17.6.2 Code a real-time FIR filter for a DSP that does not support data-move in
parallel with the MAC, but has hardware support for a circular buffer.
17.6.3 Code a real-time IIR routine similar to the FIR one given in the text. The
filter should be a cascade of N second order sections, and the main loop
should contain four lines and be executed N times.
17.6.4 Write a filtering program for a real DSP and run it in real-time.
17.7. DSP PROGRAMMING PROJECTS 639
All too often we are driven by the need to make our algorithms faster and
faster, and are tempted to do so at the expense of system stability or thor-
ough testing. These temptations are to be avoided at all costs.
Once this is understood I suggest that the task of implementing a new
system is CHILDs play. Here the word CHILD is a mnemonic for:
Collect requirements and decide on architecture
High-level design
Intermediate level, simulation and porting to platform
Low-level coding and efficiency improvement
Deliver and document
We shall devote a paragraph or two to each of these stages.
The collection stage is a critical one, all too often incompletely executed.
The implementor must collect all the requirements, including the expected
range of inputs, the exact output(s) required, the overall development sched-
ule and budget, the desired end user cost, interface specifications, etc. Some-
times someone else has done the preliminary work for you and you receive a
Hardware Requirements Specification (HRS) and a Software Requirements
Specification (SRS). Remember that anything missed during the collection
stage will be difficult or impossible to reintroduce later on, One of the things
to be decided at this stage is how the final product is to be tested, and the
640 DIGITAL SIGNAL PROCESSORS
exact criteria for success. You should make sure the end users (or techni-
cal marketing personnel) sign off on the requirement specifications and
acceptance procedures.
Between the end of the collection stage and the beginning of the high-
level design stage it is highly recommended to go on vacation.
The technical output of the high-level design stage will usually be a pair
of documents, the Hardware Design Document (HDD) and the Software
Design Document (SDD). There will also be project management litera-
ture, including various charts detailing precisely what each team member
should be doing at every time, dates by which critical tasks should be com-
pleted, milestones, etc. We will focus on the SDD. The SDD explains the
signal processing system, first in generality, and then increasingly in de-
tail. The function of each subsystem is explained and its major algorithms
noted. There are two ways to write an SDD. The first (and most commonly
encountered) is to have done something extremely similar in the past. In
this case one starts by cutting and pasting and then deleting, inserting, and
modifying until the present SRS is met. The more interesting case is when
something truly new is to be built. In this case a correct SDD cannot be
written and the project management literature should be considered science
fiction. Remember that the R and D in R&D are two quite different tasks,
and that a true research task cannot be guaranteed to terminate on a certain
date and in a certain way (the research would be unnecessary if it could).
Often simulations must be performed during the high-level design stage.
For these simulations efficiency is of no concern; but development speed,
ease of use and visualization ability are of the utmost importance. For this
reason special development environments with graphics and possibly visual
programming are commonly used. The output of these simulations, both
block diagrams and performance graphs, can be pasted into the design doc-
uments. The amount of memory and processing power required for each
subsystem can now be better estimated. It is best to plan on using only 50%
to 75% of the available processing power (it will always turn out to require
a lot more than you anticipate).
At the end of the high-level design a Design Review (DR) should be
carried out. Here the HDD and SDD are explained and comments solicited.
Invite as many relevant people as possible to the DR. Remember that mis-
takes in the high-level design are extremely costly to repair later on.
The intermediate-stage may be bypassed only for very small projects.
Here the block diagrams developed in the high-level stage are fleshed out
and a complete program is written. This is often done first in a high-level
language, liberally using floating point numbers and library functions. While
17.8. DSP DEVELOPMENT TEAMS 641
the high-level software design stage is often carried out by a single person,
the intermediate stage is usually handed over to the full development team.
Once the team starts to work, the project should be placed under revision
control. Once completed and integrated the full program can be tested with
test inputs and outputs to ensure passing the final acceptance procedures.
Next the high-level language program is rewritten in a real-time style, using
the proper block lengths, converting to fixed point if required, etc. After
each major step the program behavior can be compared to that of the orig-
inal to ensure correctness. Mechanisms for debugging, exception handling,
maintainability, and extensibility should be built into the code.
The low-level programming commences as a straightforward port of the
intermediate level program to the final platform. Once again the first code
should be written to maintain correctness at the expense of efficiency. After
the first port, decisions must be made as to what can remain in a high-
level language and what must be coded in assembly language; where major
improvements in efficiency are required; where memory usage is excessive;
etc. Efficiency is increased incrementally by concentrating on areas of code
where the program spends most of its time. Various debugging tools, such
as simulators, emulators, debug ports, and real-time monitoring are used.
Eventually a correct version that is fast enough is generated.
Delivery of a version is something that no one likes doing, but the project
is not complete without it. The final version must be cleaned up and accep-
tance tests thoroughly run (preferably with the end user or disinterested
parties present). User and programmer documentation must be completed.
The former is usually written by professional publications personnel. The
latter include internal documentation (the final code should be at least 25%
comments), an updated SDD, and a Version Description Document (VDD)
that describes all limitations, unimplemented features, changes, and out-
standing problems.
After delivery the boss takes the development team out to lunch, or lets
everyone take an extended weekend. The following week the whole process
starts over again.
the TRM will only last about a half hour since the boss has to go to the
TLM (team leader meeting) that has to take place before the GLM (group
leader meeting) headed by the director of ASP. Any questions? Somewhat
timidly you speak up-why is it called the FEU and not the DSP? In all
the jobs you worked on up to now the module with the DSPs was simply
called the DSP. The boss explains patiently (while several coworkers smile)
that the name has been chosen to be accurate from a system point of view.
DSP stands for digital signal processing. Here the fact that the processing
is digital is irrelevant to the rest of the system; indeed in an earlier version
a lot of the processing was analog. The words signal and processing turn
out to be incorrect from a system point of view. The purpose of this unit is
acquisition of the data needed by the rest of the system, even if this data
already requires a great deal of processing from raw input.
After the meeting you spend a few hours in the library. The SDD refer-
ences a lot of international standards and each of these in turn references still
other standards. The standards documents seem even less comprehensible
than the SDD itself. You spend the next few hours by the copying machine.
On the fourth day you are invited to attend a discussion between the
hardware and software guys. Not yet having learned about the subsystem in
question you cant quite make out what is going on, other than the hardware
guys saying its an obvious software bug and the software guys saying its
a hardware failure. You speak up asking why a simple test program cant
be used to test the hardware. The hardware people explain that they had
written such a program and that is precisely how they know its a software
problem. The software people reply that the hardware test program was
unrealistically simplistic and didnt really test this aspect of the design.
They, however, have written a simulation of the hardware, and their software
runs perfectly on it.
You glance at your watch. Although its after six oclock you think youll
spend a few more hours reading the SDD. Just this morning you had at last
found out what the DSP processing elements were, and were beginning to
feel more confident that it was, in principle, possible to extract information
from these documents. How will you ever finish reading all this by tomorrow?
The fifth day starts auspiciously-your development system arrives. You
are given a table in the lab; its one floor down from your cubicle and you learn
to use the stairs rather than wait for the elevator. The entire morning is spent
on unpacking, reading the minimal amount of documentation, hooking up all
the cables, and configuring the software. A coworker helps you out, mostly
in order to see the improvements in the new version you have received. You
go out to lunch with your co-workers, and the entire time is spent talking
644 BIBLIOGRAPHICAL NOTES
about work. Once back you return to your cubicle only to find that someone
has borrowed your copy of the SDD. You return to the lab only to find
that several power cables are missing as well. Its after three and you start
to panic. Are your coworkers playing some kind of initiation prank or does
this kind of thing happen all the time?
Bibliographical Notes
Jonathan Allen from MIT gave two early, but still relevant, overviews of the basic
architecture of digital signal processors [3, 41. More modern reviews are [142, 143,
1411. In particular [61] can be used as a crash course in DSP processors: what defines
a DSP, how DSPs differ from CPUs, and how they differ one from another.
Manuals supplied by the various processor manufacturers are the best source for
information on DSP architecture and how to best exploit it. Usually each processor
has a Processor Users Guide that details its architecture and instruction set; an
Assembly Language Reference with explanations of its programming environment;
and an Applications Library Manual with sample code and library routines for
FFTs, FIR and IIR filters, etc. For popular processors many useful DSP functions
and full applications will be available for licensing or in the public domain. The
annual EDN DSP directory [148] is a treasure-trove of information regarding all
extant DSPs.
[140] is devoted entirely to fundamentals of DSP processors, and its authors also
publish an in-depth study and comparison of available DSPs. Readers considering
implementing DSP functions in VLSI should consult [154].
Part IV
Applications
18
In this chapter we will survey various topics in signal processing for com-
munications. Communications, like signal processing itself, is commonly di-
vided into analog and digital varieties. Analog communications consist of
techniques for transmitting and receiving speech, music or images as analog
signals, as in telephones, broadcast radio and television. Digital communi-
cations are methods of transferring digital information, usually in the form
of bit streams. Digital communications are often between computers, or be-
tween human and computer, although increasingly digital communications
are being used between people as well (email). Both analog and digital sig-
nal processing may be used for various portions of both analog and digital
communications systems.
A device that takes an analog input signal and creates an analog com-
munications signal is called a transmitter, while a receiver inputs an analog
communications signal and attempts to recover, as accurately as possible,
the original analog message signal. A device that takes a digital input and
creates a digital communications signal is usually called a modulator, while a
demodulator inputs a digital communications signal and attempts to recover,
with as few bit errors as possible, the original digital message. Transmitters
and receivers are sometimes packaged together and called transceivers; for
digital communications it is almost universal to package the modulator and
demodulator together, and to call the combined device a modem.
Digital communications systems include such diverse objects as fax ma-
chines, telephone-grade modems, local area networks, wide area networks,
private digital telephone exchanges, communications satellites and their
ground stations, the public switched telephone network (yes, it too has be-
come digital), and the Internet. Although the history of data communica-
tions is relatively short, the present scope of its theory and application is
huge, and we will have to stringently restrict the scope of our treatment.
647
648 COMMUNICATIONS SIGNAL PROCESSING
Table 18.1: The Morse code. Every letter, number, or punctuation mark is assigned a
unique combination of dots and dashes.
Telegraphy using Morse code still had a few disadvantages. It was rel-
atively slow and error prone. It required skilled telegraphers at both ends
and could not be directly used by individuals. Unless special codes were
employed the messages could be read by others, and it was difficult to au-
thenticate the senders identity. For some time people strived to mechanize
the transfer of text using the Morse code, but this was a difficult task due to
the variable-length characters. In 1875, Emile Baudot from France created
a new code, one optimized for mechanized text transfer. In the Baudot code
each letter took five equal time units, where each unit could be current flow
(ma&) or lack thereof (space). Actual commercial exploitation of this code
began in early twentieth century, under the trademark name teletype.
A further breakthrough was announced within a year of Baudots code
when, on March 10, 1876, Dr. Alexander Graham Bell in Boston and Elisha
Gray in Chicago both filed for patents for a new invention, later to be called
the telephone. Like the telegraph it used voltage signals traveling over a wire,
but rather than being simple on-off, these signals carried a voice. Eventually,
Dr. Bell won the protracted legal battle that reached the level of the US.
Supreme Court. The telephone could be placed in every home, and used by
anyone without the need for intervention of skilled middlemen. For the first
time,point-to-point communication was direct, reliable, relatively private,
and the voice of the person at the other end could be recognized.
Another development was born out of a purely mathematical insight.
In 1865, James Clerk Maxwell wrote down differential equations describing
all that was then known about electricity and magnetism. These equations
described how an electric charge created an electric field (Coulombs law),
18.1. HISTORY OF COMMUNICATIONS 651
how an electric current created a magnetic field (Amperes law), and how
a changing magnetic field created an electric field. Far away from currents
and charges the equations
VE = 0 V-B = 0
Ox-ii - = 1 aB
-;Jj- Vii3 = 0
-
VE = 0 VB = 0
1 i3B
vxk -
= -;-J-f VxB
-
= +g
a phenomenon that had not previously been observed. These new equations
admitted a new type of solution, a changing electric field inducing a changing
magnetic field reinforcing the original changing electric field. This electro-
magnetic field could travel at the speed of light (not surprising since light is
exactly such a field) and carry a signal far away without the need for wires.
In 1887, Hertz performed an experiment to test Maxwells purely theoretical
prediction. He made sparks jump between two polished brass knobs sepa-
rated by a small gap, and detected the transmitted electromagnetic waves
using a simple receiver of looped wire and similar knobs several meters away.
Radio waves can carry Morse or Baudot code by transmitting or not
transmitting (on-off keying). They can also carry voice by continuously
changing some characteristic of the field, such as its amplitude (AM) or
frequency (FM). In the next section we will learn how this can be done.
EXERCISES
18.1.1 Compute the time durations of the 26 letters in Morse code. What is the
average duration assuming all characters are equally probable? What is the
average duration assuming that the letter probabilities are roughly E:12%,
TAOINS:8%, HRDLU:4%, MCFGPB:2%, and all the rest 1%. Is Morse
code better or worse than Baudot code for actual text?
18.1.2 Write a program that inputs a text file and outputs Morse code. You will
need a computer with minimal sound capabilities. Whenever s = 1 play a
tone (1000 Hz is good). Make the speed an adjustable parameter, specified in
words per minute (figure an averageword as 5 characters). Add an option to
your program to output two different tones, a high-frequency tone for s = 1
and a low-frequency one for s = 0.
652 COMMUNICATIONS SIGNAL PROCESSING
18.1.3 Modify the above program to output a file with sampled signal values (use
a sampling rate of 8000 Hz and a tone of 750Hz). Now write a program that
inputs this file and decodesMorse code (converts signal values back to text).
Improve your program to take into account small amounts of noise and small
variabilities in speed (and add these features to the generating program). Do
you think you could write a program to read Morse code sent by hand on a
noisy channel?
Definition: modulation
Modulation is the exploitation of any observable characteristic of a signal
to carry information. The signal whose characteristics are varied is called
a carrier. We modulate the carrier by the information signal in order to
create the modulated signal, and demodulate the modulated signal in order
to recover the information signal. The systems that perform these functions
are called the modulator and demodulator, respectively. n
For example, the very existence of the carrier can be used to send Morse or
Baudot code. This is called On-Off Keying (OOK) and mathematically is
represented by
SooK = A(t) cos(2d t) (18.2)
18.2. ANALOG MODULATION TYPES 653
where A(t) takes the values zero or one, fc is the carrier frequency, and (with-
out limiting generality) we choose the phase to be zero. In order to carry
voice or other acoustic modulating signals w(t), we need more freedom. Now
equation (18.2) is strongly reminiscent of the instantaneous representation
of a signal of equation (4.66); but there the amplitude A(t) was a continu-
ously varying function. This leads us to the idea of conveying a continuously
varying analog signal v(t) by varying the carriers amplitude
= Ao (1 + mAMv(t)) co@f,t)
SAM@>
where we assume Iv(t) 1 < 1. This modulation technique, known as Amplitude
Modulation (AM), is depicted in Figure 18.1. The coefficient 0 < ?nAM 5 1
is known as the modulation index, and is often specified as a percentage.
Amplitude is not the only signal characteristic that one can modulate.
The sinusoidal carrier of equation (18.1) has two more characteristics that
may be varied, the frequency f and the phase 4. Morse- or Baudot-encoded
text may be sent by Frequency Shift Keying (FSK), that is, by jumping
between two frequencies.
Here it is f(t) that can take on two different values. The third alternative is
called Phase Shift Keying (PSK),
EXERCISES
18.2.1 Why is equation (18.3) not simplyAov(t) cos(27rf,t)? Plot sinusoidally mod-
ulated AM signals for various values of modulation index. What index do
you think should be used?
18.2.2 Write a program that generates an AM-modulated wave. (For concreteness
you may assume a sampling frequency of 2.048 MHz, a carrier of 455 KHz,
and take the modulating signal to be a sinusoid of frequency 5 KHz.) Plot 1
millisecond of signal. What does the spectrum look like?
18.2.3 Why do we prefer sinusoidal carriers to other waveforms (e.g., square waves)?
18.2.4 Can we simultaneously modulate with AM and FM? AM and PM? FM and
PM?
18.3 AM
Now that we know what modulation is, we can commence a more systematic
study of modulated signals and the signal processing systems used to mod-
ulate and demodulate. For now we are only interested in modulating with
continuous analog signals such as speech; digital modulation will be treated
later.
How can we create an amplitude modulated signal using analog electron-
ics? The simplest way would be to first create the carrier using an oscillator
set to the desired frequency. Next the output of this oscillator is input to
an amplifier whose gain is varied according to the modulating signal (see
Figure 18.3). Since both oscillators and variable gain amplifiers are stan-
dard electronic devices, building an AM transmitter in analog electronics
oscillator v amplifier
c A
I
microphone I
Figure 18.3: The basic analog AM transmitter built from an oscillator and a variable gain
amplifier. The oscillator has a single parameter fc that is not varied during transmission.
The amplifiers gain parameter A is varied according to the signal. The inverted triangle
at the top right is the conventional graphic representation of an antenna.
656 COMMUNICATIONS SIGNAL PROCESSING
7
synthesizer =
Figure 18.4: The basic hybrid digital-analog AM transmitter. The digital components
operate at an intermediate frequency and at low power. After conversion to the analog
domain the signal is upmixed to the desired carrier frequency and amplified to the required
output power. The synthesizer is a (digital) local oscillator.
18.3. AM 657
Figure 18.5: The basic analog envelope detector for the demodulation of AM signals.
In (A) we see the AM signal to be demodulated. After half wave rectification the signal
depicted in (B) results. Subsequent low-pass filtering removes the RF and leaves (C) the
desired AF to within DC.
the operation of which can be best understood by studying Figure 18.5. Since
the desired signal is the envelope of the received signal, it can be retrieved
from either the top or bottom of Figure 18.5.A by connecting the peaks.
Choosing to use the top half, half wave rectification results in the signal
of Figure 18.5.B. We next low-pass filter this signal in order to remove the
high-frequency RF, leaving only the envelope as in Figure 18.5.C (with a
strong DC component). This filtering is performed by placing the rectified
signal onto a capacitor that charges up to the voltage peaks and slowly
interpolates between them. Finally a DC blocking filter is used to remove
the 1 from 1 + v(t).
Unfortunately, the envelope detector is ill suited to digital implementa-
tion. It assumes fc to be very high compared to fm, otherwise the envelope
will not be well sampled, and thus downmixing to a low IF will decrease its
efficacy. More importantly, in order to actually see the analog signals peaks
in its digital representation, a sampling frequency much higher than Nyquist
is required. Even sampling at several times Nyquist we can not expect most
of the sampling instants to fall close enough to the peaks.
A better way of digitally performing AM demodulation is to use the
instantaneous representation of Section 4.12. There are two closely related
ways of doing this. The first is to apply the Hilbert transform to the IF signal
658 COMMUNICATIONS SIGNAL PROCESSING
and to obtain the instantaneous amplitude by the square root of the sum
of the squares. The second involves a complex downmix to zero including a
complex low-pass filter to remove everything except the frequency compo-
nents from zero to BW. We can then proceed to obtain the instantaneous
amplitude as before. These methods of digital AM demodulation do not
require high fC and function with sampling frequencies close to Nyquist.
Up to now we have been thinking of AM only in the time domain. What
does the spectrum of an AM signal look like? Well first consider modulating
with a single sinusoid, so that equation (18.3) becomes
where wm and wCare the modulating and carrier angular frequencies. A little
algebra proves
so that the spectrum contains three discrete lines, one corresponding to the
original carrier frequency, and two lines at the carrier plus and minus the
modulation frequency (Figure 18.6.A).
What if we modulate the carrier not with a single sinusoid but with
a general signal w(t)? The modulating signal can be Fourier analyzed into
a collection of sinusoids each of which causes two lines spaced fm away
from the carrier. We thus obtain a carrier and two sidebands as depicted in
Figure 18.6.B. The two sidebands are inverted in frequency with respect to
each other but contain precisely the same information.
-L
A
/ !A
Figure 18.6: The generation of sidebands of an AM signal. In (A) we modulate a sinusoid
of frequency fc by a single sinusoid of frequency f7n to obtain an AM signal with three
frequency lines, fc , fc f fm. In (B) we modulate a sinusoid by a signal with an entire
spectrum of frequencies, conventionally depicted as a triangle. We obtain the carrier and
two sidebands.
18.4. FM AND PM 659
EXERCISES
18.3.1 Our basic analog AM receiver assumed that only a single signal is received
at the antenna, while in fact many signals are received simultaneously. One
method of isolating the signal of interest uses a band-pass filter centered at fc ;
the more conventional method uses a mixer and a band-pass filter centered at
an intermediate frequency (IF). Diagram the two methods and discuss their
advantages and disadvantages.
18.3.2 Diagram an entire AM receiver including antenna, local oscillator and mixer,
IF filter, a half wave rectifier, a low-pass filter, DC blocking filter, and speaker.
Show representative signals at the output of each block.
18.3.3 Implement a digital envelope detector. Create a sinusoidally modulated sig-
nal with fc = 50, fm = 2, and sampling frequency fs = 500. Compare the
demodulated signal with the correct modulating signal. Now decrease fs to
200. Finally decrease fc to 10. What do you conclude?
18.3.4 Show that half of the energy of an AM signal with index of modulation
?nAM = 1 is in the carrier and one-quarter is in each of the sidebands.
18.3.5 Double sideband (DSB) is a more energy-efficient variant of AM, whereby
the carrier is removed and only the two sidebands are transmitted. Diagram
a transmitter and receiver for DSB.
18.3.6 Single sideband (SSB) is the most efficient variant of AM, whereby only a
single sideband is transmitted. Diagram a transmitter and receiver for SSB.
18.3.7 Can AM demodulation be performed by a filter? If yes, what is its frequency
response? If not, what portion of the analog and digital detectors is not a
filter?
18.4 FM and PM
You might expect that frequency modulation of a carrier A cos(w,t) with a
signal w(t) would be accomplished by
where rn is the index of modulation. Indeed the amplitude is constant and the
frequency varies around the carrier frequency according to the modulating
signal; yet this is not the way FM is defined. To see why not, assume that
660 COMMUNICATIONS SIGNAL PROCESSING
Figure 18.7: Frequency modulation according to the naive equation (18.8) has obvious
artifacts. True frequency modulation should look sinusoidal.
the modulating signal v(t) is a sinusoid (lets use sine rather than cosine this
time) of frequency wm.
For a modulating signal that consists of a single sinusoid, the entire difference
between PM and FM is a phase shift and a change in the modulation index;
for a more general modulating signal, FM and PM are less compatible. The
integral of v(t) = sin(w,t) is -& cos(wt) , and so high-frequency Fourier
components of v(t) are much weaker in FM than in PM, a phenomenon
known as de-emphasis. A PM signal heard on an FM receiver has too much
treble and sounds tinny, while using a receiver designed for PM to intercept
an FM signal produces a bassy sound. FM may be generated using a PM
transmitter, if pre-emphasis is performed on the modulating audio in order
to compensate for the later loss of high frequencies.
The PM/FM transmitter is very similar to the AM one, with the ex-
ception that the amplified microphone voltage is used to vary the phase
rather than the amplitude of the carrier; but how do we make an analog
FM receiver? One way is to use frequency-to-voltage conversion to convert
the received FM signal into an AM one. An FM discriminator is a circuit
with gain that varies linearly with frequency, and can thus be used for the
frequency-to-voltage conversion.
The digital FM receiver can derive the instantaneous frequency from the
instantaneous phase through differentiation. Were we to drastically over-
sample we could get by with the simple difference, since
For more rapidly varying phases we must use a true differentiation filter (see
Section 7.3).
The instantaneous phase signal is bounded in the interval between -r
and 7r (or perhaps [0 . . .27r]) and has discontinuities when it crosses these
boundaries. These phase jumps have no physical meaning, they are simply
artifacts of the nonuniqueness of inverse trigonometric functions. Differenti-
ation of such discontinuities would give rise to tremendous unphysical spikes
in the frequency demodulation. Hence we must first unwrap the phase be-
fore differentiation. This can be done by setting a phase change threshold,
and adding ~t27r whenever the phase jumps by more than this threshold.
For oversampled signals this threshold can be relatively small, but close to
Nyquist it must be carefully chosen in order to avoid unwrapping legitimate
changes in phase.
The unwrapped phase signal resulting from the above operation is con-
siderably smoother than the original phase. If, however, the signal has not
been correctly mixed down to zero frequency, the residual carrier frequency
causes linear phase increase or decrease, which will eventually cause the
phase to overflow. In sophisticated implementations one models this phase
change by linear regression and corrects the mixer frequency accordingly.
A simpler technique to avoid phase overflow is not to correct the phase at
all, only the phase diference. Differentiation of the phase difference signal
gives the frequency difference, and the actual frequency is found by adding
the frequency difference to the previous frequency. This frequency is in the
vicinity of the residual carrier frequency, and thus never overflows.
An alternative method of phase differentiation is called the dual differ-
entiator method. It exploits the fact that the specific differentiation to be
performed is
2!@(t)
= -&an- (g) = tix-ky (18.13)
A2(t>
2(t) + y2(t) is the amplitude detection. If we are interested
in the frequency alone, we can limit the input signal (giving a constant
amplitude) and then the above is directly proportional to the instantaneous
frequency, If the amplitude is to be calculated in any event, it should be
done first, and then a division carried out.
We turn now to the spectrum of PM and FM signals, wondering whether
there are sidebands here as there were in the AM case. Even if there are
sidebands, they must be much different than those we saw for AM. For
example, assume the power of the modulating signal increases. For AM the
carrier remains unchanged and the sideband energy increases; for PM/FM
18.4. FM AND PM 663
the total power must remain unchanged (otherwise there would be unwanted
AM!) and thus an increase in sideband power must result in a decrease in
carrier power. At some point the carrier will even have to entirely disappear!
Using the same type of algebra that led to equation (18.7) we find
Plugging these in, and using the trigonometric product identities (A.32)
multiple times, we obtain the desired spectral representation.
This is quite different from equation (18.7) with its sidebands at wC& +w,!
Here we have an infinite number of sidebands at w, f kw, with amplitudes
varying according to the Bessel functions. The carrier amplitude is propor-
tional to JO and thus starts at unity for zero modulation index and decreases
as m increases. All the sidebands start at zero amplitude for m = 0 and at
first increase, but later oscillate. Of course, for constant modulation index
m, the amplitude of the sidelobes tends to decrease with distance from the
carrier. As a rough estimate we can say that &(m) is close to zero for n > m,
so that the number of significant sidebands is 2n and the bandwidth is given
by BW z 2nw,.
664 COMMUNICATIONS SIGNAL PROCESSING
EXERCISES
18.4.1 Prove that equation (18.9) has extrema other than those of the carrier by
differentiating and setting equal to zero.
18.4.2 Diagram an analog transmitter and receiver for FM.
18.4.3 Find the spectral representation of the PM signal.
18.4.4 AM reception suffers from noise more than FM does, for the simple reason
that additive wideband noise directly changes the received signals amplitude,
while most noise does not masquerade as frequency or phase changes. This is
the reason FM is commonly used for high quality music broadcasting. Explain
why FM receivers use a hard-limiter before the demodulator.
18.4.5 Communications-grade FM receivers come equipped with a squelch circuit
that completely silences the receiver when no FM signal is present. Explain
how this works and why such a circuit is not used in AM receivers.
18.4.6 What happens when two AM signals transmit too close together in fre-
quency? What happens with FM?
EXERCISES
18.5.1 Assume that someone uses a dial-up modem to connect to the World Wide
Web. Try to identify as many communications protocols as you can, and at
what OS1 layer(s) they operate. (Hint: The modem has connection, physi-
cal layer transfer and perhaps error correction facilities. The application on
the users computer uses a serial protocol to communicate with the service
provider. The Internet is based on TCP/IP. The web sits above the Internet.)
18.5.2 Do we really need to divide communications systems into layers? If not, what
are there advantages and disadvantages?
&I: Is the x1 = x greater than or equal to 128? Al: Yes (x1 = 137 2 128).
Q2: Isx2=x1- 128 greater than or equal to 64? A2 No (x2 = 9 < 64).
Q3: Is x3 = x2 greater than or equal to 32? As: No (x3 = 9 < 32).
Q4: Is x4 = x3 greater than or equal to 16? Ad: No (x4 = 9 < 16).
Q5: Is x5 = x4 greater than or equal to 8? As: Yes (xg = 9 2 8).
Q6: Is x6 = x5- 8 greater than or equal to 4? As: No (zfj = 1 < 4).
Q7: Is x7 = 26 greater than or equal to 2? A7: No (x7 = 1 < 2).
Qs: Is xp, = x7 equal to l? As: Yes (x7 = 1).
Only the number 137 will give this particular sequence of yes-no an-
swers, and interpreting yes answers as 1 and no answers as 0 produces the
binary representation of x from MSB to LSB. Similarly we can determine
the number of bits of information in arbitrary messages by constructing a
set of yes-no questions that uniquely determines that message.
Lets assume a source wishes to convey to the destination a message
consisting of an integer between 0 and 255. The transmitter neednt wait for
the receiver to ask the questions, since the questioning tactic is known. All
the transmitter needs to do is to transmit the answers Al through Ag.
Signals that carry information appear to be random to some extent. This
is because information is only conveyed by surprising its receiver. Constant
signals, constant amplitude and frequency sinusoids or square waves, convey
no information, since one can predict exactly what the signals value will be
at any time. Yet consider a signal that can take only two values, say s = 0
or s = 1, that can change in value every T seconds, but remains constant
between kT and (k + l)T. Such a signal is often called a Non Return to Zero
(NRZ) signal, for reasons that will become clear shortly. If the signal jumps
in an apparently randomly fashion between its two values, one can interpret
its behavior as a sequence of bits, from which text, sound, or images may
be derived. If one bit is inferred every T seconds, the information transfer
rate is $ bits per second.
According to this point of view, the more random a signal is, the higher
its information transfer rate. Longer T implies a lower information transfer
rate since the signal is predictable for longer times. More complex predictable
behavior also reduces the information transfer rate. For example, a Return
to Zero (RZ) signal (see Figure 18.8.B) is similar to the NRZ signal described
above, but always returns to s = 0 for odd k (we count from k = 0). Since an
unpredictable signal value only appears every 2T seconds, the information
is transferred at half the rate of the NRZ signal. Predictability may be even
more subtle. For example, the Manchester signal used in Ethernet LANs
668 COMMUNICATIONS SIGNAL PROCESSING
--
A
I-T-I I I I I I I
Figure 18.8: Comparison of (A) NRZ, (B) RZ, and (C) Manchester signals. The message
is 11100101 and our channel bandwidth requires transitions to be spaced T seconds apart.
Using NRZ this message requires 8T seconds. RZ and Manchester both require 16T seconds
to transmit the same message.
(see Figure l&&C) encodes a binary one by having s = 1 for even Ic and
s = 0 for the subsequent Ic + 1 interval; a zero is encoded by s = 0 followed
by s = 1. Once again the information transfer rate is only half that of the
NRZ signal, although the lack of randomness is less obvious. Whereas the
NRZ signal has no correlation between signal values spaced T seconds apart,
the Manchester signal never allows odd k intervals to have the same value
as the previous even k interval.
The moral is that any correlation between signal values at different times
reduces the amount of information carried. An infinite amount of informa-
tion is carried by a signal with no correlation between different times (i.e.,
by white noise). Of course a true white noise signal, which has frequency
components up to infinite frequency, cannot pass unaltered through a chan-
nel with finite bandwidth. Thus for a finite bandwidth channel, the signal
with maximal information content is one whose sole predictability is that
caused by the bandwidth constraint. Such a signal has a spectrum that is
flat in the allowed pass-band.
We can similarly define the information transfer rate when the signal
may take on many values (called symbols), not just zero and one. A signal
that can jump randomly every T seconds, but that is a constant s = 0, 1,2,
or 3 in between these jumps, obviously carries 2 bits every T seconds, or $
bits per second.
What if the different symbols are not equally probable? For example, a
signal that takes on 26 values corresponding to a message containing text
18.6. INFORMATION THEORY 669
H(s)
=- (logzp(s))
=- x P(S) log2 P(S) (18.15)
9
where s represents the possible signal values, and the triangular brackets
stand for the expected value (see Appendix A. 13).
To understand this result lets return to the simple case of a sending a
message that consists of a number z between 0 and 255. Before transmission
commences, the receiver has no information as to the value of x other than
the fact that it is between 0 and 255. Thus the receiver assigns an equal
probability of & to each of the integers 0 . . . 255. A priori the transmitter
may send a first symbol of 0 or 1 with probability 3. In the previous example
it would send a 1; immediately the receiver updates its probability estimates,
now 0. . . 127 have zero probability and 128.. .255 have probability 1 in 128.
The receivers uncertainty has been reduced by a factor of two, corresponding
to a single bit of information. Now the second answer (in our example a zero)
is sent. Since the second answer is independent of the first, the probability
of both answers is the product of the individual probabilities i . i = 1.
Similarly, the probability of any particular sequence of three answers is 214.
1 1
z-g= 18. In general it is clear that after each subsequent answer is received
the probability of the message is halved, as is the uncertainty of the receiver.
After eight answers have been received the probability of the message has
been reduced to & and all uncertainty removed.
Now we prefer to think of information as being added after each answer
has been received, although the probabilities were multiplied. The only way
of making an arbitrary multiplication into an addition is to employ a log-
arithmic relation, such as (18.15). If we wish each reduction of probability
by a factor of one half to correspond to the addition of a single bit, the
base of the logarithm must be 2 and a minus sign must be appended (since
- log, $ = + log, 2 = 1). Thus, for our simple example, each answer Ai
contributes
I(Ai) = - log, p(Ai) = log2 $ = 1
bits of information. The information of the sequence of answers is
8 8
I(x) = c I(Ai) = - c log,p(Ai) = 8
i=l i=l
bits, as we claimed.
670 COMMUNICATIONS SIGNAL PROCESSING
EXERCISES
18.6.1 Consider a signal that can take one of two values, s = 0 with probability
p and s = 1 with probability 1 - p. Plot the entropy of a single value as a
function of p. Explain the position and value of the extrema of this graph.
18.6.2 Compute the entropy in bits per character of English text. Use the proba-
bilities from exercise 18.1.1 or collect histograms using some suitably large
on-line text to which you have access. Is a byte required to encode each letter?
18.6.3 Use a file compression program to reduce the size of some English text. What
is the connection between final file size and entropy?
18.6.4 Repeat the previous two exercises for other languages that use the same
alphabet (French, Spanish, Italian, German, etc.). Can these probabilities be
used to discriminate between different languages?
18.6.5 What are the most prevalent pairs of letters in English? How can letter pairs
be used to aid text compression? To aid in language identification?
18.6.6 Using Table 18.1 compute the time durations of Morse code letters and sort
them in increasing order. Did Morse maximize the information transfer rate?
noise
B t
source L channel = digital -r channel = source
SW+- encoder A encoder channel decoder decoder = s(t)
noise
C t
analog line EC e.ource
-decoder- decoder-decoder-)- s @>
channel
Figure 18.9: The conversion of an analog communications system into digital one. In (A)
we see the original analog system. In (B) we have performed the separation into source
and channel coding guaranteed by Shannons theorem. In (C) we add line coding in order
to utilize an analog channel (EC stands for error correction).
send this essentially without error over a digital channel, and then recover
the original signal (Figure 18.9.B).
It should be stressed that Shannons separation of communications into
two parts is fundamentally different from the OS1 separation of communi-
cations into seven layers. There is no theory stating that the division of the
OS1 model does not impair the communications system; the layers are only
separated in order to facilitate human comprehension. In a similar fashion
the channel coding of Shannons theorem is often further divided into two
separate parts, error correction coding and line coding. An error correction
code converts digital data into protected digital data, which can be trans-
mitted over a digital channel with less fear of corruption due to noise. Of
course all real transmission channels are analog, and so digital channels are
actually an abstraction. The conversion of the (protected) digital signal into
an analog one suitable for the physical transmission line is called line coding.
The entire process is thus that of Figure 18.9.C. The division of the channel
code into error correction code and line code is performed solely as an aid to
the designers (its hard to find one person expert in both fields!) but is not
guaranteed to be conserve optimality. Indeed one can increase performance
by combining the two (see Section 18.19).
Shannons theorem, although in many ways satisfying, has not yet con-
vinced us to convert over to digital communications systems. All we have
seen is that we have nothing to lose by converting; we have yet to see that
we have something to gain. Can digital systems actually increase band-
width efficiency, improve the quality, reduce the cost, or provide any other
measurable advantage as compared with analog communications? Shannon
affirmatively answered these questions in a series of theorems about source
and channel coding. Source coding theorems are beyond the scope of our
present treatment, yet we can readily understand how proper source and
channel coding can help us attain some of these goals.
For maximal efficiency source coding should produce a bit stream with
no more bits than absolutely needed. We know that the minimal number of
bits required to encode a message is the information (entropy), and thus the
ideal source coder produces no more bits than entropy requires. For example,
speech can be source encoded into 8 Kb/s or less (see Chapter 19) and there
are modems (line codes) of over 32 Kb/s; hence using digital techniques one
can transfer four conversations over a single telephone line. Thus proper
source encoding can increase bandwidth efficiency.
Digital compact disks have replaced analog long playing records mainly
due to their superior audio quality. This quality is obtained because of the
use of digital error correcting channel codes that guarantee accurate re-
18.7. COMMUNICATIONS THEORY 673
production of the original sound. Analog music signals that have become
contaminated with noise cannot generally be corrected, and the noise man-
ifests itself as various hisses and pops. Thus proper channel encoding can
indeed increase signal quality.
While we will not delve into all of Shannons theorems, there is one
that will be essential for us. Before Shannon, engineers knew that noise and
interference on digital channels cause errors in the reconstructed bit stream;
and they thought that there was only one way of overcoming this problem, by
increasing the power of the communications signal. The principle in which all
designers believed was that no matter what the noise or interference is like,
if we transmit a strong enough signal it will wipe them out. Then there was
the separate issue of bandwidth; the higher the bandwidth the more data
one could reliably transfer in a given time. Thus common wisdom stated
that the probability of error for digital communications was a function of
the SNR, while the speed was determined by the bandwidth. Shannons
capacity theorem completely changed this picture; by explaining that the
SNR and bandwidth establish a maximum transmission rate, under which
information could be transferred with arbitrarily low error rate. This result
will be the subject of the next section.
EXERCISES
18.7.1 Shannon introduced entropy (defined in the previous section) in connection
with source coding. The ultimate purpose of source coding is to produce no
more bits than required by the entropy content of the source. When is simple
A/D conversion the optimal source coding for an analog signal? What should
one do when this is not the case?
18.7.2 In order to achieve the maximum efficiency predicted by Shannon, source
coding is often required even for digital data. Explain and give several exam-
ples. (Hint: Data compression, fax.)
18.7.3 The Baudot code and ASCII are source codes that convert letters into bits.
What are the essential differences between them? Which is more efficient for
the transfer of plain text? How efficient is it?
18.7.4 In todays world of industrial espionage and computer hackers sensitive data
is not safe unless encrypted. Augment the diagram of Figure 18.9.C to take
encryption into account.
18.7.5 We often want to simultaneously send multiple analog signals (for example,
all the extensions of an office telephone system) over a single line. This process
is called multiplexing and its inverse demultiplexing, Show how this fits into
Figure 18.9.C.
674 COMMUNICATIONS SIGNAL PROCESSING
rNi
0 1 2 3 4 5 6 7
Figure 18.10: The effect of noise on amplitude resolution. The minimum possible spacing
between quantization levels is the noise amplitude N, and the total spread of possible signal
values is the peak-to-peak signal amplitude S plus the noise N. The number of levels is
thus the ratio between the signal-plus-noise and the noise, and the number of bits is the
base-two logarithm of this ratio.
encode the characters as bytes, and represent the entire text as a single
(rather lengthy) number. Normalizing this number to the interval between
0 and 1 we could, in principle, send the entire text as a single voltage in one
second through a noiseless channel. This demonstrates that the information-
carrying capacity of a noiseless channel is infinite.
What happens when there is noise? The precision to which the amplitude
can be reliably measured at the receiver is now limited by the noise. We cant
place quantization levels closer than the noise amplitude, since the observed
signals would not be reliably distinguishable. As is clarified by Figure 18.10
the noise limits the number of bits to the base-two logarithm of the signal-
plus-noise-to-noise ratio, SNNR = SNR + 1.
Of course, even if the noise limits us to sending b bits at a time, we can
always transmit more bits by using a time varying signal. We first send b
bits, and afterwards another b bits, then yet another b, and so on. Were
the channel to be of unlimited bandwidth we could abruptly change the
signal amplitude as rapidly as we wish. The transmitted waveform would
be piecewise constant with sharp jumps at the transitions. The spectral
content of such jump discontinuities extends to infinite frequency, but since
our channel has infinite bandwidth the waveform is received unaltered at the
receiver, and once again there is no fundamental limitation that hinders our
receiver from recovering all the information. So even in the presence of noise,
with no bandwidth limitation the channel capacity is effectively infinite.
Signals that fluctuate rapidly cannot traverse a channel with finite band-
width without suffering the consequences. The amount of time a signal must
676 COMMUNICATIONS SIGNAL PROCESSING
The last part of the capacity theorem tells us that a signal that optimally
fills the channel has no structure other than that imposed by the channel.
This condition derives from the inverse relation between predictability and
information. Recall from Section 5.2 that white noise is completely unpre-
dictable. Any deviation of the signal from whiteness would imply some pre-
dictability, and thus a reduction in information capacity. Were the signal to
be of slightly narrower bandwidth, this would mean that it obeys the differ-
ence equation of a band-pass filter that filters it to this shape, an algebraic
connection between sample values that needlessly constrains its freedom to
carry information.
The channel capacity theorem as expressed above is limited by two con-
ditions, namely that the bandwidth is filtered by an ideal band-pass filter,
and that the noise is completely white. However, the extension to arbitrary
channels with arbitrary stationary noise is (at least in principle) quite sim-
ple. Zoom in on some very small region of the channels spectrum; for a small
enough region the attenuation as a function of frequency will be approxi-
mately constant and likewise the noise spectrum will be approximately flat.
Hence for this small spectral interval the channel capacity theorem holds and
we can compute the number of bits per second that could be transferred us-
ing only this part of the total spectrum. Identical considerations lead us to
conclude that we can find the capacities of all other small spectral intervals.
In principle we could operate independent modems at each of these spectral
regions, dividing the original stream of bits to be transmitted between the
different modems. Hence we can add the information rates predicted by the
capacity theorem for all the regions to reach an approximate prediction for
the entire spectrum. Lets call the bandwidth of each spectral interval Sf,
and the signal-to-noise ratio in the vicinity of frequency f we shall denote
SNR(f). Then
.- r
C = xlog,(SNR(f) + 1) Sf
s
and for this approximation to become exact we need only make the regions
infinitesimally small and integrate instead of adding.
C= 1%2(swf) + 1) df (18.17)
s
We see that for the general case the channel capacity depends solely on the
frequency-dependent signal-to-noise ratio.
From the arguments that lead up to the capacity theorem it is obvious
that the SNR mentioned in the theorem is to be measured at the receiver,
where the decisions must be made. It is not enough to specify the transmitted
678 COMMUNICATIONS SIGNAL PROCESSING
Unfortunately, equation (18.18) is not directly useful for finding the max-
imal information capacity for the common case where we are given the line
attenuation A(t), the noise power distribution N(f) and the total transmit-
ted power P.
r
(18.19)
In order to find the maximal capacity we have to know the optimal trans-
mitter power distribution P(f). Should we simply take the entire power at
the transmitters disposal and spread it equally across the entire spectrum?
Or can we maximize the information rate of an arbitrary channel by trans-
mitting more power where the attenuation and noise are greater? A little
thought leads us to the conclusion that the relevant quantity is the noise-
to-attenuation ratio N(f)/A(f). I n regions where this ratio is too high we
shouldnt bother wasting transmitted power since the receiver SNR will end
up being low anyway and the contribution to the capacity minimal. We
should start spending power where the N/A ratio is lower, and expend the
greatest amount of power where the ratio is lowest and thus the received
SNR highest.
In other words, we should distribute the power according to
(18.20)
where the value of 0 is determined by the requirement (18.19) that the total
Power should equal P. Gallager called this the water pouring criterion. To
understand this name, picture the attenuation to noise distribution ratio as
an irregularly shaped bowl, and the total amount of power to be transmit-
ted as the amount of water in a pitcher (Figure 18.11). Maximizing signal
capacity is analogous to pouring the water from the pitcher into the bowl.
Where the bowls bottom is too high no water remains, where the bowl is
low the height of water is maximal.
18.8. CHANNEL CAPACITY 679
Figure 18.11: The water pouring criterion states that the information rate is maximized
when the amount of power available to be transmitted is distributed in a channel in the
same way as water fills an irregularly shaped bowl.
EXERCISES
188.1 SNR always refers to the power ratio, not the signal value ratio. Show that
assuming the noise is uncorrelated with the signal, the capacity should be
proportional to a log, SNR.
18.8.2 Using the sampling theorem, show that if the bandwidth is W we can trans-
mit 2W pulses of information per second. Jump discontinuities will not be
passed by a finite bandwidth channel. Why does this not affect the result?
18.83 Put the results of the previous examples together and prove Shannons the-
orem.
18.8.4 When the channel noise is white its power can be expressedas a noise power
density No in watts per Hz. Write the information capacity in terms of BW
and NO.
680 COMMUNICATIONS SIGNAL PROCESSING
18.8.5 Early calculations based on Shannons theorem set the maximum rate of
information transfer lower than that which is now achieved. The resolution
of this paradox is the improvement of SNR and methods to exploit more of
the bandwidth. Calculate the channel capacity of a telephone line that passes
from 200 Hz to 3400 Hz and has a signal-to-noise ratio of about 20-25 dB.
Calculate the capacity for a digital telephone line that passesfrom 200 Hz to
3800 Hz and encodesusing logarithmic PCM (12-13 bits).
18.8.6 The maximum reach of a DSL modem is defined to be the distance over
which it can function when the only source of interference is thermal white
noise. The attenuation of a twisted pair of telephone wires for frequencies
over 250 KHz can be approximated by
A(f) = e-s(nq/7+nsf)L
where L is the cable length in Km. For 24-gaugewire ~1 = 2.36. 10s3, K,Z =
-0.34.10-8 and for thinner 26-gaugewire i~r = 2.98e10-3, ~2 = -1.06~10-8.
Assume that the transmitter can transmit 13 dBm between 250 KHz and 5
MHz and that the thermal noise power is -140 dBm per Hz. Write a program
to determine the optimal transmitter power distribution and the capacity for
lengths of 1, 2, 3, 4, and 5 Km.
corrupted and how to recover the intended information. How can bit errors
be detected? Were we to send 00011011 and 01011010 was received instead,
how could this possibly be discovered? The strategy is that after optimizing
the source coding to use the minimal number of bits possible, the channel
coding adds new bits in order to be able to detect errors. A parity bit is
a simple case of this; to seven data bits we can add an eighth bit that en-
sures that the number of ones is even. Any single-bit error will be detected
because there will be an odd number of ones, but we will not know which
bit is in error. A simplistic error correction scheme could send each bit three
times in succession (e.g., send 000000000111111000111111 rather than di-
rectly sending the message 00011011). Were any single bit to be incorrectly
received (e.g. 000010000111111000111111), we could immediately detect this
and correct it. The same is true for most combinations of two bit errors, but
if the two errors happen to be the same bit triplet, we would be able to
detect the error but not to correctly correct it.
The error detection and correction method we just suggested is able
to correct single-bit errors, but requires tripling the information rate. It
turns out that we can do much better than that. There is a well-developed,
mathematically sophisticated theory of ECCs that we will not be able to
fully cover. This and the next two sections are devoted to presentation of
concepts of this theory that we will need.
All ECCs work by allowing only certain bit combinations, known as
codewords. The parity code only permits codewords with an even number of
ones; thus only half the possible bitvectors are codewords. The bit tripling
ECC works because only two of the eight possible bit triplets are allowed;
thus of the 231cbitvectors of length 3k, only one out of every eight are
codewords.
The second essential concept is that of distance between bitvectors. The
most commonly used distance measure is the Humming distance cl&, b2).
(the same Hamming as the window). The Hamming distance is defined as the
number of positions in which two bitvectors disagree (e.g., d(0011,0010) =
1). For bitvectors of length N, 0 5 d(bl, bz) 5 N and d(bl, b2) = 0 if and
only if br = bf2.
If we choose codewords such that the minimum Hamming distance dmin
between any two is M, then the code will be able to detect up to M-l errors.
Only if M errors occur will the error go unnoticed. Similarly, a code with
minimum Hamming distance M will be able to correct less than i M errors.
Only if there are enough errors to move the received bitvector closer to
another codeword (i.e., half the minimum Hamming distance) will choosing
the closest codeword lead to an incorrect result.
682 COMMUNICATIONS SIGNAL PROCESSING
EXERCISES
18.9.1 Consider the bit-tripling code. Assume the channel is such that the probabil-
ity of an error is p (and thus the probability of a bit being correctly detected
is 1 -p). Show that the average probability of error of the original bit stream
is Per, = 3p2(1 - p) + p3. Obviously, Per,. = p for p = 0 and p = 1. What is
P,,,. for p = i? Graph Per,. as a function of p. For i < p < 1 we see that
P,,,. > p, that is, our error correction method increases the probability of
error. Explain.
18.9.2 The bit-tripling code can correct all single-bit errors, and most two-bit errors.
Starting with eight information bits, what percentage of the two-bit errors
can be corrected? What percentage of three-bit errors can be detected?
18.9.3 A common error detection method is the checksum. A checksum-byte is gen-
erated by adding up all the bytes of the message modulo 256. This sum is then
appended to the message and checked upon reception. How many incorrectly
received bytes can a checkbyte detect? How can this be improved?
CO = do + dl + d3
d2 means cl and c2 will not sum correctly, and finally a mistaken d3 causes
all three checkbits to come out wrong. What if a checkbit is incorrectly
received? Then, and only then, a single ci will not check out. If no checkbits
are incorrect the bitvector has been correctly received (unless a few errors
happened at once).
The Hamming code is a linear code; it doesnt matter if you sum (xor)
two messages and then encode them or encode the two messages and then
sum them. It is thus not surprising that the relationship between the Ic-
dimensional information bitvector d and the n-dimensional codeword a can
be expressed in a more compact fGhion using matrices, -a = --Gd -
-
where all the operations are to be understood modulo 2. The n-by-k matrix
G is called the generator matrix since it generates the codeword from the
=
information bits. All linear ECCs can be generated using a generator matrix;
all systematic codes have the Ic-by-lc identity matrix as the top IGrows of G.
The Hamming 7/4 code can correct all single-bit errors, and it is optim%
since there are no 6/4 codes with this characteristic. Although it does make
the job of locating the bit in error simpler than checking all codewords, Ham-
ming found a trick that makes the job easier still. He suggested sending the
bits in a different order, d3d2dlcldocpzo and calling them h7h&&&3h&.
Now hr, h2 and h4 are both received and computed, and the correction pro-
cess is reduced to simply adding the indices of the incorrect checkbits. For
example, if hl and h2 dont check out then h1+2 = h3 should be corrected.
Hammings code avoids searching all the codewords by adding algebraic
structure to the code. To see this more clearly lets look at all the codewords
(in the original order)
0000000 1000101 0010110 1010011
0001011 1001110 0011101 1011000
(18.22)
0100111 1100010 0110001 1110100
0101100 1101001 0111010 1111111
and note the following facts. Zero is a codeword and the sum (modulo 2) of
every two codewords is a codeword. Since every bitvector is its own additive
18.10. BLOCK CODES 685
the coset leader is the error. This decoding strategy is too computationally
complex to actually carry out for large codes, so we add a mechanism for
algebraically locating the coset leader. Once the coset leader has been found,
subtracting it (which for binary arithmetic is the same as adding) from the
received bitvector recovers the most probable original codeword.
In order to efficiently find the coset leader we need to introduce two
more algebraic concepts, the parity check matrix and the syndrome. The
codewords form a k-dimensional subspace of n space; from standard linear
algebra we know that there must be an (n - k)-dimensional subspace of
vectors all of which are orthogonal to all the codewords. Therefore there is
an (n - k)-by-n matrix H called the parity check matrix, such that -- Ha = 0
for every codeword a. ItTactually easy to find H from the generatormatrix
G since we require % --- Gd = 0 for all possible information vectors d, which
= --
means the (n - k)-by-k matrix -- H G must be all zeros. Hence the parity check
matrix for the 7/4 Hamming code is
1101100
H= 0111010
=
1011001
as can be easily verified. The parity check matrix of a systematic n/k code
has the (n - k)-by-(n - k) identity matrix as its rightmost n - k columns,
and the rest is the transpose of the nonidentity part of the generator matrix.
What happens when the parity check matrix operates on an arbitrary
n-dimensional bitvector v = a + e? By definition the codeword does not
contribute, hence --H v = --
He the right-hand side being a (n - k)-dimensional
vector called the syndro6. The syndrome is thus zero for every codeword,
and is a unique indicator of the coset leader. Subtracting (adding) the coset
from the bitvector gives the codeword. So our efficient method of decoding a
linear code is simply to multiply the incoming bitvector by the parity check
matrix to obtain the syndrome, and then adding the coset leader with that
syndrome to the incoming bitvector to get the codeword.
By mapping the encoding and decoding of linear ECCs onto operations
of linear algebra we have significantly reduced their computational load.
But there is an even more sophisticated algebraic approach to ECCs, one
that not only helps in encoding and decoding, but in finding, analyzing,
and describing codes as well. This time rather than looking at bitstreams
as vectors we prefer to think of them as polynomials! If that seems rat her
abstract, just remember that the general digital signal can be expanded as
a sum over shifted unit impulses, which is the same as a polynomial in the
18.10. BLOCK CODES 687
time delay operator x -l. The idea here is the same, only we call the dummy
variable x rather than z-l, and the powers of x act as place keepers. The
k bits of a message (d&r&~& . . . &-r) are represented by the polynomial
d(x) = de + drx + d2x2 + d3x3 + . . . + dk-lx-; were we to take x = 2 this
would simply be the bits understood as a binary number.
The polynomial representation has several advantages. Bit-by-bit addi-
tion (xor) of two messages naturally corresponds to the addition (modulo
2) of the two polynomials (not to the addition of the two binary numbers).
Shifting the components of a bitvector to the left by r bits corresponds to
multiplying the polynomial by xr. Hence a s.ystematic n/k code that encodes
a k-bit message d into an n-bit codeword a by shifting it (n - k) bits to the
left and adding (n - k) checkbits c, can be thought of as a transforming of a
(k - l)-degree polynomial d(x) into a code polynomial a(x) of degree (n - 1)
by multiplying it by the appropriate power of x and then adding the degree
(n.- k - 1) polynomial c(x).
a(x) = xn+d(x) + c(x) (18.23)
Of course multiplication and division are defined over the polynomials,
and these will turn out to be useful operations-operations not defined in
the vector representation. In particular, we can define a code as the set of
all the polynomials that are multiples of a particular generator polynomial
g(x). The encoding operation then consists of finding a c(x) such that a(x)
in equation (18.23) is a multiple of g(x).
Becoming proficient in handling polynomials over the binary field takes
some practice. For example, twice anything is zero, since anything xored with
itself gives zero, and thus everything equals its own negative. In particular,
x+x = 0 and thus x2 + 1 = x2 + (x + x) + 1 = (x + l)2; alternatively,
we could prove this by x2 + 1 = (x + 1) (x - 1) which is the same since
-1x 1. How can we factor x4 + l? x4 + 1 = x4 - 1 = (x2 + 1)(x2 - 1) =
(x2 + 1)2 = (x + l)4. As a last multiplication example, its easy to show
that (x3 + x2 + 1)(x2 + x + 1) = x5 + x + 1. Division is similar to the usual
long division of polynomials, but easier. For example, dividing x5 + x + 1
by x2 + x + 1 is performed as follows. First x5 divided by x2 gives x3, so
we multiply x3 (x2 + x + 1) = x5 + x4 + x3. Adding this (which is the same
as subtracting) leaves us with x4 + x3 + x + 1 into which x2 goes x2 times.
This time we add x4 + x3 + x2 and are left with x2 + x + 1, and therefore
the answer is x3 + x2 + 1 as expected,
With this understanding of binary polynomial division we can now de-
scribe how c(x) is found, given the generator polynomial g(x) and the mes-
sage polynomial d(x). We multiply d(x) by xn-lc and then divide by g(x), the
688 COMMUNICATIONS SIGNAL PROCESSING
remainder being taken to be c(z). This works since dividing x~-~(x) + c(z)
by g(x) will now leave a remainder 2c(z) = 0. For example, the 23/12 Golay
code has the generator polynomial g(x) = 1+z+25+Z6+57+Zg+511; in
order to use it we must take in 12 bits at a time, building a polynomial of
degree 22 with the message bits as coefficients of the 12 highest powers, and
zero coefficients for the 11 lowest powers. We then divide this polynomial
by g(x) and obtain a remainder of degree 10, which we then place in the
positions previously occupied by zeros.
The polynomial representation is especially interesting for cyclic codes
due to an algebraic relationship between the polynomials corresponding to
codewords related by circular shifts. A circular shift by m bits of the bitvec-
tor (aoal . . . a,-~) corresponds to the modulo n addition of r-n to all the
powers of z in the corresponding polynomial.
l+mmodn+
aoxO+m mod n
+ a12 . . . + un-lx(n-l)+m mod n
and thus say that xu(x) and G(x) are the same modulo xn + 1. Simi-
larly, (x2u(x)) mod (xn + 1) corresponds to a circular shift of two bits,
and (xmu(z)) mod (xn + 1) to a circular shift of m bits. Thus cyclic codes
have the property that if u(x) corresponds to a codeword, then so does
(x%(x)) mod (xn + 1).
In 1960, two MIT researchers, Irving Reed and Gustave Solomon, realized
that encoding bit by bit is not always the best approach to error detection
and correction. Errors often come in bursts, and a burst of eight consecutive
bit errors would necessitate an extremely strong ECC that could correct
any eight-bit errors; but eight consecutive bit errors are contained in at
most two bytes, thus if we could work at the byte level, we would only need
a two-byte correcting ECC. The simplest byte-oriented code adds a single
checkbyte that equals the bit-by-bit xor of all the data bytes to a block of
byte-oriented data. This is equivalent to eight interleaved parity checks and
can detect any single byte error and many multibyte ones, but cannot correct
18.10. BLOCK CODES 689
any errors. What Reed and Solomon discovered is that by using r checkbytes
one can detect any T errors and correct half as many errors. Discussing the
theory of Reed-Solomon codes would take us too far astray, but the basic
idea is to think of the bytes as polynomials with bits as coefficients
where the b, are bits. Dividing the bit stream into TI bytes and adding all
these bytes together as polynomials
EXERCISES
18.10.1 A systematic code adds r = n - k checkbits and thus allows for 2r - 1
different errors to be corrected. So for all single-bit errors (including errors
of the checkbits themselves) to be corrected we require 2 - 1 2 n = k + r.
For k = 4 we require at least r = 3, i.e., a 7/4 code. What does this bound
predict for the minimum sizes of systematic codes for k = 3,5,8,11,12,16?
18.10.2 Find the codewords of the 7/4 Hamming code in Hammings order. Show that
the inverse of every Hamming 7/4 codeword is a code word. Show that the
sum (modulo 2) of every two codewords is a codeword. What is the minimum
Hamming distance dmin ? Show that the code is not cyclic but that the code
in the original order (as given in the text) is.
18.10.3 In the 7/4 Hamming code the inverse of every codeword (i.e., with ones and
zeros interchanged) is a codeword as well. Why?
18.10.4 Why are systematic codes often preferred when the error rate is high?
18.10.5 Find equations for the checkbits of the 9/5 Hamming code. How can the bits
be arranged for the sum of checkbit indices to point to the error? How should
the bits be arranged for the code to be cyclic?
18.10.6 Show that over the binary field xn + 1 = (x + l)(xnwl + xnw2 + . . . + 1).
18.10.7 The 16-bit cyclic redundancy check (CRC) error detection code uses the
polynomial 1 + x5 + x12 + x l6 . Write a routine that appends a CRC word
to a block of data, and one that tests a block with appended CRC. How can
the encoding and decoding be made computationally efficient? Test these
routines by adding errors and verifying that the CRC is incorrect.
690 COMMUNICATIONS SIGNAL PROCESSING
18.10.8 To learn more about block codes write four programs. The first bencode
inputs a file and encodes it using a Hamming code; the second channel inputs
the output of the first and flips bits with probability p (a parameter); the third
bdecode decodes the file with errors; the last compare compares the original
and decoded files and reports on the average error Per,.. Experiment with
binary files and plot the empirical Pe,-,. as a function of p. Experiment with
text files and discover how high p can be for the output to be decipherable.
* Y2n
A Al B AL
Xn k - Xn G-1 Xn
1r
l?
Y2n+l Y2n+l
Figure 18.12: A trivial convolutional code. In (A) we see the type of diagram used in
error correcting code texts, while in (B) we have the conventional DSP flow diagram
equivalent. The output bit signal is formed by interleaving the two outputs into a single
bit stream, although this parallel to serial conversion is not explicitly shown.
Why does this code work? As for block codes the idea is that not every
combination of bits is a valid output sequence. We cannot say that given a
bit the next bit must be the same, since we do not know whether the present
bit is already the replica of the previous one. However, an isolated zero or
one, as in the bit triplets 010 and 101, can never appear. This fact can be
used to detect a single-bit error and even some double-bit errors (e.g., the
two middle bits of 00001111). Actually all single-bit errors can be corrected,
since with few errors we can easily locate the transitions from 1 to 0 and
back and hence deduce the proper phase.
In Figure 18.13 we see a somewhat more complex convolutional code.
This code is universally the first presented in textbooks to demonstrate the
A
X,--t- Xn Xn-1 G-2 x:I...y: Y2n
llY2n+ 1
Figure 18.13: A simple convolutional code (the one universally presented in ECC texts).
In (A) we see the ECC style diagram, while in (B) we have the conventionalDSP flow
diagram equivalent. Note that the addition in the latter diagram is modulo 2 (xor).
692 COMMUNICATIONS SIGNAL PROCESSING
Figure 18.14: The function table and diagram for the simple convolutional code. There
are four states and both the output and the new state are dependent on the state and
the input bit. The new state is obtained by shifting the present state to the right and
placing the input bit into the twos place. In the diagram each arc is labeled by two binary
numbers, input/output.
Figure 18.15: Trellis diagram for the simple convolutional code. We draw all the paths
from state zero at n = 0 that return to state zero at time n = 6.
~~~~1
010000 001110110000
000010
010010
000000001110
001110111110
000001 000000000011
~~~~~~
000101 000000111000 000111 000000110110
Table 18.2: Exhaustive enumeration of the simple convolutional code (the one universally
presented in ECC texts). We input all possible input sequencesof six bits and generate
the outputs (assuming the shift register is reset to all zeros each time). These outputs can
be compared with received bit signal.
Section 8.7. To understand this algorithm consider first the following related
problem. You are given written directions to go from your house to the
house of a fellow DSP enthusiast in the next town. A typical portion of the
directions reads something like take the third right turn and proceed three
blocks to the stop sign; make a left and after two kilometers you see a bank
on you right. Unfortunately, the directions are handwritten on dirty paper
and you are not sure whether you can read them accurately. Was that the
third right or the first? Does it say three blocks or two blocks? One method
to proceed is to follow your best bet at each step, but to keep careful track of
all supplied information. If you make an error then at some point afterward
the directions no longer make any sense. There was supposed to be a bank
on the right but there isnt any, or there should have been a stop sign after
3 blocks but you travel 5 and dont see one. The logical thing to do is to
backtrack to the last instruction in doubt and to try something else. This
may help, but you might find that the error occurred even earlier and there
just happened to be a stop sign after three blocks in the incorrect scenario.
This kind of problem is well known in computer science, where it goes
under the name of the directed search problem. Search problems can be
represented as trees. The root of the tree is the starting point (your home)
and each point of decision is a node. Solving a search problem consists of
going from node to node until the goal is reached (you get to the DSP
enthusiasts house). A search problem is directed when each node can be
assigned a cost that quantifies its consistency with the problem specification
18.11. CONVOLUTIONAL CODES 695
(how well it matches the directions so far). Directed search can be solved
more systematically than arbitrary search problems since choosing nodes
with lower cost brings us closer to the goal.
There are two main approaches to solving search problems. Depth-first
solution, such as that we suggested above, requires continuing along a path
until it is obviously wrong, and then backtracking to the point of the last
decision. One then continues along another path until it becomes impossible.
The other approach is breadth-first. Breadth-first solution visits all decision
nodes of the same depth (the same distance from the starting node) before
proceeding to deeper nodes. The breadth-first solution to the navigation
problem tries every possible reading of the directions, going only one step.
At each such step you assign a cost, but resist the temptation to make any
decision even if one node has a low cost (i.e., matches the description well)
since some other choice may later turn out to be better.
The decoding of a convolutional code is similar to following directions.
The algebraic connection between the bits constitute a consistency check
very much like the stop sign after three blocks and the bank being on the
right. The Viterbi algorithm is a breadth-first solution that exploits the
state-space description of the encoder. Assume that we know that the en-
coder is in state 0 at time n = 0 and start receiving the encoded (output)
bits. Referring back to Figure 18.15, just as the decoder generated particular
transitions based on the input bits (the first number of the input/output pair
in Figure 18.14) the Viterbi decoder tries to guess which transition took place
based on the output bits (the second number of the pair). Were the received
bit signal error free, the task would be simple and uniquely defined. In the
presence of bit errors sometimes we will make an improper transition, and
sometimes we cannot figure out what to do at all. The breadth-first dynamic
programming approach is to make all legal transitions, but to calculate the
Hamming distance between the received bits and the encoder output bits
that would actually have caused this transition. We store in each node of the
trellis the minimal accumulated Hamming distance for paths that reach that
node. We have thus calculated the minimal number of errors that need to
have occurred for each possible internal state. We may guess that the path
with minimal number of errors is the correct one, but this would be too
hasty a decision. The proper way to identify the proper path in state-space
is to have patience and watch it emerge.
To see how this happens lets work out an example, illustrated in Fig-
ure 18.16. Well use as usual the simple convolutional code of Figure 18.13
and assume that the true input to the encoder was all zeros. The encoder
output was thus all zeros as well, but the received bit signal has an erroneous
696 COMMUNICATIONS SIGNAL PROCESSING
00 01 00 00 00 00 00 00
Figure 18.16: Viterbi decoding for the simple convolutional code. The bottom line con-
tains the received bit signal, which contains a single bit error. The trellis nodes are labeled
with the accumulated number of bit errors and the arcs with the output bits corresponding
to the transition. Dash arcs are ones that do not survive because there is a path to the
post-arc node that has fewer bit errors. The single bit error is corrected after 6 time steps.
one due to noise. We assign costs to the nodes in the trellis starting from
state zero at time n = 0. In the first time step we can reach state 0 if 00 were
transmitted and state 2 were 11 transmitted. Since 00 is received, state 0 is
labeled as being reachable without bit errors, while state 2 is labeled with
a 2 representing two bit errors. In the next time step the bit error occurs.
From the 0 state at n = 1 the code can transition either to itself or to state
2, implying single-bit errors. From state 2 at n = 1 we would arrive at state
1 assuming the 01 that was received was indeed transmitted, thus accruing
no additional error; the state 1 node at n = 2 is thus labeled with total error
2 since the entire path from state 0 at n = 0 was 1101 as compared with
the 0001 that was received. From state 2 at n = 1 we could also arrive at
state 3 at n = 2 were 10 to have been sent, resulting in the maximal total
number of bit errors (1110 as compared to 0001).
In this way we may continue to compute the total number of bit errors to
reach a given state at a given time. In the next step we can reach the 0 state
via two different paths. We can transition from state 0 at n = 2, implying
that the total bit sequence transmitted was 000000, or we can get there from
state 2 at n = 2 were the sequence transmitted to have been 110111. Which
cost do we assign to the node? Viterbis insight (and the basis of all dynamic
programming algorithms) was that we need only assign the lower of the two
costs. This reasoning is not hard to follow, since we are interested in finding
the lowest-cost path (i.e., the path that assumes the fewest bit errors in the
received bit signal). Suppose that later on we determine that the lowest-cost
path went through this node, and we are interested in determining how the
states evolved up to this point. It is obvious that the lowest-error path that
18.11. CONVOLUTIONAL CODES 697
reaches this node must have taken the route from the 0 state at n = 2. So
the global optimum is found by making the local decision to accept the cost
that this transition implies. In the figure we draw the transition that was
ruled out as a dashed line, signifying that the best path does not go through
here. So at each time only four surviving paths must be considered.
In general, at time step n we follow all legal transitions from nodes 0, I,
2, and 3 at time n - 1, add the new Hamming distance to the cost already
accrued in the pre-arc node, and choose the lower of the costs entering
the post-arc node. In the figure the transitions that give minimal cost are
depicted by solid lines. If two transitions give the same accumulated cost we
show both as solid lines, although in practical implementations usually one
is arbitrarily chosen. The reader should carefully follow the development of
the trellis in the figure from left to right.
Now that we have filled in the trellis diagram, how does this help us
decode the bit signal? You might assume that at some point we must break
down and choose the node with minimal cost. For example, by time n = 5
the path containing only state 0 is clearly better than the other four paths,
having only one error as compared with at least three errors for all other
paths. However, there is no need to make such risky quantitative decisions.
Continuing on until time n = 8 the truth regarding the error in the fourth
bit finally comes to light. Backtracking through the chosen transitions (solid
lines in the figure) shows that all the surviving paths converge on state 0 at
time n = 6. So without quantitatively deciding between the different states
at time n = 8 we still reach the conclusion that the most likely transmitted
bit signal started 00000000, correcting the mistakenly received bit. It is easy
to retrace the arcs of the selected path, but this time using the encoder
input bits (the first number from Figure 18.13) to state that the uncoded
information started 000.
In a more general setting, the Viterbi decoder for a convolutional code
which employs m delays (i.e., has s = 2 possible internal states), fills in a
trellis of s vertical states. At each time step there will be s surviving paths,
and each is assigned a cost that equals the minimal number of bit errors
that must have occurred for the code to have reached this state. Backtrack-
ing, we reach a time where all the surviving paths converge, and we accept
the bits up to this point as being correct and output the original informa-
tion corresponding to the transitions made. In order to save computational
time most practical implementations do not actually check for convergence,
but rather resume that all paths have converged some fixed time L in the
past. The Viterbi decoder thus outputs predictions at each time step, these
information bits being delayed by L.
698 COMMUNICATIONS SIGNAL PROCESSING
EXERCISES
18.11.1 In exercise 8.7.1 we introduced the games of doublets. What relationships
exist between this game and the decoding of convolutional codes?
18.11.2 Each of the convolutions that make up a convolutional code can be identified
by its impulse response, called the generator sequence in coding theory. What
is the duration of the impulse response if the shift register is of length K?
The constraint length is defined to be the number of output bits that are
influenced by an input bit. What is the constraint length if the shift register
length is K, and each time instant Ic bits are shifted in and n are output?
18.11.3 Which seven-tuples of bits never appear as outputs of the simple convolu-
tional code given in the text?
18.11.4 Repeat the last exercise of the previous section for convolutional codes. You
need to replace the first program with cencode and the third with cdecode.
Use the Viterbi algorithm.
18.11.5 The convolutional code yzn = xn + x,,+1,~2~+1 = x, + x,+2 is even simpler
than the simple code we discussed in the text, but is not to be used. To find
out why, draw the trellis diagram for this code. Show that this code suffers
from catastrophic error propagation, that is, misinterpreted bits can lead to
the decoder making an unlimited number of errors. (Hint: Assume that all
zeros are transmitted and that the decoder enters state 3.)
Ac
Bnn
c- 1 1 1 0 0 1 0 1
Figure 18.17: NRZ signals and DC. In (A) we see a straight NRZ signal. For each 1 bit
there is a positive voltage, while there is no voltage for a 0 bit. In (B) we see what happens
to this NRZ signal when DC is blocked. The positive voltages tend to decay, wiping out
the information in long runs of 1 bits. In (C) we see an attempt at removing DC from the
NRZ signal. For each 1 there is a positive voltage, while for each 0 bit there is a negative
voltage.
NRZ has a major problem. Unless all the input bits happen to be zero,
the NRZ signal will have a nonzero DC component. This is not desirable
for an information transmission system. We want the transmitter to supply
information to the receiver, not DC power! To ensure that DC power is not
inadvertently transferred many channels block DC altogether. The telephone
system supplies DC power for the telephone-set operation on the same pair
of wires used by the signal, so low frequencies cannot be made available for
the signals use. This restriction is enforced by filtering out all frequencies
less than 200 Hz from the audio. Attempts at sending our simple NRZ
signal through a channel that blocks DC will result in the signal decaying
exponentially with time, as in Figure 18.17.B. We see that single 1 bits can
be correctly interpreted, but long runs of 1s disappear.
A simple correction that eliminates the major part of the DC is to send
a signal value of + 3 for every one bit, and - % for every zero bit, as in
Figure 18.17.C. However, there is no guarantee that the input bit stream
will be precisely balanced, with the same number of ones and zeros. Even
if this is true for long time averages, for short times there is still some DC
(either positive or negative), and so the bits still decay.
Encoding methods discussed in Section 18.6 such as RZ or Manchester
coding completely remove DC, but at the price of doubling the bandwidth;
this is a price Shannon doesnt want us to pay. A better DC removing method
700 COMMUNICATIONS SIGNAL PROCESSING
1 1 1 0 0 1 0 1
Figure 18.18: Simple digital communications signals. In (A) we seeon-off keying (OOK),
in (B) frequency shift keying (FSK), and in (C) phase shift keying (PSK).
than to simply look for energy at the carrier frequency, probably by using a
band-pass filter of some sort, and judge whether it has passed a threshold
value. Passing merits a 1 bit, lower energy is taken as a 0. If noise appears in
the area of the frequency of interest, an intended 0 will be wrongly classified
as a 1, and negative noise values summing with the signal of interest can
cause a 1 to masquerade as a 0.
In order to ameliorate this problem noise should be explicitly taken into
account, and this could be done in two nonexclusive ways. The first is based
on the supposition that the noise is white and therefore its energy is the
same over all frequencies. We can thus require that the energy at the output
of a matched band-pass filter be much larger than the energy at nearby
frequencies. The second way is based on the assumption that the noise is
stationary and therefore its energy does not vary in time. We can thus
monitor the energy when the signal is not active or sending zeros, and require
that it pass a threshold set higher than this background energy. Such a
threshold is often called a Noise Riding Threshold (NRT).
While these mechanisms make our decision-making process more robust
they are not immune to error. Impulse noise spikes still thwart NRTs and
narrow-band noise overcomes a threshold based on nearby frequencies. In
702 COMMUNICATIONS SIGNAL PROCESSING
addition, we have added the possibility of new error types (e.g., when the
noise fluctuates at adjacent frequencies a perfectly good 1 can be discarded).
While the effects of noise can never be entirely overcome, OOK does not give
us very much to work with in our efforts to combat it.
Perhaps the simplest method to combat noise at adjacent frequencies is
to replace OOK with Frequency Shift Keying (FSK). Here we transmit a
sinusoid of frequency fo when a 0 bit is intended and a sinusoid of frequency
fl when we wish to send a 1 bit (see Figure 18.18.B). One can build an
FSK demodulator by using two band-pass (matched) filters, one centered
at fe and the other at f 1. Such a demodulator can be more robust in noise
since two energies are taken into account. One decision method would be to
output a 0 when threshold is exceeded at fo but not fr and a 1 when the
reverse occurs. When neither energy is significant we conclude that there is
no signal, and if both thresholds are surpassed we conclude that there must
be some noise or interference. When such a demodulator does output a 0
or 1 it is the result of two independent decisions, and we are thus twice as
confident. An alternative to FSK is Phase Shift Keying (PSK), depicted
in Figure 18.18.C. Here we employ a single frequency that can take on two
different phases; a demodulator can operate by comparing the received signal
with that of a sinusoid of constant phase. At first this seems no better than
OOK, but we will see that PSK is a highly effective method.
There are several ways of understanding why FSK is better than OOK.
Our first interpretation consisted of treating FSK as OOK with frequency
diversity (i.e., two independent OOK signals carrying the same information
but at different frequencies). Such diversity increases the robustness with
which we can retrieve information at a given SNR. This is as useful since
we can increase channel capacity by attacking either the bandwidth or the
noise constraints.
A second interpretation has to do with the orthogonality of sinusoids of
different frequencies. An alternative to the dual band-pass filter FSK de-
modulator multiplies the received signal by sinusoids of frequencies fo or fr
and integrates the output over time. Since this is essentially downmixing
and low-pass filtering, this demodulator is actually a specific implementa-
tion of the dual band-pass filters, but we can give it a new interpretation.
From equation (A.34) we know that sinusoids of different frequencies are
orthogonal, so multiplication by one of the sinusoids and integrating leads
to a positive indication if and only if this frequency is being transmitted.
This exploitation of sinusoid orthogonality is a new feature relative to OOK.
Were the component signals in FSK truly orthogonal then FSK would
be the answer to all our wishes. The problem is that sinusoids are only
18.12. PAM AND FSK 703
11 10 01 01 11 00 00 10
Figure 18.19: A four level PAM signal. Here the same information bits are transmitted
as in the previous figures, but twice as many bits are sent in the same amount of time.
orthogonal when integrated over all time. When only a short time duration
AT is available, the uncertainty theorem puts a constraint on the accuracy
of recognizing the difference between the two frequencies Af.
For example, the telephone channel is less than 4 KHz wide, and so a rather
large separation would be Af x 2 KHz. This implies that telephone line
FSK information transfer rates will not exceed around 300 b/s.
Of course the uncertainty theorem only directly limits the rate at which
we can change between different frequencies. By using a repertoire of more
than two frequencies we can increase the information transfer rate. Using
four possible frequencies fo, fr , fi, or f3 we simultaneously transmit two
bits of information at each time instant, doubling the information rate. This
technique is not limited to FSK; simple NRZ can be extended to multilevel
PAM by sending one of four different voltages, as in Figure 18.19. The
signal sent at each time period is usually called a symbol or baud (after
Emile Baudot), and the bit rate is double the symbol rate or baud rate. If
we use symbols that can take on 2m possible values, the data rate in bits per
second is m times the baud rate. Of course increasing the data rate in this
way has its drawbacks. The demodulator becomes more complex, having to
distinguish between many different levels. More significantly, if we compare
two signals with the same transmission power, the one with more levels
has these levels closer together. So we cannot increase the number of levels
without incurring a higher probability of misdetection in noise. The eventual
limit is when the level spacing is of the order of the noise intensity. This is
the way Shannon limits the capacity of multilevel PAM and multifrequency
FSK signals.
704 COMMUNICATIONS SIGNAL PROCESSING
EXERCISES
18.12.1 A differential encoder encodes a 1 bit as a change in the signal and a 0 bit as
no change. Show how to transmit differential NRZ, FSK and PSK. For which
of these three is the differential coding most useful? How does one make
a decoder that reconstructs the original bit stream from the differentially
encoded one? How is the ambiguity broken? What signal causes an output
of all Is? All OS?
18.12.2 If an FSK modulator employs two completely independent oscillators sr(t) =
A sin(2rrfit) and sz(t) = Asin(2nfit) then at the instant of switching the
signal will generally be discontinuous. Continuous Phase FSK (CPFSK)
changes frequency without phase jumps. Why is CPFSK better than non-
continuous phase FSK? Write a routine that inputs a bit stream and outputs
a CPFSK signal.
18.12.3 Program a multifrequency FSK modulator on a computer with an audio
output or speaker. How high can the symbol rate be before your ear can no
longer distinguish the individual tones?
18.12.4 The fundamental limitation on the FSK symbol rate is due to the time-
frequency uncertainty relation. There is no fundamental time-value uncer-
tainty relationship, so what is the source of the limitation on PAM symbol
rates?
18.12.5 Figure 18.19 uses the natural encoding of the numbers from 0 to 2. The PAM
signal called 2BlQ (used in ISDN and HDSL) uses the following mapping:
-3 -+ 00, -1 --+ 01, +l + 11, +3 + 10. What is the advantage of this 2BlQ
encoding, which is a special case of a Gray code? How can this be extended
to eight-level PAM? 2m-level PAM?
18.12.6 In the text we didnt mention Amplitude Shift Keying (ASK). Draw a signal
with four-level ASK. Why isnt this signal popular?
18.13 PSK
FSK demodulation is based on the orthogonality of the signals representing
the bit 0 and the bit 1; unfortunately, we have seen that this orthogonal-
ity breaks down as we try to increase the information transfer rate. Over
telephone lines FSK can be readily used for rates of 300-1200 b/s, but
becomes increasing problematic thereafter. In the previous section we men-
tioned PSK; here we will present it in more detail and show why it can carry
information at higher rates. PSK is commonly used for 1200-2400 b/s over
telephone lines.
18.13. PSK 705
so(t> = sin(27& t)
Sl (t) = sin(2nf,t + ?J)
where fc is the carrier frequency. We suggest using these two signals as our
information carriers, transmitting se when a 0 bit is to sent, and si for a
1 bit. We call this signal BPSK for Binary PSK. There are basically two
methods of demodulating PSK signals. Coherent demodulators maintain a
local oscillator of frequency fc and compare the frequency of incoming signals
to this clock. Incoherent demodulators do not maintain a precise internal
clock, but look for jumps in the incoming signals instantaneous phase.
Both BPSK demodulators are more complex than those we have seen so
far. Are BPSKs advantages worth the extra complexity? Yes, since unlike
FSK, where the two basic signals become orthogonal only after a relatively
long time has elapsed, SOand si are already orthogonal over a half cycle. So
we can transmit one of the signals SOor sr for as little as one-half of a cycle
of the carrier, and still discriminate which was transmitted. This is a major
step forward.
Is this the best discrimination available? The coherent demodulator mul-
tiplies the incoming signal by sin(27rf, t) and so after filtering out the com-
ponent at twice fc its output is either 0 or 5. We can increase the phase
difference by using the two signals
so(t> = sin(27r&t - 5)
Sl (t> = sin(27rf, t + 5)
and the difference between the output signals is now maximal. It is not hard
to see that sr = -se, so using a sinusoid and its inverse results in the best
discrimination. Plotting multiple traces of the demodulator output results
in an eye pattern, such as that of Figure 10.8. Using a phase difference of 7r
opens the eye as much as is possible.
We can now go to a multiphase signal in order to get more bits per
symbol. QPSK uses four different phases
so(t> = sin(2nf, t)
Sl w = sin(27rf,t + 5)
4 w = sin(2nf, t+ 7r)
S3 w = sin(27rht + 3;)
706 COMMUNICATIONS SIGNAL PROCESSING
Figure 18.20: I-Q plot for a QPSK signal with noise. We see that the effect of additive
channel noise is to replace the true constellation points with clouds centered at the original
points. Although with this noise level we could still make accurate decisions, were we to
add four more points to make 8PSK the clouds would touch and reception errors would
be inevitable.
detected are not exactly those transmitted. We see that the four points on
the unit circle have been transformed into four small clouds centered on the
original points. In order to recover the original information we have to decide
to which true constellation point each received point should be associated.
The decision is performed by a slicer and will be discussed in Section 18.18.
How much noise can be tolerated before the decisions become faulty? From
the figure it is obvious that if the radius of the noise cloud is less than half
the Euclidean distance between the constellation points, then most of the
decisions will be correct.
EXERCISES
18.13.1 What is the Euclidean distance between constellation points of an nPSK
signal? Why cant we increase the distance between the constellation points
by simply placing them on a larger circle?
18.13.2 Simulate a baseband nPSK signal and find its empirical spectrum. Vary n.
Do you see any change? Vary tb. What happens now?
18.13.3 In exercise 18.12.5 we saw how to use a Gray code for multilevel PAM. What
is the difference between a Gray code for PAM and one for nPSK? How is
this related to the Hamiltonian cycles of Section 12.1?
18.13.4 We saw that resetting of the clock rotates the nPSK constellation. How can
we ever be sure that we are properly interpreting the data?
18.13.5 Write programs that implement a QPSK modulator and demodulator. Try
adding noise to the signal. What happens if the demodulator carrier frequency
is slightly wrong? What if the baud rate is inaccurate?
708 COMMUNICATIONS SIGNAL PROCESSING
Figure 18.21: PSD of digital communications signals. In (A) we see the autocorrelation
of a NRZ signal carrying white random data. In (B) is depicted the PSD of this signal,
a sine squared with its first null at the baud frequency. In (C) we present the PSD of a
OOK signal with the same data.
18.14. MODEM SPECTRA 709
obvious that a PAM signal with n levels can be thought of as the sum of n
NRZ signals. From the linearity of the FT we can conclude that multilevel
PAM has exactly the same spectrum as NRZ. In particular, the bandwidth
of PAM is independent of the number of levels, but of course more levels
with the same energy effects the noise sensitivity. Of course we have been
comparing signals with the same baud rate; when we compare PAM signals
with the same bit rate, the signal with more bits per symbol has a lower
baud rate and hence a lower bandwidth.
What about an OOK signal? We dont have to recompute autocorre-
lations since we know that OOK is simply NRZ upmixed by the carrier.
Accordingly, we immediately conclude that its PSD is that depicted in Fig-
ure 18.2X, centered on the carrier frequency and taking up double the
bandwidth of the NRZ signal. This spectrum is shared by the multilevel
ASK signal as well. In fact, it is a quite general result that only the carrier
frequency and baud rate affect the spectrum.
Why should the bandwidth have doubled for the same baud rate? This
result hints that there is another degree of freedom that we are not ex-
ploiting, but that would not change the PSD. This degree of freedom is the
phase; by simultaneously modulating both the amplitude and the phase we
can double the bit rate without increasing the bandwidth. We will return to
this idea in Section 18.17.
EXERCISES
18.14.1 In our derivation of the PSD for the NRZ signal we didnt dwell on the DC.
What is the difference between the spectra of the NRZ and DC-removed NRZ
signals?
18.14.2 Derive the PSD of BPSK from that of DC-removed NRZ. Compare this
spectrum with that of OOK.
18.14.3 Summing n independent NRZ signals does not result in a nPAM signal with
equally probable levels. Why does this not affect the conclusion regarding
the PAM spectrum?
18.14.4 Compare the PSDs for BPSK, QPSK, and 8PSK with the same information
transfer rate.
18.14.5 Create random data NRZ and BPSK signals and compute their periodogram
using the FFT. Now use 01010101input (alternations). How is the spectrum
qualitatively different? Starting from the deterministic input of alternations
add progresively more randomness. How does the spectrum change?
710 COMMUNICATIONS SIGNAL PROCESSING
i i i t Y I t ; 3 i,
1 1 1 0 0 1 0 1
Figure
- 18.22: The effect of improper baud rate recovery. In this example the demodulator
clock runs slightly faster than the modulators, so that bit insertions result. On occasion
a symbol clock may fall directly on a transition, causingindefinite decisions.
18.15. TIMING RECOVERY 711
One could significantly improve this algorithm with little effort. Its bet-
ter to base the output decision on several successive sn, so some filtering
or median filtering should be performed. If the signal has some residual DC
(that may be time varying) it should be removed immediately after input
of the signal. Also, the linear interpolation between sn and s+i can be im-
proved. However, all such algorithms rapidly deteriorate in performance with
SNR degradation. Noise pulses can look like transitions or alternatively hide
true transitions, and even when a transition is properly identified its correct
location becomes obscure. What we really need is a method that exploits
the entire signal, not just those few signal values that straddle a transition;
exploiting the entire signal in order to derive a frequency requires spectral
analysis or narrow band-pass filtering.
Were the data to alternate like 01010101, the transmitted signal would
be a square wave, and its Fourier series would consist of a basic sinusoid at
half the baud rate, and all odd harmonics thereof. Even in severe noise this
harmonic structure would be easily discernible and by band-pass filtering
a sinusoid related to the desired clock could be recovered. A more direct
method would be to differentiate the signal (accenting the transitions), and
to take the absolute value (removing the direction of the transition) thus
creating a pulse train whose spectrum has a strong line at precisely the
baud rate. Of course the differentiation operation is very sensitive to noise
but the baud line will be strong enough to stand out.
When the NRZ data is random the differentiation and absolute value
operations produce a train of pulses similar to that of the alternations, but
with many of the pulses missing. The basic frequency of this signal is still
the baud rate, but the baud line in the spectrum is not as strong, However,
as long as there are enough transitions the baud rate can still be determined.
Using a PLL is helpful, since it is designed to lock onto approximately known
frequencies in noisy signals.
If there are long stretches of constant zeros or ones in the data the baud
spectral line will tend to disappear, and no amount of filtering will be able to
bring it back. We mentioned previously that by using a bit scrambler we can
eliminate long runs of 1 bits. The most popular scrambler in use is the two-
tap self-synchronizing LFSR scrambler, depicted in Figure 18.23.A. Why
is this scrambler called self-synchronizing? Contrast it with the alternative
method of running the LFSR locked upon itself (see Section 5.4) to create
an LFSR sequence, and xoring the data with this sequence. That method
also increases the number of alternations in sections where there are long
18.15. TIMING RECOVERY 713
Figure 18.23: Two-tap scrambler and descrambler. In (A) we see the LFSR-based scram-
bler. The output is composed of the xor of the input bit with two previous input bits.
The length of the shift register equals the delay of the oldest bit required. In (B) we see
the descrambler. Note that there is no feedback and hence bit errors do not propagate
without limit.
that these signal points are precisely the nth roots of unity, and so raising
them to the nth power gives one. The usual convention is rotated by 45
with respect to this, (i.e., all the signal points are multiplied by &a). Hence
in the usual convention all the signal points raised to the nth power still
return the same value, only now that value is ey . Thus for BPSK squaring
the signal points at 45 and 225 gives i and for QPSK raising any of the
points 45, 135) 225, 315 to the fourth power gives -1. The important fact
is that raising any of the constellation points to the n th power returns the
same value.
Now the signal in the time domain is sin(wt + &) , where the & are the
n possible signal phases. It is clear from the result of the previous paragraph
that raising the signal to the nth power on a sample-by-sample basis will
wipe out the & dependence; and so the nth root of this will be a simple
sine of constant phase at the carrier frequency.
EXERCISES
18.15.1 Assume that a signal contains no runs of single bits, but only runs of two,
three and longer. Can the baud rate be recovered? Suggest a method.
18.15.2 Generate a PSK signal with random data, limit its bandwidth by FIR low-
pass filtering, and perform amplitude demodulation. Do you see the AM
dips? Now filter the AM using a narrow IIR filter centered at the nominal
baud rate. Empirically determine the delay between the zero crossings of the
sinusoidal output of this filter and the center of the symbols. Does this system
give accurate baud rate recovery?
18.15.3 Can baud or carrier recovery be performed on signals attaining the Shannon
capacity?
18.15.4 In systems with very high baud rate there can be a problem in providing the
timing on a second pair of wires. What is this problem and how can it be
overcome?
18.16 Equalization
We discussed adaptive modem equalizers in Section 10.3. The problem with
standard (linear) equalizers for telephone modems is that near the band
edges (under 400 Hz or above 3600 Hz) there can be 10 to 20 dB of attenu-
ation. In DSL modems the higher frequency ranges can be attenuated by 50
18.16. EQUALIZATION 715
from feedforward
channel equalizer
. feedback
equalizer
I I
Figure 18.24: Decision feedback equalizer. The DFE consists of two FIR filters, one in
the feedforward path and one in the feedback path. The slicer becomes an integral part
of the equalizer.
EXERCISES
18.16.1 Taking the IS1 to be weak and from precisely one previous symbol, show that
each constellation point splits into a small cluster of points that resembles the
entire constellation. What happens if the IS1 is from two previous symbols?
What happens when the IS1 is large and its duration long?
18.16.2 Simulate a QPSK signal traversing a noisy channel sharply attenuated at its
edges. Compare the optimal linear equalizer with the optimal DFE.
18.16.3 An inverse filter at the modulator may cause the transmitted signal to reach
values much larger than originally intended. The Tomlinson equalizer over-
comes this by a modulo operation, and a compensating operation at the
demodulator. Explain how this can be accomplished.
18.17 QAM
In Section 18.14 we saw that the bandwidth of an nPSK signal is not n
dependent. Accordingly, we can achieve higher information transfer rates in
a given bandwidth simply by increasing n. The problem is that for larger n
the constellation points are closer together. Since channel noise causes the
received signal phase to deviate from that transmitted, as depicted in the
constellation plot of Figure 18.20, there is a limit on how close we can place
constellation points. This is how the channel capacity theorem limits capac-
ity for PSK signals. Were there to be no noise we could achieve arbitrarily
large transfer rates by using large n; were there no bandwidth limitation we
could use BPSK and arbitrarily large baud rates.
Looking closely at the constellation plot of Figure 18.20 we can see a way
out. The additive channel noise expands the constellation points into circular
clouds in the I-Q plane, and our decision making is optimized by maximizing
the Euclidean distance between constellation points. One way this can be
done is by placing constellation points as shown in Figure 18.25. Here the
symbols differ in both phase and amplitude. This type of signal (being simul-
taneously PSK and ASK), is sometimes called names like APSK, but more
usually goes under the name Quadrature Amplitude Modulation (QAM).
Understanding the meaning of QAM requires thinking of the I and Q com-
ponents as two independent PAM signals in quadrature. This is indeed
another way of building a QAM signal; rather than altering the amplitude
and phase of a single carrier, we can independently amplitude modulate a
sine and its corresponding cosine and add the results.
18.17. QAM 717
Figure 18.25: Two-ring constellation and signal. From either representation it can be
seen that the symbols differ in both amplitude and phase. The two-ring signal allows pre-
cisely two different amplitudes. The phases are chosen to maximize the minimum distance
between constellation points.
Lets calculate how much we gained by using QAM. The minimum dis-
tance between points in the 8PSK constellation is 2 sin (3) x 0.765. The two-
ring constellation consists of the symbols (1, l), (-1, l), (-1, -l), (1, -l),
(3, o>, (0,3), (-TO), and (0, -3) and so its minimum distance is 2. However,
this is not a fair comparison since the average energy of the two-ring con-
stellation is higher than that of 8PSK. We can always increase the minimum
distance by increasing the energy; were we to put the 8PSK on a circle of
radius 4 the minimum distance would be 4 0.765 > 3! The proper way to
l
distance between points for a constellation with given energy. The problem
is thus purely a geometric one, and some of the solutions presently in use
are shown in Figure 18.26.
It has become conventional to use square constellations with odd inte-
ger coordinates. For example, the 16QAM constellation consists of symbols
(-3,3), (-1,3), . . . (1, -3), (3, -3). What is the energy of this constellation?
It has four points with energy (zH)~ + (&1)2 = 2, eight with (zH)~ +
2
(9 = 10, and four with (~t3)~ + (&3)2 = 18, so that the average is
(4.2 + 8.10 + 4.18)/16 = 10. Since the unnormalized minimum distance is
2, the normalized minimum distance is 2/m M 0.632. This is lower than
that of the previous constellations, but each symbol here contains 4 bits of
information, one bit more than that of the eight-point constellations.
We will see in the next section that it is easiest to build slicers for square
constellations, but rectangular constellations have a drawback. The corners
have high energy, and may even be illegal in channels with maximum power
restrictions. The optimum constellation boundary would be a circle, and this
is closely approximated by the V.34 constellation. The cross-shaped constel-
lations are a compromise whereby the worst offenders are removed, the slicer
remains relatively simple, and the number of points in the constellation re-
mains a power of two.
EXERCISES
18.17.1 Why are cross constellations used for odd numbers of bits per symbol and
square-shaped constellations for even numbers?
1817.2 Write a program to compute the average energy and normalized minimum
distance for all the constellations in Figure 18.26.
1817.3 Can you write a program that outputs the points in the V.34 constellation?
(Hint: There are 1664 odd-integer-coordinate points bounded by a circle.)
18.17.4 Show that PAM constellations with m bits have average energy E = $(4m-l)
and hence require about four times (6 dB) more energy to add a bit. Repeat
the calculation for square QAM constellations.
18.17.5 Square QAM constellations suffer from the same 90 ambiguity as nPSK.
Show how differential encoding can be combined with 16QAM.
18.17.6 Some people have suggested using hexagonal constellations. What are these
and why have they been suggested?
18.17.7 Prove that by squaring a QAM signal one can recover the baud rate. Prove
that taking the fourth power of a QAM signal enables carrier recovery. Show
that the rounder the constellation the harder it is to recover the carrier.
18.17. QAM 719
BPSK QPSK
.I. .. ...
8PSK
I
V.29 7200 bps Al&. 3 bit
. . . .
. t .
f-hip II-
l-iQALl V.29 9600 bps
....................
................
..........
V.34 33600 bps
Figure 18.26: Some popular QAM constellations. The BPSK and QPSK constellations
are used by the simplest modems (e.g., V22bis telephone-grade modem at 2400 b/s). The
second row all have three bits per symbol, and the third row all four bits per symbol. The
fourth row contains the constellations used by the V.32bis standard modem, with 5, 6,
and 7 bits per symbol. The V.34 1664-point constellation has been magnified for clarity.
720 COMMUNICATIONS SIGNAL PROCESSING
Figure 18.27: Input to the slicer. The input represents three seconds of received two-
dimensional points from a V.32bis modem operating at 14.4 Kb/s. The constellation is
readily recognizable to the eye, but the slicers decisions are not always clear cut.
18.18. QAM SLICERS 721
Figure 18.28: Operation of slicer for 8PSK. In (A) we see that the eight Voronoy regions
are pie-slice in shape, being only phase dependent. In (B) only the phase of the same
constellation is depicted, and we see that the slicer has been reduced to a quantizer.
tional smearing dominates. However, well assume that the clouds are cir-
cularly symmetric, as would be the case if the major contribution is from
additive channel noise. Under this assumption the optimal operation of the
slicer is to choose the constellation point that is closest to the received point.
So a straightforward slicer algorithm loops on all the N constellation points
and selects that constellation point with minimal distance to the received
point. This algorithm thus requires N computations of Euclidean distance
(sum of squares) and comparisons. Were the constellation points to be ran-
domly chosen this complexity would perhaps be warranted, but for the types
of constellation actually used in practice (see Figure 18.26) much more ef-
fective algorithms are available.
The principle behind all efficient slicers is exploit ation of Voronoy region
symmetries. Given an ar bitrary collection of points, the Voronoy region as-
sociated with the n th point is the set of all points closer to it than any
of the other points; the collection of all the Voronoy regions tessellate the
plane. For the nPSK modem, having all its constellation points on the unit
circle, it is not hard to see that the Voronoy zones are pie-slice in shape
(see Figure 18.28.A). The optimal slicer will slice up space into these pie
slices and determine into which slice a received point falls. In particular we
neednt consider the amplitude of the received point, and the optimal deci-
sion involves only its phase (as was assumed when we originally discussed the
nPSK demodulator). When depicted in one-dimensional (phase-only) space
(see Figure 18.28.B) the decision regions are even simpler. Neglecting the
wrapping around of the phase at the edges, the slicing is reduced to simple
inequalities. By correctly choosing the scale*and offset, the slicing can even
be reduced to simple quantizing!
722 COMMUNICATIONS SIGNAL PROCESSING
Figure 18.29: The Voronoy regions for square QAM constellations are square. To show
this we investigate the immediate vicinity of an arbitrary symbol point, and connect this
point with all neighboring symbols (the gray lines). We cut these lines with perpendicular
bisectors (the dark lines) in order to separate points closer to the center symbol from those
closer to the neighboring symbol. We shade in gray the area containing all points closest
to the center symbol.
The slicer for nPSK was so simple that it could be reduced to a single
quantization operation; but the more complex constellations are inherently
two-dimensional. Many non-PSK constellations are based on square arrange-
ments of symbol points, the Voronoy regions for which are themselves square,
as can be seen from Figure 18.29. Square Voronoy regions are only slightly
more complex to manipulate than their one-dimensional counterpart, evenly
spaced points on the line. By properly choosing the scale and offsets the de-
cision algorithm is reduced to independent quantizing along both axes. Of
course some points quantize to grid points that are outside the constellation,
and in such cases we need to project the decision back toward a constella-
tion point. For cross-shaped constellations a slight generalization of this
algorithm is required; for example, in Figure 18.27 we observe a point in the
lower-left corner and another in the upper right that have to be specially
handled.
We have succeeded in simplifying the slicer from the straightforward
algorithm that required N distance computations and comparisons to one
or two quantizations and some auxiliary comparisons. These simplifications
depend on the geometric structure of the constellation. Indeed the efficiency
of the slicer is often taken as one of the major design considerations when
choosing the constellation.
18.19. TRELLIS CODING 723
EXERCISES
18.18.1 Write a routine that efficiently implements a slicer for square 16QAM. Be
careful about how you handle inputs that quantize to grid points outside the
constellation. Are divisions required?
18.18.2 Write a routine that efficiently implements a slicer for the cross-shaped 128-
point constellation used in V.32bis. Inputs that quantize to grid points in the
corners should only require a single additional comparison.
18.18.3 What are the Voronoy regions for the 16-point V29 constellation?
18.18.4 What are. . .
the Voronoy regions for a hexagonal constellation such as the 12
. . . ? How can a slicer be efficiently implemented here?
point .....
0.10 0.92 0.05 0.49 1.02 0.94 0.08 0.04 c- input signal
0 1 0 0 1 1 0 0 c- demodulated bits
0 1 0 1 1 1 0 0 c- corrected bits
A o B oo c ooo
Figure 18.30: Set partitioning for the simplest TCM system. In step (A) the eight points
are partitioned into two QPSK subconstellations, named 0 and 1. In step (B) each of
the QPSK subconstellations is partitioned into two BPSK subsubconstellations, the 0
subconstellation into 00 and 01, and the 1 subconstellation into 10 and 11. In (C) the
subsubconstellation points themselves are labeled.
18.19. TRELLIS CODING 725
state
C cbmOOO
A =
Figure 18.31: The modulator for the simplest TCM system. One of the input bits is
passed directly to the 8PSK modulator, and the other is encoded by a trivial rate l/2
convolutional code. The state is the previous bit and the outputs are the present bit and
the state.
Figure 18.32: The trellis table and diagram for the simplest TCM system. The table
gives the output and new state given the present state and the input bits. The diagram
depicts a single time step. Between each state and each new state there are two parallel
transitions, labeled BA/cba.
Figure 18.33: The minimal error event for uncoded QPSK. The symbol s was transmit-
ted but the symbol -s + -n was received. The noise vector is of length fi and hence of
energy 2.
18.19. TRELLIS CODING 727
Without error we would stay in state zero all the time and receive only
the 0 constellation point. Since we are in state zero there is no way we
could receive an odd-numbered constellation point, so an error must result
in receiving one of the points 2, 4, or 6, with 2 and 6 being the closest. Both
2 and 6 correspond to the same noise energy as in the QPSK case, but also
switch the state to one, so that the next constellation point received will
be odd numbered. The odd numbered points of minimal distance from the
true point of zero are 1 and 7, both with distance about 0.765 and energy
0.5858. Thereafter the state reverts to 0 and the proper constellation point
may be received again. So the combined noise energy of the two errors that
make up this error event is about 2.5858. If the constellation point labeled
4 is mistakenly received the state does not change, but this corresponds to
noise of energy 22 = 4. So the minimal noise energy is 2.5858 as compared
with 2 for the uncoded case. This corresponds to an improvement of a little
over 1.1 dB.
By using more complex ECCs we can get more significant gains. For
example, the four-state code of Figure 18.34 is described in Figure 18.35.
It is not hard to see that the minimal energy error event occurs when we
substitute the constellation point 001 for 000, taking the parallel transition
and remaining in state 0. The energy of this event is 22 = 4 rather than
2 for a coding gain of 3 dB. By using even more complex ECCs we can
achieve further coding gains, although the returns on such computational
investment decrease.
The first standard modem to use TCM was the CCITT V.32 modem at
9600 b/s. In order to make TCM practical, the trellis code must be made
011 G 8PSK
modulator
010
101 110
Figure 18.34: The modulator for a four-state 8PSK TCM system. One of the input bits
is passed directly to the 8PSK modulator, and the other is encoded by a two state l/2
convolutional code. The state consists of the previous two bits and the outputs are formed
by binary additions (xor) of the present bit and the state bits.
728 COMMUNICATIONS SIGNAL PROCESSING
I I I
I oiooj I 2iloj I silloj 1 2iioj I
Figure 18.35: The trellis table and diagram for the four-state 8PSK TCM system. Once
again there are two parallel transitions between each two states, labeled by BA/cba.
EXERCISES
18.19.1 The simplest TCM described in the text uses the replicating convolutional
code with a particular phase, outputting at a given time the present input
bit and the previous one. What would happen if we output the present bit
twice instead?
18.19.2 Calculate the noise energies for the different possible error events for the
four-state 8PSK TCM system and show that the minimal event is indeed
that mentioned in the text.
18.19.3 Devise a set partition for the 16 QAM constellation. To do this first partition
the 16 points into two subconstellations of 8 points each, with each subcon-
stellation having maximal minimal distance. What is this distance and how
does it relate to the original minimal distance? Now continue recursively until
each point is labeled by four bits. By how much does the minimal distance
increase each time?
18.19.4 Why does TCM simplify (or even eliminate the need for) the slicer?
18.20. TELEPHONE-GRADE MODEMS 729
The next major breakthrough was the 1971 introduction by Co&x and
the 1976 standardization of V.29. This modem achieved a half-duplex rate of
up to 9.6 Kb/s by using an unusual 16-point QAM constellation with carrier
1700 Hz and baud rate 2400. Another innovative aspect of this modem was
its adaptive equalizer. This modem is still popular as the 9600 fax, where
half-duplex operation is acceptable.
Surprisingly, more than two decades after Shannon had predicted much
higher rates, technology appeared to stop at 9.6 Kb/s. Popular wisdom be-
lieved Shannons predictions to be overoptimistic, and efforts were devoted
to implementational issues. Then in 1982, Ungerboeck published his paper
revealing that an eight-state TCM code could provide a further 3.6 dB of
gain, and the race toward higher rates was on again. The next step should
have been to double the 9.6 Kb/s to 19.2 Kb/s, but that leap wasnt achieved.
At first the V.33 modem achieved 14.4 Kb/s full-duplex on a four-wire line,
and its two-wire half-duplex version (V.17) is the standard 14.4 fax used to-
day. Next, with the introduction of DSP echo cancelling techniques, V.32bis
achieved that same rate for full-duplex on two wires. All of these modems
use fc = 1800 Hz, fb = 2400 Hz and a 12%point cross-shaped constellation.
Since one of the seven bits is used for the coding, the remaining six bits times
2400 baud result in 14.4 Kb/s. These modems also provide 64-point square
and 32-point cross constellations for 12 Kb/s and 9.6 Kb/s respectively.
V.32bis had attained 14.4 Kb/s, so the next major challenge, dubbed
V.fast, was to attempt to double this rate (i.e., to attain 288Kb/s). For
several years different technologies and signal processing techniques were
tried, until finally in 1994 the V.34 standard was born. V.34 was a quantum
leap in signal processing sophistication, and we will only be able to mention
its basic principles here. Due to the complexity of the signal processing, most
V.34 modems are implemented using DSP processors, rather than special-
purpose DSP hardware.
The original ITU-T V.34 specification supported all data rates from 2.4
Kb/s to 28.8 Kb/ s in increments of 2.4 Kb/s, and an updated version added
two new rates of 31.2 and 33.6 Kb/s as well. Two V.34 modems negotiate
between them and connect at the highest of these data rates that the channel
can reliably provide.
We have already seen in Figure 18.26 the constellation used by V.34 for
33.6 Kb/s operation. For lower rates subconstellations of this one are used.
This constellation is by far the most dense we have seen.
Recall from equation (18.20) that the information transfer rate in any
given channel is maximized by matching the PSD to the channel charac-
teristics. One of the problems with the modems up to V.32bis is that they
18.20. TELEPHONE-GRADE MODEMS 731
baud rate (Hz) low carrier (Hz) high carrier (Hz) maximum data rate (Kb/s)
2400 1600 1800 21.6
2743* 1646 1829 26.4
2800* 1680 1867 26.4
3000 1800 2000 28.8
3200 1829 1920 31.2
3429* 1959 - 33.6
Table 18.3: The basic parameters for V.34. The baud rates marked with an asterisk
are optional. Each baud rate, except the highest, can work with two possible carrier
frequencies.
have a single predefined carrier and single baud rate, and hence their PSD is
predetermined. The PSD of V.32bis stretches from 2400 - 1800 = 600 Hz to
2400 + 1800 = 3000 Hz irrespective of the channel characteristics. V.34 pro-
vides six possible baud rates (three mandatory and three optional) as well as
nine carrier frequencies. A V.34 modem starts by probing the channel with a
probe signal that creates the distinctive bong noise you hear when trying to
connect with a V.34 modem. This probe signal (see exercise 2.6.4) consists
of a comb of sinusoids spaced 150 Hz apart from 150 Hz to 3750, except that
the 900, 1200, 1800 and 2400 Hz tones have been removed. Using the probe
signal the receiving modem can determine the frequency-dependent SNR,
decide on the maximum data rate that can be supported with reasonably
low bit error rate, and inform the transmitting modem which carrier and
baud rate best match the channel. The possible baud rate and carriers are
given in Table 18.3.
The second half of the channel capacity theorem specifies how the signal
that maximizes information transfer rate should look. It should appear as
white noise other than the water-pouring filtering. The suboptimality of
V.32bis can be easily ascertained by observing its spectrum. With V.34
techniques were added to whiten the modems spectrum.
Looking at Figure 18.3 we note that the data rate is not always an
integer multiple of the baud rate (i.e., there is a noninteger number of bits
per symbol). For example, the 33.6 Kb/s maximum bit rate requires 8.4 bits
per symbol at 3429 baud. This feat is accomplished using a shell mapper.
In ordinary QAM the constellation points are used with equal proba-
bility, so that the received (imperfectly equalized) I-Q plot is homogeneous
inside a disk. A noise signal would have its I-Q plot distribution decrease as
a Gaussian function of the radius. We can imitate this behavior by divid-
ing the constellation into concentric circles called shells, and, based on the
data to be transmitted, first choose a shell and then the point within the
732 COMMUNICATIONS SIGNAL PROCESSING
shell. By using an algorithm that prefers interior shells we can transmit the
constellation with a more Gaussian distribution.
V.34 also uses more powerful TCM codes than the Wei code used in V.32.
The standard specifies three codes, a 16-state code (also invented by Wei)
with 4.2 dB gain, a 32-state code with 4.5 dB gain and a 64-state code with
4.7 dB gain. All three of these codes are four-dimensional, meaning that they
are based on four-dimensional symbols built up from two consecutive two-
dimensional ones. Why should we want to group two transmitted symbols
into a more complex one? The reason has to do with the geometry of n-
dimensional space. Note that in one-dimensional space we can only place two
points at unity distance from a given point, while in two-dimensional space
there can be four such, and in n-dimensional space, 2n nearest neighbors.
Thus for a given amount of energy, we can place more constellation points
and thus carry more information, in higher-dimensional space. Of course the
four-dimensional symbols are actually transmitted as two two-dimensional
ones, but not every combination of consecutive two-dimensional symbols is
possible.
In order to widen the usable bandwidth V.34 uses a more powerful equal-
ization technique. Although DFE is capable of attaining close to the Shannon
capacity, it has several drawbacks, the most important being that it is hard
to combine with TCM. For V.34 a Tomlinson type equalizer was chosen
instead. During the initialization a DFE is trained and the feedback coef-
ficients sent to the modulator, where they are used as a precoder. Taking
the decision element out of the receive data path now makes integration
of the equalizer with the TCM possible. A new mechanism called flexible
preceding was invented to specifically integrate the precoder with the rest
of the V.34 engine.
The logarithmic encoding used in the digital telephone system (p-law
or A-law) compresses the outer constellation points, making decisions diffi-
cult. V.34 has an option called nonlinear encoding or warping designed
to combat these distortions. When enabled, the constellation is distorted,
increasing the distance between outer constellation points, at the expense of
decreasing that of more interior points.
The extremely sophisticated signal processing of the V.34 standard took
years to develop and several years more to agree upon in standards com-
mittees. Yet, paradoxically, although for all intents and purposes V.34 at
last approached the Shannon limit, it reigned supreme for only about a
year. The next step, the step that would once again double the transmission
speed from 28K to 56K, was just around the corner.
18.21. BEYOND THE SHANNON LIMIT 733
EXERCISES
18.20.1 Plot the PSDs of V.22bis, V.32, and the various modes of V.34 and compare
spectral utilization.
18.20.2 Obtain diagrams of th.e initialization phases of V.32bis and V.34. Can you
explain what happens at each stage?
lower levels are more probable than the higher ones; when digital data is
sent all levels are equally probable, resulting in a higher average power,
There is another problem with this attempt at transmitting 64 Kb/s. In
some parts of the world not all 8 bits are available. Every now and then
the least significant bit is robbed for other uses. This does not degrade
voice quality very much, but would be most undesirable for data. In order
to reduce the average power we can use shell mapping, and because of this,
together with overcoming the robbed-bit phenomenon, we should not expect
more than 7 bits 8000 times a second, for a grand total of 56 Kb/s.
What would happen if someone at the CO were to send us digital data at
56 Kb/s with an appropriate shell mapping? Would we be able to distinguish
between these closely spaced levels? There would be ISI, but that could be
overcome by an equalizer. We would need an echo canceller to remove our
own transmission, but that too is well-known theory. It turns out that if we
send digital levels directly on the pair of wires going to the other modem,
then it is possible to recover the original levels. The data source need not
sit physically in the telephone office, as long as its connection to that office
is completely digital.
This is how the V.90 56 Kb/s modem works. A V.34 modem is used in
the upstream direction, that is, from the consumer to the service provider.
In the downstream direction a shell-mapped digital signal of up to 56 Kb/s
is sent. This asymmetry is acceptable for many applications (e.g., for In-
ternet browsing where the downstream often consumes ten times the data
rate as the upstream). In a newer version dubbed V.92 even the upsteam
transmission tries to overcome the last mile and jump onto the digital link.
V.90 exceeds Shannon by exploiting the fact that the telephone system
is no longer a 4 KHz analog channel, and thus the maximum possible rate is
the 64 Kb/s used by the telephone system itself. Getting even higher than
64 Kb/s requires an even more radical departure from our model of the
telephone system.
We have mentioned that the telephone network remains analog only in
the last mile to the subscriber, more formally called the subscriber line.
Now if we look at the frequency response of such subscriber lines, we find
behaviors such as those of Figure 18.36. Although there is strong attenuation
at high frequencies, the bandwidth is definitely higher than 4 KHz.
The 4 KHz restriction is actually enforced by filters at the telephone
office, in order to enable multiplexing of multiple telephone conversations
on a single carrier. There is nothing inherent in the subscriber line that
recognizes this bandwidth restriction. So if we can place our modem before
the filters and are allowed to use the subscriber line as a general-purpose
18.21. BEYOND THE SHANNON LIMIT 735
Figure 18.36: The attenuation for unshielded twisted-pair lines. Depicted is the line
attenuation in dB for one kilometer of standard telephone-grade 24.gauge (upper curve)
and 26-gauge (lower curve) unshielded cable. For two kilometers of cable the attenuation
in dB is doubled.
cable, the so-called Digital Subscriber Line (DSL) modems can reach much
higher capacities. Of course we may desire to continue to use the same
subscriber line for our regular phone conversations, in which case a splitter
is placed at the end of the line. A splitter is simply a low-pass filter that
passes the low frequencies to the phone, and a high-pass filter that delivers
the high frequencies to the DSL modem.
The longer the subscribers cable, the higher the attenuation and thus
the lower the capacity. Long lengths can support G.lite or ADSL rates,
short lengths VDSL rates. The maximum capacity can be estimated using
water-pouring calculations. The strong difference in attenuation between low
frequencies and higher ones can be compensated for by an equalizer.
The DSL environment is more complex than we have described so far. In
addition to the attenuation there is the acute problem of crosstalk. At high
frequencies a significant portion of the signal energy leaks between adjacent
cables, causing one DSL modem to interfere with another. The interferer
may be located close by, as in the case of a bank of DSL modems at the
telephone office, or remotely located but transmitting to a co-located DSL
demodulator. The former case is called NEXT (Near End XTalk) and the
latter FEXT (Far End XTalk). In addition, signals such as AM broadcast
transmissions can be picked up by the subscriber line and cause narrow
bands of frequencies to be unusable. DSL modems must be able to cope
with all these types of interference.
736 COMMUNICATIONS SIGNAL PROCESSING
Multicarrier modems were proposed, but not accepted, for Vfast. A mul-
ticarrier scheme called Discrete MultiTone (DMT) has become the recog-
nized standard for G.lite and ADSL. These modems transmit a large number
of independent equally spaced carriers, each with an nPSK or QAM constel-
lation, and all with the same baud rate. This baud rate is very low compared
to the bit rate, and so each carrier has a narrow bandwidth. These narrow
bandwidth transmissions remind us of those used in the proof of the second
half of Shannons capacity theorem, and indeed the multicarrier approach is
successful partly due to its ability to approach the water-pouring limit. Fur-
thermore, we can assume that the channel attenuation and phase delay are
approximately constant over the narrow bandwidth of these transmissions,
hence equalization in the normal sense is not required. All that is needed
is a single gain to compensate for the attenuation at the carrier frequency,
and a single phase rotation to bring the constellation to the proper angle.
This Frequency EQualizer (FEQ) can be performed by a single complex
multiplication per carrier. The coefficient can be found as in exercise 6.148
The narrow bandwidth and slow baud rate make the IS1 less important;
however, if the carriers are close together we would expect InterChannel
Interference (ICI) to become a problem. ICI is removed in DMT by choos-
ing the intercarrier spacing to be precisely the baud rate. In this fashion
each carrier sits on the zeros of its neighbors sines, and the ICI is negligi-
ble. Multicarrier signals with this spacing are called Orthogonal Frequency
Division Multiplexing (OFDM) signals, since the carriers are spaced to be
orthogonal.
How do we demodulate DMT signals? The straightforward method would
be to use a bank of band-pass filters to separate the carriers, and then
downmix each to zero and slice. However, it is obvious that this bank of
filters and downmixers can be performed in parallel by using a single FFT
algorithm, making the DMT demodulator computationally efficient. Indeed,
the modulator can work the same way; after dividing the bit stream into
groups, we create complex constellation points for each of the constellations,
and then perform a single iFFT to create the signal to be transmitted!
EXERCISES
18.21.1 What is the SNR needed to achieve 56 Kb/s using every other PCM level
and assuming 3.8 KHz of bandwidth and that the consumers modem has a
16-bit linear A/D? Is this reasonable? Why is it harder to transmit 56 Kb/s
upstream?
BIBLIOGRAPHICAL NOTES 737
18.21.2 Why is the POTS splitter implemented using passive analog filters rather
than digital filters?
18.21.3 A DMT modem still has some IS1 from previous symbols. This IS1 is removed
by using a cyclic prefix. Explain. The overhead of a long cyclic prefix can be
reduced by using a Time EQualizer, which is a filter whose sole purpose is to
decrease the length of the channel impulse response. What is the connection
between the TEQ, FE&, and a regular equalizer?
18.21.4 DMT modems suffer from high Peak to Average Ratio (PAR). Explain why.
Why is this an undesirable feature? What can be done to lower the PAR?
Bibliographical Notes
The general theory and practice of digital communications systems is covered in
many texts [242, 951, and modems in particular are the subject of [144, 1991. [262]
covers real-time DSP programming (using a floating point processor) for communi-
cations, including AM, FM, SSB, PAM, QAM, and echo cancellation for full-duplex
modems.
Harry Nyquist published in 1928 a precursor to information theory [183]. Shan-
nons separation theorems appear in [237], which later appeared as a book. The
first part of the channel capacity theorem first appears in [238], an article that very
much deserves reading even today. The water-pouring criterion is due to Gallager,
and appears in his book [67].
A good modern textbook on information theory is [46], while error correcting
codes are covered in many books, e.g., [194]. This latter is an updated version of
one of the first texts on the subject. A fascinating mathematically oriented book
on topics relevant to error correcting codes is [42]. Reed and Solomon published
their code in [218]. Viterbi presented his algorithm for decoding convolution codes
in [265], but the classic overview is [63].
A dated, but still useful, reference on constellation design is [119]. Multidimen-
sional constellations are covered in [70].
Timing recovery is reviewed in [64] and a suggested original article is [76].
TCM was first presented by Ungerboeck in [263], and Wei [270] discovered how
to make it rotationally invariant, leading to the trellis code used in V.32. For TCM
in multidimensional constellations consult [271].
Since standard texts go only as far as V.32, it is worthwhile consulting the V.34
review in [117]. Tomlinson and flexible preceding is explained in [118].
The classic, but dated, reference for multicarrier modulation is [18].
Readers interested in a nontechnical introduction to DSL modems should con-
sult [82], while in-depth coverage is provided in [217, 36, 2511.
19
In this chapter we treat of one of the most intricate and fascinating signals
ever to be studied, human. speech. The reader has already been exposed
to the basic models of speech generation and perception in Chapter 11. In
this chapter we apply our knowledge of these mechanisms to the practical
problem of speech modeling.
Speech synthesis is the artificial generation of understandable, and (hope-
fully) natural-sounding speech. If coupled with a set of rules for reading text,
rules that in some languages are simple but in others quite complex, we get
text-to-speech conversion. We introduce the reader to speech modeling by
means of a naive, but functional, speech synthesis system.
Speech recognition, also called speech-to-text conversion, seems at first to
be a pattern recognition problem, but closer examination proves understand-
ing speech to be much more complex due to time warping effects. Although
a difficult task, the allure of a machine that converses with humans via natu-
ral speech is so great that much research has been and is still being devoted
to this subject. There are also many other applications-speaker verifica-
tion, emotional content extraction (voice polygraph), blind voice separation
(cocktail party effect), speech enhancement, and language identification, to
name just a few. While the list of applications is endless many of the basic
principles tend to be the same. We will focus on the deriving of features,
i.e., sets of parameters that are believed to contain the information needed
for the various tasks.
Simplistic sampling and digitizing of speech requires a high information
rate (in bits per second), meaning wide bandwidth and large storage re-
quirements. More sophisticated methods have been developed that require
a significantly lower information rate but introduce a tolerable amount of
distortion to the original signal. These methods are called speech coding
or speech compression techniques, and the main focus of this chapter is
to follow the historical development of telephone-grade speech compression
techniques that successively halved bit rates from 64 to below 8 Kb/s.
739
740 SPEECH SIGNAL PROCESSING
Figure 19.1: LPC speech model. The U/V switch selaects one of two possible excitation
signals, a pulse train created by the pitch generator, or white noise created by the noise
generator. This excitation is input to an all-pole filter.
19.1. LPC SPEECH SYNTHESIS 741
This extremely primitive model can already be used for speech synthesis
systems, and indeed was the heart of a popular chip set as early as the 1970s.
Lets assume that speech can be assumed to be approximately stationary
for at least T seconds (T is usually assumed to be in the range from 10 to
100 milliseconds). Then in order to synthesize speech, we need to supply our
model with the following information every T seconds. First, a single bit
indicating whether the speech segment is voiced or unvoiced. If the speech
is voiced we need to supply the pitch frequency as well (for convenience
we sometimes combine the U/V bit with the pitch parameter, a zero pitch
indicating unvoiced speech). Next, we need to specify the overall gain of
the filter. Finally, we need to supply any set of parameters that completely
specify the all-pole filter (e.g., pole locations, LPC coefficients, reflection
coefficients, LSP frequencies). Since there are four to five formants, we expect
the filter to have 8 to 10 complex poles.
How do we know what filter coefficients to use to make a desired sound?
What we need to do is to prepare a list of the coefficients for the various
phonemes needed. Happily this type of data is readily available. For example,
in Figure 19.2 we show a scatter plot of the first two formants for vowels,
based on the famous Peterson-Barney data.
3600
3M)o
2ooo
2ooo
IMIO
1000
#TM
Figure 19.2: First two formants from Peterson-Barney vowel data. The horizontal axis
represents the frequency of the first formant between 200 and 1250 Hz, while the vertical
axis is the frequency of the second formant, between 500 and 3500 Hz. The data consists of
each of ten vowel sounds pronounced twice by each of 76 speakers. The two letter notations
are the so-called ARPABET symbols. IY stands for the vowel in heat, IH for that in hid,
and likewise EH head, AE had, AH hut, AA hot, A0 fought, UH hood, UW hoot, ER
heard.
742 SPEECH SIGNAL PROCESSING
EXERCISES
19.1.1 The Peterson-Barney data is easily obtainable in computer-readable form.
Generate vowels according to the formant parameters and listen to the result.
Can you recognize the vowel?
19.1.2 Source code for the Klatt formant synthesizer is in the public domain. Learn
its parameters and experiment with putting phonemes together to make
words. Get the synthesizer to say digital signal processing. How natural-
sounding is it?
19.1.3 Is the LPC model valid for a flute? What model is sensible for a guitar? What
is the difference between the excitation of a guitar and that of a violin?
there is no reason to try to model speech that doesnt exist. Simple devices
that trigger on speech go under the name of VOX, for Voice Operated
X (X being a graphic abbreviation for the word switch), while the more
sophisticated techniques are now called Voice Activity Detection. Simple
VOXes may trigger just based on the appearance of energy, or may employ
NRT mechanisms, or use gross spectral features to discriminate between
speech and noise. The use of zero crossings is also popular as these can
be computed with low complexity. Most VADs utilize parameters based on
autocorrelation, and essentially perform the initial stages of a speech coder.
When the decision has been made that no voice is present, older systems
would simply not store or transfer any information, resulting in dead silence
upon decoding. The modern approach is to extract some basic statistics of
the noise (e.g., energy and bandwidth) in order to enable Comfort Noise
Generation, (CNG).
Once the VAD has decided that speech is present, determination of the
voicing (U/V) must be made; and assuming the speech is voiced the next
step will be pitch determination. Pitch tracking and voicing determination
will be treated in Section 19.5.
The finding of the filter coefficients is based on the principles of Sec-
tion 9.9, but there are a few details we need to fill in. We know how to find
LPC coefficients when there is no excitation, but here there is excitation.
For voiced speech this excitation is nonzero only during the glottal pulse,
and one strategy is to ignore it and live with the spikes of error. These spikes
reinforce the pitch information and may be of no consequence in speech com-
pression systems. In pitch synchronous systems we first identify the pitch
pulse locations, and correctly evaluate the LPC coefficients for blocks start-
ing with a pulse and ending before the next pulse. A more modern approach
is to perform two separate LPC analyses. The one we have been discussing
up to now, which models the vocal tract, is now called the short-term predic-
tor. The new one, called the long-term predictor, estimates the pitch period
and structure. It typically only has a few coefficients, but is updated at a
higher rate.
There is one final parameter we have neglected until now, the gain G.
Of course if we assume the excitation to be zero our formalism cannot be
expected to supply G. However, since G simply controls the overall volume, it
carries little information and its adjustment is not critical. In speech coding
it is typically set by requiring the energy of the predicted signal to equal the
energy in the original signal.
744 SPEECH SIGNAL PROCESSING
EXERCISES
19.2.1 Multipulse LPC uses an excitation with several pulses per pitch period. Ex-
plain how this can improve LPC quality.
19.2.2 Mixed Excitation Linear Prediction (MELP) does switch between periodic
and noise excitation, rather uses an additive combination of the two. Why
can this produce better quality speech than LPC?
19.2.3 Record some speech and display its sonogram. Compute the LPC spectrum
and find its major peaks. Overlay the peaks onto the sonogram. Can you
recognize the formants? What about the pitch?
19.2.4 Synthesize some LPC data using a certain number of LPC coefficients and
try to analyze it using a different number of coefficients. What happens? How
does the reconstruction SNR depend on the order mismatch?
19.3 Cepstrum
The LPC model is not the only framework for describing speech. Although
it is currently the basis for much of speech compression, cepstral coefficients
have proven to be superior for speech recognition and speaker identification.
The first time you hear the word cepstrum you are convinced that the
word was supposed to be spectrum and laugh at the speakers spoonerism.
However, there really is something pronounced cepstrum instead of spec-
trum, as well as a quefrency replacing frequency, and liftering displacing
filtering. Several other purposefully distorted words have been suggested
(e.g., alanysis and saphe) but have not become as popular.
To motivate the use of cepstrum in speech analysis, recall that voiced
speech can be viewed as a periodic excitation signal passed through an all-
pole filter. The excitation signal in the frequency domain is rich in harmonics,
and can be modeled as a train of equally spaced discrete lines, separated by
the pitch frequency. The amplitudes of these lines decreases rapidly with in-
creasing frequency, with between 5 and 12 dB drop per octave being typical.
The effect of the vocal tract filtering is to multiply this line spectrum by a
window that has several pronounced peaks corresponding to the formants.
Now if the spectrum is the product of the pitch train and the vocal tract
window, then the logarithm of this spectrum is the sum of the logarithm of
the pitch train and the logarithm of the vocal tract window. This logarithmic
spectrum can be considered to be the spectrum of some new signal, and since
19.3. CEPSTRUM 745
the FT is a linear operation, this new signal is the sum of two signals, one
deriving from the pitch train and one from the vocal tract filter. This new
signal, derived by logarithmically compressing the spectrum, is called the
cepstrum of the original signal. It is actually a signal in the time domain,
but since it is derived by distorting the frequency components its axis is
referred to as qzlefrency. Remember, however, that the units of quefrency
are seconds (or perhaps they should be called cesonds).
We see that the cepstrum decouples the excitation signal from the vocal
tract filter, changing a convolution into a sum. It can achieve this decou-
pling not only for speech but for any excitation signal and filter, and is thus
a general tool for deconvolution. It has therefore been applied to various
other fields in DSP, where it is sometimes referred to as homomorphic de-
convolution. This term originates in the idea that although the cepstrum is
not a linear transform of the signal (the cepstrum of a sum is not the sum
of the cepstra), it is a generalization of the idea of a linear transform (the
cepstrum of the convolution is the sum of the cepstra). Such parallels are
called homomorphisms in algebra.
The logarithmic spectrum of the excitation signal is an equally spaced
train, but the logarithmic amplitudes are much less pronounced and decrease
slowly and linearly while the lines themselves are much broader. Indeed
the logarithmic spectrum of the excitation looks much more like a sinusoid
than a train of impulses. Thus the pitch contribution is basically a line
at a well defined quefrency corresponding to the basic pitch frequency. At
lower quefrencies we find structure corresponding to the higher frequency
formants, and in many cases high-pass liftering can thus furnish both a
voiced/unvoiced indication and a pitch frequency estimate.
Up to now our discussion has been purposefully vague, mainly because
the cepstrum comes in several different flavors. One type is based on the
z transform S(Z), which being complex valued, is composed of its absolute
value R(z) and its angle 8(z). Now lets take the complex logarithm of S(z)
(equation (A.14)) and call the resulting function S(Z).
We assumed here the minimal phase value, although for some applications
it may be more useful to unwrap the phase. Now S(Z) can be considered to
be the zT of some signal sVn,this signal being the complex cepstrum of s,.
To find the complex cepstrum in practice requires computation of the izT,
a computationally arduous task; however, given the complex cepstrum the
original signal may be recovered via the zT.
746 SPEECH SIGNAL PROCESSING
The power cepstrum, or real cepstrum, is defined as the signal whose PSD
is the logarithm of the PSD of sn. The power cepstrum can be obtained as
an iFT, or for digital signals an inverse DFT
1 =
Y
Sn =-
27r -r J
log (S(w) leiwn dw
Sn = $(Sn + S*-n)
Although easier to compute, the power cepstrum doesnt take the phase of
S(w) into account, and hence does not enable unique recovery of the original
signal.
There is another variant of importance, called the LPC cepstrum. The
LPC cepstrum, like the reflection coefficients, area ratios, and LSP coeffi-
cients, is a set of coefficients ck that contains exactly the same information
as the LPC coefficients. The LPC cepstral coefficients are defined as the
coefficients of the zT expansion of the logarithm of the all-pole system func-
tion. From the definition of the LPC coefficients in equation (9.21), we see
that this can be expressed as follows:
G -k
1% (19.1)
l- c,M,1 b,rm = ck lcz
Given the LPC coefficients, the LPC cepstral coefficients can be computed
by a recursion that can be derived by series expansion of the left-hand side
(using equations (A.47) and (A.15)) and equating like terms.
co = 1ogG
cl = bl (19.2)
1 k-l
ck = bk -k x c mcmbk-m
m=l
This recursion can even be used for cI, coefficients for which k > Mby taking
bk = 0 for such k. Of course, the recursion only works when the original LPC
model was stable.
LPC cepstral coefficients derived from this recursion only represent the
true cepstrum when the signal is exactly described by an LPC model. For
real speech the LPC model is only an approximation, and hence the LPC
cepstrum deviates from the true cepstrum. In particular, for phonemes that
19.4. OTHER FEATURES 747
are not well represented by the LPC model (e.g., sounds like f, s, and sh that
are produced at the lips with the vocal tract trapping energy and creating
zeros), the LPC cepstrum bears little relationship to its namesakes. Nonethe-
less, numerous comparisons have shown the LPC cepstral coefficients to be
among the best features for both speech and speaker recognition.
If the LPC cepstral coefficients contain precisely the same information
as the LPC coefficients, how can it be that one set is superior to the other?
The difference has to do with the other mechanisms used in a recognition
system. It turns out that Euclidean distance in the space of LPC cepstral
coefficients correlates well with the Itakuru-Saito distance, a measure of how
close sounds actually sound. This relationship means that the interpretation
of closeness in LPC cepstrum space is similar to that our own hearing system
uses, a fact that aids the pattern recognition machinery.
EXERCISES
19.3.1 The signal z(t) is corrupted by a single echo to become y(t) = ~(t)+aa(t--7).
Show that the log power spectrum of y is approximately that of x with an
additional ripple. Find the parameters of this ripple.
19.3.2 Complete the proof of equation (19.2).
19.3.3 The reconstruction of a signal from its power cepstrum is not unique. When
is it correct?
19.3.4 Record some speech and plot its power cepstrum. Are the pitch and formants
easily separable?
19.3.5 Write a program to compute the LPC cepstrum. Produce artificial speech
from an exact LPC model and compute its LPC cepstrum.
are also used. It is obvious that all of these are spectral descriptions. The
extensive use of these parameters is a strong indication of our belief that
the information in speech is stored in its spectrum, more specifically in the
position of the formants.
We can test this premise by filtering some speech in such a way as to con-
siderably whiten its spectrum for some sound or sounds. For example, we can
create an inverse filter to the spectrum of a common vowel, such as the e in
the word feet. The spectrum will be completely flat when this vowel sound
is spoken, and will be considerably distorted during other vowel sounds. Yet
this inverse-E filtered speech turns out to be perfectly intelligible. Of course
a speech recognition device based on one of the aforementioned parameter
sets will utterly fail.
So where is the information if not in the spectrum? A well-known fact
regarding our senses is that they respond mainly to change and not to steady-
state phenomena. Strong odors become unnoticeable after a short while, our
eyes twitch in order to keep objects moving on our retina (animals without
the eye twitch only see moving objects) and even a relatively loud stationary
background noise seems to fade away. Although our speech generation sys-
tem is efficient at creating formants, our hearing system is mainly sensitive
to changes in these formants.
One way this effect can be taken into account in speech recognition
systems is to use derivative coefficients. For example, in addition to using
LPC cepstral coefficients as features, some systems use the so-called delta
cepstral coefficients, which capture the time variation of the cepstral coeffi-
cients. Some researchers have suggested using the delta-delta coefficients as
well, in order to capture second derivative effects.
An alternative to this empirical addition of time-variant information is to
use a set of parameters specifically built to emphasize the signals time varia-
tion. One such set of parameters is called RASTA-PLP (Relative Spectra-
Perceptual Linear Prediction). The basic PLP technique modifies the short
time spectrum by several psychophysically motivated transformations, in-
cluding resampling the spectrum into Bark segments, taking the logarithm
of the spectral amplitude and weighting the spectrum by a simulation of the
psychophysical equal-loudness curve, before fitting to an all-pole model. The
RASTA technique suppresses steady state behavior by band-pass filtering
each frequency channel, in this way removing DC and slowly varying terms.
It has been found that RASTA parameters are less sensitive to artifacts;
for example, LPC-based speech recognition systems trained on microphone-
quality speech do not work well when presented with telephone speech. The
performance of a RASTA-based system degrades much less.
19.4. OTHER FEATURES 749
changing from place to place. Yet as long as the paper is not crumpled into
a three-dimensional ball, its local dimensionality remains two. Performing
such experiments on vowel sounds has led several researchers to conclude
that three to five local features are sufficient to describe speech.
Of course this demonstration is not constructive and leaves us totally
in the dark as to how to find such a small set of features. Attempts are
being made to search for these features using learning algorithms and neural
networks, but it is too early to hazard a guess as to success and possible
impact of this line of inquiry.
EXERCISES
19.4.1 Speech has an overall spectral tilt of 5 to 12 dB per octave. Remove this tilt
(a pre-emphasis filter of the form 1 - 0.99z-1 is often used) and listen to the
speech. Is the speech intelligible? Does it sound natural?
19.4.2 If speech information really lies in the changes, why dont we differentiate
the signal and then perform the analysis?
times during which the signal is stationary would provide unacceptably large
uncertainties in the pitch determination. Hoarse and high-pitched voices are
particularly difficult in this regard.
All this said, there are many pitch tracking algorithms available. One
major class of algorithms is based on finding peaks in the empirical autocor-
relation. A typical algorithm from this class starts by low-pass filtering the
speech signal to eliminate frequency components above 800 or 900 Hz. The
pitch should correspond to a peak in the autocorrelation of this signal, but
there are still many peaks from which to choose. Choosing the largest peak
sometimes works, but may result in a multiple of the pitch or in a formant
frequency. Instead of immediately computing the autocorrelation we first
center clip (see equation (8.7)) the signal, a process that tends to flatten out
vocal tract autocorrelation peaks. The idea is that the formant periodicity
should be riding on that of the pitch, even if its consistency results in a larger
spectral peak. Accordingly, after center clipping we expect only pitch-related
phenomena to remain. Of course the exact threshold for the center clipping
must be properly set for this preprocessing to work, and various schemes
have been developed. Most schemes first determine the highest sample in
the segment and eliminate the middle third of the dynamic range. Now au-
tocorrelation lags that correspond to valid pitch periods are computed. Once
again we might naively expect the largest peak to correspond to the pitch
period, but if filtering of the original signal removed or attenuated the pitch
frequency this may not be the case. A better strategy is to look for con-
sistency in the observed autocorrelation peaks, choosing a period that has
the most energy in the peak and its multiples. This technique tends to work
even for noisy speech, but requires postprocessing to correct random errors
in isolated segments.
A variant of the autocorrelation class computes the Average Magnitude
Difference Function
AMDF(m)
=c lxn- zn+ml
n
EXERCISES
19.5.1 In order to minimize time spent in computation of autocorrelation lags, one
can replace the center clipping operation with a three-level slicing operation
that only outputs -1, 0 or +l. How does this decrease complexity? Does this
operation strongly affect the performance of the algorithm?
19.5.2 Create a signal that is the weighted sum of a few sinusoids interrupted every
now and then by short durations of white noise. You can probably easily
separate the two signal types by eye in either time or frequency domains.
Now do the same using any of the methods discussed above, or any algorithm
of your own devising.
19.5.3 Repeat the previous exercise with additive noise on the sinusoids and narrow
band noise instead of white noise. How much noise can your algorithm toler-
ate? How narrow-band can the unvoiced sections be and still be identifiable?
Can you do better by eye than your algorithm?
without error. Extending techniques that work on general bit streams to the
lossy regime is fruitless. It does not really make sense to view the speech
signal as a stream of bits and to minimize the number of bit errors in the
reconstructed stream. This is because some bits are more significant than
others-an error in the least significant bit is of much less effect than an
error in a sign bit!
It is less obvious that it is also not optimal to view the speech signal
as a stream of sample values and compress it in such a fashion as to mini-
mize the energy of error signal (reconstructed signal minus original signal).
This is because two completely different signals may sound the same since
hearing involves complex physiological and psychophysical processes (see
Section 11.4).
For example, by delaying the speech signal by two samples, we create a
new signal completely indistinguishable to the ear but with a large error
signal. The ear is insensitive to absolute time and thus would not be able
to differentiate between these two different signals. Of course simple cross
correlation would home-in on the proper delay and once corrected the error
would be zero again. But consider delaying the digital signal by half a sample
(using an appropriate interpolation technique), producing a signal with com-
pletely distinct sample values. Once again a knowledgeable signal processor
would be able to discover this subterfuge and return a very small error. Sim-
ilarly, the ear is insensitive to small changes in loudness and absolute phase.
However, the ear is also insensitive to more exotic transformations such as
small changes in pitch, formant location, and nonlinear warping of the time
axis.
Reversing our point-of-view we can say that speech-specific compression
techniques work well for two related reasons. First, speech compression tech-
niques are lossy (i.e., they strive to reproduce a signal that is similar but not
necessarily identical to the original); significantly lower information rates can
be achieved by introducing tolerable amounts of distortion. Second, once we
have abandoned the ideal of precise reconstruction of the original signal, we
can go a step further. The reconstructed signal neednt really be similar to
the original (e.g., have minimal mean square error); it should merely sound
similar. Since the ear is insensitive to small changes in phase, timing, and
pitch, much of the information in the original signal is unimportant and
neednt be encoded at all.
It was once common to differentiate between two types of speech coders.
Waveform coders exploit characteristics of the speech signal (e.g., energy
concentration at low frequencies) to encode the speech samples in fewer bits
than would be required for a completely random signal. The encoding is a
19.6. SPEECH COMPRESSION 755
toll quality) is ranked 4.0. To the uninitiated telephone speech may seem
almost the same as high-quality speech, however, this is in large part due
to the brain compensating for the degradation in quality. In fact different
phonemes may become acoustically indistinguishable after the band-pass
filtering to 4 KHz (e.g. s and f), but this fact often goes unnoticed, just as
the blind spots in our eyes do. MOS ratings from 3.5 to 4 are sometimes
called communications quality, and although lower than toll quality are
acceptable for many applications.
Usually MOS tests are performed along with calibration runs of known
MOS, but there still are consistent discrepancies between the various labo-
ratories that perform these measurements. The effort and expense required
to obtain an MOS rating for a coder are so great that objective tests that
correlate well with empirical MOS ratings have been developed. Perceptual
Speech Quality Measure (PSQM) and Perceptual Evaluation of Speech
Quality (PESQ) are two such which have been standardized by the ITU.
EXERCISES
19.6.1 Why cant general-purpose data compression techniques be lossy?
19.6.2 Assume a language with 64 different phonemes that can be spoken at the
rate of eight phonemes per second. What is the minimal bit rate required?
19.6.3 Try to compress a speech file with a general-purpose lossless data (file) com-
pression program. What compression ratio do you get?
19.6.4 Several lossy speech compression algorithms are readily available or in the
public domain (e.g., LPC-lOe, CELP, GSM full-rate). Compress a file of
speech using one or more of these compressions. Now listen to the before and
after files. Can you tell which is which? What artifacts are most noticeable
in the compressed file? What happens when you compress a file that had
been decompressed from a previous compression?
19.6.5 What happens when the input to a speech compression algorithm is not
speech? Try single tones or DTMF tones. Try music. What about babble
noise (multiple background voices)?
19.6.6 Corrupt a file of linear 16-bit speech by randomly flipping a small percentage
of the bits. What percentage is not noticed? What percentage is acceptable?
Repeat the experiment by corrupting a file of compressed speech. What can
you conclude about media for transmitting compressed speech?
19.7. PCM 757
19.7 PCM
In order to record and/or process speech digitally one needs first to acquire
it by an A/D. The digital signal obtained in this fashion is usually called
linear PCM (recall the definition of PCM from Section 2.7). Speech con-
tains significant frequency components up to about 20 KHz, and Nyquist
would thus require a 40 KHz or higher sampling rate. From experimentation
at that rate with various numbers of sample levels one can easily become
convinced that using less than 12 to 14 bits per sample noticeably degrades
the signal. Eight bits definitely delivers inferior quality, and since conven-
tional hardware works in multiples of 8-bit bytes, we usually digitize speech
using 16 bits per sample. Hence the simplistic approach to capturing speech
digitally would be to sample at 40 KHz using 16 bits per sample for a total
information rate of 640 Kb/s. A ssuming a properly designed microphone,
speaker, A/D, D/A, and filters, 640 Kb/s digital speech is indeed close to
being indistinguishable from the original.
Our first step in reducing this bit rate is to sacrifice bandwidth by low-
pass filtering the speech to 4 KHz, the bandwidth of a telephone channel.
Although 4 KHz is not high fidelity it is sufficient to carry highly intelligible
speech. At 4 KHz the Nyquist sampling rate is reduced to 8000 samples per
second, or 128 Kb/s.
From now on we will use more and more specific features of the speech
signal to further reduce the information rate. The first step exploits the
psychophysical laws of Weber and Fechner (see Section 11.2). We stated
above that 8 bits were not sufficient for proper digitizing of speech. What we
really meant is that 256 equally spaced quantization levels produces speech
of low perceived quality. Our perception of acoustic amplitude is, however,
logarithmic, with small changes at lower amplitudes more consequential than
equal changes at high amplitudes. It is thus sensible to try unevenly spaced
quantization levels, with high density of levels at low amplitudes and much
fewer levels used at high amplitudes. The optimal spacing function will be
logarithmic, as depicted in Figure 19.3 (which replaces Figure 2.25 for this
case). Using logarithmically spaced levels 8 bits is indeed adequate for toll
quality speech, and since we now use only 8000 eight-bit samples per second,
our new rate is 64 Kb/s, half that of linear PCM. In order for a speech
compression scheme to be used in a communications system the sender and
receiver, who may be using completely different equipment, must agree as
to its details. For this reason precise standards must be established that
ensure that different implementations can interoperate. The ITU has defined
a number of speech compression schemes. The G.711 standard defines two
758 SPEECH SIGNAL PROCESSING
and although it is hard to see this from the expression, its behavior is very
similar to that of p-law. By convention we take A = 87.56 and as in the
p-law case approximate the true form with 16 staircase line segments. It
is interesting that the A-law staircase has a rising segment at the origin
and thus fluctuates for near-zero inputs, while the approximated p-law has
a horizontal segment at the origin and is thus relatively constant for very
small inputs.
EXERCISES
19.7.1 Even 640 Kb/s does not capture the entire experience of listening to a speaker
in the same room, since lip motion, facial expressions, hand gestures, and
other body language are not recorded. How important is such auxiliary infor-
mation? When do you expect this information to be most relevant? Estimate
the information rates of these other signals.
19.7.2 Explain the general form of 1-1and A laws. Start with general logarithmic
compression, extend it to handle negative signal values, and finally force it
to go through the origin.
19.7.3 Test the difference between high-quality and toll-quality speech by perform-
ing a rhyme test. In a rhyme test one person speaks out-of-context words
and a second records what was heard. By using carefully chosen words, such
as lift-list, lore-more-nor, jeep-cheep, etc., you should be able to both esti-
mate the difference in accuracy between the two cases and determine which
phonemes are being confused in the toll-quality case.
19.7.4 What does p-law (equation (19.3)) re t urn for zero input? For maximal input?
When does y = Z? Plot p-law for 16-bit linear PCM, taking xmaZ = 215 =
32768, for various p from 1 to 255. What is the qualitative difference between
the small and large 1-1cases?
19.7.5 Plot the p-law (with p = 255) and A-law (with A = 87.56) responses on
the same axes. By how much do they differ? Plot them together with true
logarithmic response. How much error do they introduce? Research and plot
the 16 line segment approximations. How much further error is introduced?
760 SPEECH SIGNAL PROCESSING
N
Sn = Pi%+ (19.6)
c
i=l
19.8. DPCM, DM, AND ADPCM 761
Figure 19.4: Unquantized DPCM. The encoder predicts the next value, finds the pre-
diction error en = sn - ,sn,
- and transmits this error through the communications channel
to the receiver. The receiver, imitating the transmitter, predicts the next value based on
all the values it has recovered so far. It then corrects this prediction based on the error E~
received.
we call the predictor a linear predictor. If the predictor works well, the
prediction error
E, = sn - sn (19.7)
is both of lower energy and much whiter than the original signal sn. The
error is all we need to transmit for the receiver to be able to reconstruct the
signal, since it too can predict the next signal value based on the past values.
Of course this prediction Zn is not completely accurate, but the correction E~
is received, and the original value easily recovered by sn = s,+en. The entire
system is depicted in Figure 19.4. We see that the encoder (linear predictor)
is present in the decoder, but that there it runs as feedback, rather than
feedforward as in the encoder itself.
The simplest DPCM system is Delta Modulation (DM). Delta modula-
tion uses only a single bit to encode the error, this bit signifying whether the
true value is above or below the predicted one. If the sampling frequency is
so much higher than required that the previous value ~~-1 itself is a good
predictor of sn, delta modulation becomes the sigma-delta converter of Sec-
tion 2.11. In a more general setting a nontrivial predictor is used, but we
still encode only the sign of the prediction error. Since delta modulation
provides no option to encode zero prediction error the decoded signal tends
to oscillate up and down where the original was relatively constant. This
annoyance can be ameliorated by the use a post-jilter, which low-pass filters
the reconstructed signal.
There is a fundamental problem with the DPCM encoders we have just
described. We assur;led that the true value of the prediction error E~ is
transferred over the channel, while in fact we can only transfer a quantized
version rnQ.The very reason we perform the prediction is to save bits after
quantization. Unfortunately, this quantization may have a devastating effect
762 SPEECH SIGNAL PROCESSING
on the decoder. The problem is not just that the correction of the present
prediction is not completely accurate; the real problem is that because of
this inaccuracy the receiver never has reliable sn with which to continue
predicting the next samples. To see this, define snQas the decoders predicted
value corrected by the quantized error. In general, sz does not quite equal
sn, but we predict the next sample values based on these incorrect corrected
predictions! Due to the feedback nature of the decoders predictor the errors
start piling up and after a short time the encoder and decoder become out
of sync.
The prediction we have been using is known as open-loop prediction,
by which we mean that we perform linear prediction of the input speech.
In order to ensure that the encoder and decoder predictors stay in sync, we
really should perform linear prediction on the speech as reconstructed by the
decoder. Unfortunately, the decoder output is not available at the encoder,
and so we need to calculate it. To perform closed-loop prediction we build an
exact copy of the entire decoder into our encoder, and use its output, rather
than the input speech, as input to the predictor. This process is diagrammed
in Figure 19.5. By closing the loop in this fashion, the decoded speech is
precisely that expected, unless the channel introduces bit errors.
The international standard for 32 Kb/s toll quality digital speech is based
on Adaptive Delta-PCM (ADPCM). The adaptive is best explained by
returning to the simple case of delta modulation. We saw above that the
DM encoder compares the speech signal value with the predicted (or simply
previous) value and reports whether this prediction is too high or too low.
How does a DM decoder work? For each input bit it takes its present esti-
mate for the speech signal value and either adds or subtracts some step size
A. Assuming A is properly chosen this strategy works well for some range
of input signal frequencies; but as seen in Figure 19.6 using a single step
in .out
AL 1r
-PF-t
Figure 19.5: Closed-loop prediction. In this figure, Q stands for quantizer, IQ inverse
quantizer, PF prediction filter. Note that the encoder contains an exact replica of the
decoder and predicts the next value based on the reconstructed speech.
19.8. DPCM, DM, AND ADPCM 763
Figure 19.6: The two types of errors in nonadaptive delta modulation. We superpose
the reconstructed signal on the original. If the step size is too small the reconstructed
signal cant keep up in areas of large slope and may even completely miss peaks (as in
the higher-frequency area at the beginning of the figure). If the step size is too large the
reconstructed signal will oscillate wildly in areas where the signal is relatively constant
(as seen at the peaks of the lower-frequency area toward the end of the figure).
size cannot satisfy all frequencies. If A is too small the reconstructed signal
cannot keep up when the signal changes rapidly in one direction and may
even completely miss peaks (as in the higher-frequency area at the begin-
ning of the figure), a phenomenon called slope overload. If A is too large
the reconstructed signal will oscillate wildly when the signal is relatively
constant (as seen at the peaks of the lower-frequency area toward the end
of the figure), which is known as granular noise.
While we introduced the errors introduced by improper step size for
DM, the same phenomena occur for general DPCM. In fact the problem
is even worse. For DM the step size A is only used at the decoder, since
the encoder only needs to check the sign of the difference between the signal
value and its prediction. For general delta-PCM the step size is needed at the
encoder as well, since the difference must be quantized using levels spaced
A apart. Improper setting of the spacing between the quantization levels
causes mismatch between the digitizer and the difference signals dynamic
range, leading to improper quantization (see Section 2.9).
764 SPEECH SIGNAL PROCESSING
The solution is to adapt the step size to match the signals behavior.
In order to minimize the error we increase A when the signal is rapidly
increasing or decreasing, and we decrease it when the signal is more constant.
A simplistic way to implement this idea for DM is to use the bit stream itself
to determine whether the step size is too small or too large. A commonly
used version uses memory of the previous delta bit; if the present bit is
the same as the previous we multiply A by some constant K (K = 1.5 is
a common choice), while if the bits differ we divide by K. In addition we
constrain A to remain within some prespecified range, and so stop adapting
when it reaches its minimum or maximum value,
While efficient computationally, the above method for adapting A is
completely heuristic. A more general tactic is to set the step size for adap-
tive DPCM to be a given percentage of the signals standard deviation. In
this way A would be small for signals that do vary much, minimizing gran-
ular noise, but large for wildly varying signals, minimizing slope overload.
Were speech stationary over long times adaptation would not be needed,
but since the statistics of the speech signal vary widely as the phonemes
change, we need to continuously update our estimate of its variance. This
can be accomplished by collecting N samples of the input speech signal in
a buffer, computing the standard deviation, setting A accordingly, and only
then performing the quantization, N needs to be long enough for the vari-
ance computation to be accurate, but not so long that the signal statistics
vary appreciably over the buffer. Values of 128 (corresponding to 16 mil-
liseconds of speech at 8000 Hz) through 512 (64 milliseconds) are commonly
used.
There are two drawbacks to this method of adaptively setting the scale of
the quantizer. First, the collecting of N samples before quantization requires
introducing buffer delay; in order to avoid excessive delay we can use an
IIR filter to track the variance instead of computing it in a buffer. Second,
the decoder needs to know A, and so it must be sent as side information,
increasing the amount of data transferred. The overhead can be avoided by
having the decoder derive A, but if A is derived from the input signal, this
is not possible. The decoder could try to use the reconstructed speech to
find A, but this would not exactly match the quantization step used by the
encoder. After a while the encoder and decoder would no longer agree and
the system would break down. As you may have guessed, the solution is to
close the loop and have the encoder determine A using its internal decoder,
a technique called backward adaptation.
19.9. VECTOR QUANTIZATION 765
EXERCISES
19.8.1 Obtain a copy of the G.726 ADPCM standard and study the main block
diagrams for the encoder and decoder. Explain the function and connections
of the adaptive predictor, adaptive quantizer, and inverse adaptive quantizer.
Why is the standard so detailed?
19.8.2 Now study the expanded block diagram of the encoder. What is the purpose
of the blocks marked adaptation speed control and tone and transition
detector ?
19.8.3 How does the MIPS complexity of the G.726 encoder compare with that of
modern lower-rate encoders?
19.8.4 Show that the open-loop prediction results in large error because the quanti-
zation error is multiplied by the prediction gain. Show that with closed-loop
prediction this does not occur.
more important than higher ones; thus we can reduce the average (percep-
tual) error by placing the quantization levels closer together for small signal
values, and further apart for large values.
We will return to the perceptual importance later; for now we assume all
signal values to be equally important and just ask how to combine adaptivity
with nonequidistant quantization thresholds. Our objective is to lower the
average quantization error; and this can be accomplished by placing the
levels closer together where the signal values are more probable.
Rather than adapting quantization thresholds, we can adapt the mid-
points between these thresholds. We call these midpoints centers, and the
quantization thresholds are now midway between adjacent centers. It is then
obvious that classifying an input as belonging to the nearest center is equiv-
alent to quantizing according to these thresholds. The set of all values that
are classified as closest to a given center (i.e., that lie between the two
thresholds) is called its cluster.
The reason we prefer to set centers is that there is an easily defined
criterion that differentiates between good sets of centers and poor ones,
namely mean square error (MSE) . Accordingly, if we have observed N signal
values {~~}~=r, we want to place M centers {c,}!$r in such a way that we
minimize the mean square quantization error.
1 N
E= N c Izn - c,12
n=l
We have used here the short-hand notation cn to mean that center closest
t0 Xn-
Figure 19.7: Quantization thresholds found by the scalar quantization algorithm for
uniform and Gaussian distributed data. For both cases 1000 points were generated, and
16 centers found by running the basic scalar quantization algorithm until convergence.
EXERCISES
19.9.1 Prove the point closest to all points in a cluster is their average.
19.9.2 Generate bimodal random numbers, i.e., ones with a distribution with two
separated peaks. Determine the error for the best standard quantization. Now
run the LBG algorithm with the same number of levels and check the error
again. How much improvement did you get?
19.9.3 Generate random vectors that are distributed according to a Gaussian mix-
ture distribution. This is done as follows. Choose M cluster centers in N-
dimensional space. For each number to be generated randomly select the
cluster, and then add to it Gaussian noise (if the noise has the same variance
for all elements then the clusters will be hyperspherical). Now run the LBG
algorithm. Change the size of the codebook. How does the error decrease
with codebook size?
19.10 SBC
The next factor-of-two can be achieved by noticing that the short time spec-
trum tends to have a only a few areas with significant energy. The SubBand
Coding (SBC) technique takes advantage of this feature by dividing the
19.10. SBC 769
EXERCISES
19.10.1 Can we always decimate subbands according to their bandwidth? (Hint: Re-
call the band-pass sampling theorem.)
19.10.2 When dividing into equal-bandwidth bands, in which are more bits typically
needed, those with lower or higher frequencies? Is this consistent with what
happens with logarithmic division?
19.10.3 Will dividing the bandwidth into arbitrary bands adaptively matched to the
signal produce better compression?
770 SPEECH SIGNAL PROCESSING
times longer than those of the pitch frequency. Our assumption that the
pitch excitation could be modeled as a single pulse per pitch period and
otherwise zero has apparently been pushed beyond its limits. If we remove
the residual pitch period correlations the remaining error seems to be white
noise. Hence, trying to efficiently compress the error signal would seem to
be a useless exercise.
EXERCISES
19.11.1 You can find code for LPC-1Oe in the public domain. Encode and then decode
some recorded speech. How do you rate the quality? Can you always under-
stand what is being said? Can you identify the speaker? Are some speakers
consistently hard to understand?
19.11.2 In Residual Excited Linear Prediction (RELP) the residual is low-pass fil-
tered to about 1 KHz and then decimated to lower its bit rate. Diagram
the RELP encoder and decoder. For what bit rates do you expect RELP to
function well?
r
(y *
CB. 7 . PP LPC = (> =-PW - EC
Figure 19.8: ABS CELP encoder using short- and long-term prediction. Only the essen-
tial elements are shown; CB is the codebook, PP the pitch (short-term) predictor, LPC the
long-term predictor, PW the perceptual weighting filter, and EC the error computation.
The input is used directly to find LPC coefficients and estimate the pitch and gain. The
error is then used in ABS fashion to fine tune the pitch and gain, and choose the optimal
codebook entry.
%G> = 1 - ;+ (19.10)
where D is the pitch period. D may be found open loop, but for high quality
it should be found using analysis by synthesis. For unvoiced segments the
pitch predictor can be bypassed, sending the excitation directly to the LPC
predictor, or it can be retained and its delay set randomly. A rough block
diagram of a complete CELP encoder that uses this scheme is given in
Figure 19.8.
Adaptive codebooks reinforce the pitch period using a different method.
Rather than actually filtering the excitation, we use an effective excitation
composed of two contributions. One is simply the codebook, now called the
fixed codebook. To this is added the contribution of the adaptive codebook,
which is formed from the previous excitation by duplicating it at the pitch
period. This contribution is thus periodic with the pitch period and supplies
the needed pitch-rich input to the LPC synthesis filter.
One last trick used by many CELP encoders is post-filtering. Just as
for ADPCM, the post-filter is appended after the decoder to improve the
19.13. TELEPHONE-GRADE SPEECH CODING 775
EXERCISES
19.12.1 Explain why replacing LPC coefficient b, with yyb, with 0 < y < 1 is called
bandwidth expansion. Show that 15 Hz expansion is equivalent to y = 0.994.
19.12.2 The G.723.1 coder when operating at the 5.3 Kb/s rate uses an algebraic
codebook that is specified by 17 bits. The codewords are of length 60 but
have no more than four nonzero elements. These nonzero elements are either
all in even positions or all in odd positions. If in even positions, their indexes
modulo 8 are all either 0, 2, 4, or 6. Thus 1 bit is required to declare whether
even or odd positions are used, the four pulse positions can be encoded using
3 bits, and their signs using a single bit. Write a routine that successively
generates all the legal codewords.
19.12.3 Explain how to compute the delay of an ABS CELP coder. Take into account
the buffer, lookahead, and processing delays. What are the total delays for
G.728 (frame 20 samples, backward prediction), G.729 (frame 80 samples,
forward prediction), and G.723.1 (frame 240 samples, forward prediction)?
19.12.4 Obtain a copy of the G.729 standard and study the main block diagram.
Explain the function of each block.
19.12.5 Repeat the previous exercise with the G.723.1 standard. What is the differ-
ence between the two rates? How does G.723.1 differ from G.729?
the PSTN is growing at a rate of about 5% per year, while digital com-
munications use is growing at several hundred percent a year. The amount
of data traffic exceeded that of voice sometime during the year 2000, and
hence voice over data is rapidly becoming the more important of the two
technologies .
The history of telephone-grade speech coding is a story of rate halving.
Our theoretical rate of 128 Kb/s was never used, having been reduced to
64 Kb/s by the use of logarithmic PCM, as defined in ITU standard G.711.
So the first true rate halving resulted in 32 Kb/s and was accomplished
by ADPCM, originally designated G.721. In 1990, ADPCM at rates 40, 32,
24, and 16 Kb/ s were merged into a single standard known as G.726. At
the same time G.727 was standardized; this embedded ADPCM covers
these same rates, but is designed for use in packetized networks. It has the
advantage that the bits transmitted for the lower rates are subsets of those of
the higher rates; congestion that arises at intermediate nodes can be relieved
by discarding least significant bits without the need for negotiation between
the encoder and decoder.
Under 32 Kb/s the going gets harder. The G.726 standard defines 24 and
16 Kb/s rates as well, but at less than toll-quality. Various SBC coders were
developed for 16 Kb/s, either dividing the frequency range equally and us-
ing adaptive numbers of bits per channel, or using hierarchical wavelet-type
techniques to divide the range logarithmically. Although these techniques
were extremely robust and of relatively high perceived quality for the com-
putational complexity, no SBC system was standardized for telephone-grade
speech. In 1988, a coder, dubbed G.722, was standardized that encoded
wideband audio (7 KHz sampled at 16,000 samples per second, 14 bits per
sample) at 64 Kb/s. This coder divides the bandwidth from DC to 8 KHz
into two halves using QMFs and encodes each with ADPCM.
In the early 199Os, the ITU defined performance criteria for a 16 Kb/s
coder that could replace standard 32 Kb/s ADPCM. Such a coder was re-
quired to be of comparable quality to ADPCM, and with delay of less than
5 milliseconds (preferably less than 2 milliseconds). The coder, selected in
1992 and dubbed G.728, is a CELP with backward prediction, with LPC
order of 50. Such a high LPC order is permissible since with closed-loop
prediction the coefficients need not be transmitted. Its delay is 5 samples
(0.625 ms), but its computational complexity is considerably higher than
ADPCM, on the order of 30 MIPS.
The next breakthrough was the G.729 8 Kb/s CELP coder. This was ac-
cepted simultaneously with another somewhat different CELP-based coder
for 6.4 and 5.4 Kb/s. The latter was named G.723.1 (the notation G.723
19.13. TELEPHONE-GRADE SPEECH CODING 777
having been freed up by the original merging into G.726). Why were two
different coders needed? The G.729 specification was originally intended for
toll-quality wireless applications. G.728 was rejected for this application be-
cause of its rate and high complexity. The frame size for G.729 was set at
10 ms. and its lookahead at 5 ms. Due to the wireless channel, robustness
to various types of bit errors was required. The process of carefully evaluat-
ing the various competing technologies took several years. During that time
the urgent need arose for a low-bit-rate coder for videophone applications.
Here toll-quality was not an absolute must, and it was felt by many that
G.729 would not be ready in the alloted time. Thus an alternative selection
process, with more lax testing, was instigated. For this application it was de-
cided that a long 30 millisecond frame was acceptable, that a lower bit rate
was desirable, but that slightly lower quality could be accommodated. In
the end both G.729 and G.723.1 were accepted as standards simultaneously,
and turned out to be of similar complexity.
The G.729 coder was extremely high quality, but also required over 20
MIPS of processing power to run. For some applications, including voice
over modem, this was considered excessive. A modified coder, called G.729
Annex A, was developed that required about half the complexity, with al-
most negligible MOS reduction. This annex was adopted using the quick
standardization strategy of G.723.1. G.723.1 defined as an annex a standard
VAD and CNG mechanism, and G.729 soon followed suit with a similar
mechanism as its Annex B. More recently, G.729 has defined annexes for
additional bit rates, including a 6.4 Kb/s one.
At this point in time there is considerable overlap (and rivalry) between
the two standards families. G.723.1 is the default coder for the voice over
IP standard H.323, but G.729 is allowed as an option. G.729 is the default
for the frame relay standard FRF.11, but G.723.1 is allowed there as an
option. In retrospect it is difficult to see a real need for two different coders
with similar performance.
For even lower bit rates one must decide between MIPS and MOS. On the
low MIPS low MOS front the U.S. Department of Defense initiated an effort
in 1992 to replace LPC-1Oe with a 2.4 Kb/s encoder with quality similar to
that of the 4.8 Kb/s CELP. After comparing many alternatives, in 1997 a
draft was published based on MELP. The excitation used in this encoder
consists of a pulse train and a uniform-distributed random noise generator
filtered by time-varying FIR filters. MELPs quality is higher than that of
straight LPC-10 because it addresses the latters main weaknesses, namely
voicing determination errors and not treating partially-voiced speech.
For higher MOS but with significantly higher MIPS requirements there
778 SPEECH SIGNAL PROCESSING
EXERCISES
19.13.1 Cellular telephony networks use a different set of coders, including RPE-LTP
(GSM) and VSELP (IS-54). What are the principles behind these coders and
what are their parameters?
BIBLIOGRAPHICAL NOTES 779
Bibliographical Notes
There is a plethora of books devoted to speech signal processing. The old standard
references include [210, 2111, and of the newer generation we mention [66]. A rel-
atively up-to-date book on speech recognition is [204] while [176] is an interesting
text that emphasizes neural network techniques for speech recognition.
The first artificial speech synthesis device was created by Wolfgang von Kem-
pelen in 1791. The device had a bellows that supplied air to a reed, and a manually
manipulated resonance chamber. Unfortunately, the machine was not taken seri-
ously after von Kempelens earlier invention of a chess-playing machine had been
exposed as concealing a midget chess expert. In modern times Homer Dudley from
Bell Labs [55] was an early researcher in the field of speech production mechanisms.
Expanding on the work of Alexander Graham Bell, he analyzed the human speech
production in analogy to electronic communications systems, and built the VODER
(Voice Operation DEmonstratoR), an analog synthesizer that was demonstrated
at the San Francisco and New York Worlds Fairs. An early digital vocoder is de-
scribed in [80]. In the 198Os, Dennis Klatt presented a much improved formant
synthesizer [130, 1311.
The LPC model was introduced to speech processing by Atal [lo] in the U.S.
and Itakura [ill] in Japan. Many people were initially exposed to it in the popular
review [155] or in the chapter on LPC in [210]. The power cepstrum was introduced
in [20]; the popular DSP text [186] devotes a chapter to homomorphic processing;
and [37] is worth reading. We didnt mention that there is a nonrecursive connection
between the LPC and LPC cepstrum coefficients [239].
Distance measures, such as the Itakura-Saito distance, are the subject of (112,
113, 110, 841. The inverse-E filtering problem and RASTA-PLP are reviewed in
[102, 1011. The sinusoidal representation has an extensive literature; you should
start with [163, 2011.
For questions of speech as a dynamical system and its fractal dimension consult
[259, 156, 172, 2261. Unfortunately, there is as yet no reference that specifies for the
optimal minimal set of features.
Pitch detectors and U/V decision mechanisms are the subject of [205, 206,121].
Similar techniques for formant tracking are to be found in [164, 2301.
Once, the standard text on coding was [116], but the field has advanced tremen-
dously since then. Vector quantization is covered in a review article [85] and a text
[69], while the LBG algorithm was introduced in [149].
Postfiltering is best learnt from [35]. The old standard coders are reviewed in [23]
while the recent ones are described in [47]. For specific techniques and standards,
LPC and LPC-10: [9, 261, 1211; MELP: [170]; b asic CELP: [ll]; federal standard
1016: [122]; G.729 and its annexes: [231, 228, 229, 151; G.728: [34]; G.723.1: no
comprehensive articles; waveform interpolation: [132].
A
In this appendix we will very quickly review all the mathematical back-
ground needed for complete understanding of the text. Depending on who
you are, this chapter may be entirely superfluous, or it may be one of the
most useful chapters in the entire book. You should probably at least look
it over before starting to read Chapter 1. If most of the material in this
chapter is unfamiliar to you, then you are probably not ready to continue
reading. You should definitely consult it whenever you feel uncomfortable
with the mathematics being used in any of the chapters, and it is written to
be useful in a more general setting. Under no conditions should you read it
in its natural place, after the last chapter; if you have already finished the
book, you dont need it any more.
A.1 Numbers
Since DSP involves a lot of number crunching, we had better at least know
what a number is! The simplest type of number is the whole number or
positive integer. These are 1,2,3, . . . . You probably learned about them in
kindergarten. Kronecker (the same guy who invented the delta) once said
that the whole numbers were created by God, while all the rest are human
inventions. Indeed the whole numbers are taken as basic entities in most
of mathematics, but in axiomatic set theory their existence can actually be
derived based on even simpler axioms.
So how did people create the rest of the numbers? The basic idea is to
write equations using whatever numbers we already have, and try to solve
them. Whenever we cant solve an equation using the numbers we already
know about, we invent new ones. For instance, 1 + 1 = x: leads us to x = 2,
which is a whole number and thus no news, but when we try x + 1 = 1
we discover we need the first extension-we have to invent zero. In case
you think zero is no big deal, try writing large numbers in Hebrew numerals
(where 10,20, . . . 90 have their own symbols) or dividing Roman numerals.
781
782 WHIRLWIND EXPOSITION OF MATHEMATICS
EXERCISES
A.l.l. Prove that fi is irrational. (Hint: Assume that fi = $ and find a contra-
diction.)
A.l.2 Prove that the set of real numbers is not denumerable, and that most real
numbers are irrational.
A.1.3 Hamilton invented quarternions, which are like complex numbers but with
four real components. Why did he do this if complex numbers are sufficient?
A.2 Integers
Although most of us are used to decimal numbers, where we count from 1 to
9 before incrementing the next decimal place to the left, digital computers
prefer binary numbers. Counting from zero up in binary numbers is done as
follows.
0000,0001,0010,0011,0100,0101,0110,0111)
1000,1001,1010,1011,1100,1101,1110,1111, ...
A.2. INTEGERS 783
Each 0 or 1 is called a bit, the rightmost bit in a number is called the Least
Significant Bit (LSB), while the leftmost bit in a number (which can be
zero since we assume that a constant number of bits are used) is the Most
Significant Bit (MSB). There are several ways of extending binary numbers
to negative numbers without using a separate minus sign, the most popu-
lar of which is twos complement. The twos complement of a number with
b + 1 bits (the MSB is interpreted as the sign) is obtained by subtracting
the number from 2+l; hence addition of negative numbers is automatically
correct assuming we just discard the overflow bit. We assume that the reader
is reasonably proficient in using the integers, including the operations addi-
tion, subtraction, multiplication, division with remainder, and raising to a
power, (particularly in binary) and understands the connection between bi-
nary arithmetic and logic, and how all this facilitates the building of digital
computers.
There is another operation over the integers that we will require. We say
that two whole numbers i and j are equal modulo m
i = jmodm (A4
if when they are divided by m they give the same remainder. This oper-
ation principle can be extended to real numbers as well, and is related to
periodicity. Given an integer i, the reduction modulo m of i
imodm= j (A4
means finding the minimum whole number j to which i is equal modulo m.
Thus 15 = 8 mod 7 since 15 mod 7 = 1 and 8 mod 7 = 1.
If i divided by m leaves no remainder (i.e., i mod m = 0), we say that
m is a factor of i. A whole number is prime if it has no factors other than
itself and 1. The fundamental theorem of arithmetic states that every whole
number has a unique factorization as the product of powers of primes.
(A.31
A set is said to be finite if the number of its elements is some whole
number. A set is said to be denumerably infinite if its elements can be
placed in a list labeled by whole numbers. The interpretation is that there
are in some sense the same number of elements as there are whole numbers.
In particular the set of all integers is denumerable, since it can be listed in
the following way,
and the set of all rational numbers between 0 and 1 is denumerable, as can
be seen by the following order.
01112LPL224
9 23344 5 555
EXERCISES
A.2.1 Show that there are an infinite number of primes. (Hint: Assume that there
is a largest prime and find a contradiction.)
A.2.2 You are given two input electrical devices that perform AND, OR, and NOT
on bits. Show how to build a binary adder that inputs two 2-bit numbers and
outputs a 3-bit number. How can this be extended to b-bit numbers?
A.2.3 In ones complement notation the negative of a number is obtained by flipping
all its bits. What are the advantages and disadvantage of this method?
These series expansions are important from a theoretical point of view, but
there are more efficient computational algorithms for approximating these
numbers.
EXERCISES
A.3.1 Compare several methods for computing e and T. See exercise A.8.5 below.
A.3.2 How can you tell a rational number from an irrational one based on its binary
representation?
A.3.3 Another interesting irrational number is the golden ratio y = M x 1.618.
Show that if a line segment of length I is divided in two segmenk of lengths
a and b such that the ratio of I to a equals the ratio of a to b, then % = y.
Show that if a nonsquare rectangle has sides of length a and b such that if
a square is removed the remaining rectangle has the same proportions, then
t = y. Show that cos(g) = 3.
A.3.4 Given a decimal representation T and a tolerance E, how can the smallest a
and b such that r x f to within E be found?
x = Rz y=Sz r=IzI o= Lx
for the real part, imaginary part, absolute value (magnitude) and angle of
the complex number x.
786 WHIRLWIND EXPOSITION OF MATHEMATICS
The arctangent function tan-l(p) is usually defined only for -5 < cp <
;. For equation (A.4) we need the four-quadrant arctangent, computable
via the following algorithm:
a + tanV1(lJ/z)
if x<O
if y>O
ata+r
else
a-a-r
if a<0
ata+2n
The complex operations of addition and multiplication are simple when
addition is performed on the Cartesian form
X = x1+x2
x = Xl + 22 means
Y = Yl +y2
X = x1x2 - y1y2
2 = 2122 means
Y = XlY2 + x2y1
WN =e i2N W)
EXERCISES
A.4.1 When a complex multiplication is performed using the Cartesian forms, it
would seem that we need to perform four multiplications and two additions.
Show that this same multiplication can be performed using three multiplica-
tions and five additions.
A.4.2 Express the power of a complex number in Cartesian form.
A.4.3 Find the square roots of i in Cartesian form.
A.4.4 Give geometric interpretations for the following:
1. All complex numbers with the same magnitude
2. All complex numbers with the same real part
3. All complex numbers with the same imaginary part
4. All complex numbers with the same angle
5. All complex numbers equidistant from a given complex number
788 WHIRLWIND EXPOSITION OF MATHEMATICS
associativity: a . (b - c) = (a b) sc
l
distributivity: a (b + c) = a 4 b + a c
l l
A.5. ABSTRACT ALGEBRA 789
The real numbers are a field, as are the rationals and the complex numbers.
There are also finite fields (e.g., the binary numbers and more generally the
integers 0. . . p - 1 under modulo arithmetic).
Given a field we can define a vector space over that field. A vector space
is a set of elements called vectors; our convention is to symbolize vectors
by an underline, such as v. The elements of the field are called scalars in
this context. Between the-vectors in a vector space there is an operation of
addition; and the vectors are a commutative group under this operation. In
addition, there is a multiplication operation between a scalar and a vector
that yields a vector.
Multiplication by unity must yield the same vector
.lv=v- -
and several types of distributivity must be obeyed.
0 a(u+v)=au+a~
- - -
l (a + b)v- = uv + bv
l (ab)v- = a(bv)
nonnegativity: -u . -v 2 0
self-orthogonality: -v . -v = 0 if and only if -v = -0
commutativity: u v = --
-- l v u l
Metric spaces and linear vector spaces capture different aspects of Eu-
clidean vectors, and it is not surprising that we can define normed spaces
that are both metric and vector spaces. The norm of a vector is defined to
be 1~1 = ,/-,v a v w h ic h is easily seen to be a nonnegative number and to
fulfill all the-requirements of a metric.
EXERCISES
A.5.1 Find groups with small numbers of elements.
A.5.2 Show that true and false with or as addition and and as multiplication
form a vector space.
A.5.3 Prove formally that three-dimensional space is a vector space and a metric
space.
A.6. FUNCTIONS AND POLYNOMIALS 791
a + a0
for i + 1 to n
p + xp
a + a + sip
is usually not the best way of computing the value to be returned, since
raising to a power is computationally expensive and may introduce numerical
error. It is thus usually better to use the following algorithm, which requires
an additional memory location.
a + ao+alx
P+X
for i t 1 to n
PC-P-J
a + a+aip
a + a,
for i + n- 1 to 0
a + ax+ ai
for real coefficients, while allowing complex coefficients we can factor further.
EXERCISES
A.6.1 The derivative of a polynomial a(s) of degree n is a polynomial of degree
n - 1 given by a(a) G al + 2azx + 3u3x2 + . . . + nunxnel. What is the most
efficient method of simultaneously computing u(x) and u(z)?
A.6.2 Horners rule is not efficient for sparse polynomials which have many zero
coefficients. For example, the best way to compute p(z) = x5 is to compute
a2 + x2,u4 + 3, p(x) = qx. What is the best way of computing p(z) = IP
for general integer n?
A.7. ELEMENTARY FUNCTIONS 793
A.6.3 Show that rational functions are uniquely determined by their zeros and
poles (including multiplicities) and a single additional number.
A.6.4 We define binary polynomials as polynomials for which each power of x is
either present (i.e., its coefficient is 1) or absent (its coefficient is 0). How
many different binary polynomials are there with degree up to m? What
is the connection between these polynomials and the nonnegative integers?
The addition of two binary polynomials is defined by addition modulo 2 of
the corresponding coefficients (note that each polynomial is its own additive
inverse). To what operation of the integers does this correspond? How do you
think polynomial multiplication should be defined?
In(l) = 0
ln(a b) = In(a) + In(b)
= lna-lnb (A.12)
In(J)) = bin(a)
and one finds that ln( -1) = ir, and ln(zti) = &i$. Actually, this is only one
possible value for the complex logarithm; any multiple of 2ni is just as good.
Logarithms transform multiplication into addition since they are the
converse operation to raising to a power. The natural logarithms are loga-
rithms to base e, that is, y = lnx means that x = ev, or put in another way
794 WHIRLWIND EXPOSITION OF MATHEMATICS
x2 x3
ln(1 + 2) = 2 - 2 + 3 - . . . (A.15)
The most important alternative bases are base 10 and base 2, and it is
enough to remember In 10 x 2.3 and log2 M 0.3 to be able to mentally
convert between them. Another logarithmic relation is the decibel (dB),
being one-tenth of a Bel, which is simply the base 10 logarithm of a ratio.
Pl
r(dB) = lOlog,, pz (A.16)
Using one of the useful numbers we see that every factor of two contributes
about 3 dB to the ratio (e.g., a ratio of two to one is about 3 dB, four to
one is about 6 dB, eight to one about 9 dB, etc.). Of course a ratio of ten
to one is precisely 10 dB.
The exponential function ez is simply the irrational number e raised to
the x power. If x is not an integer, the idea of a power has to be generalized,
and this can be done by requiring the following properties:
e = 1
ea+b = ea + eb
ab
eab =
(e >
The Gaussian
G(x) = +-$9 (A.19)
7T,cT
is another function based on the exponential. This function has a maximum
at p and a width of 0, and is symmetric around ,x. The peculiar constant
is chosen so that its integral over all the argument axis is normalized to one.
O
J-0a
G(x)dx=l
EXERCISES
A.7.1 Generate three-dimensional plots of the complex exponential and the com-
plex logarithm as surfaces over the complex plane.
A.7.2 Derive the expansion (A.17) by requiring the derivative of the exponential
function to equal itself.
A.7.3 Prove the normalization of the Gaussian.
for all whole n, but there are various other symmetries as well.
796 WHIRLWIND EXPOSITION OF MATHEMATICS
sin(-x) = - sin(x)
cos(-x) = cos(x)
sin( $ + 2) = sin($ - 2) (A.22)
cos(E + 2) = -cos(5 - x)
cos(% - 2) = sin(x)
In DSP we often need the sum formulas. The fundamental ones that
we need quote are
sin(2u) = 2sinucosu
cos( 2u) = cos2 a -sin2u=2cos2u-l=l-2sin2a (A.24)
sin2(u) = -- + cos(2u)
i
cos2(u) = + + f cos(2u) (A.25)
a pair of identities that often come in handy. While not important for our
purposes, for completeness we give
tana&tanb
tan(u * b) = (A.26)
1 F tanutanb
We will also need another kind of sum formula.
Another relation derivable from the sum formulas that appears less fre-
quently in trigonometry books but is very important in DSP is
which means that summing sin and cos of the same argument with any coef-
ficients still leaves a simple sin of the same argument. The desired relations
are as follows.
a = Aces(8) b = Asin(8)
b
A =dm 6 = tan-l - (A.29)
a
On odd occasions it is useful to know other multiple angle formulas
such as
but. these complex iterative forms can be replaced by simple two step recur-
sions
(and more general formulas can be derived), but dont bother trying to
memorize these.
798 WHIRLWIND EXPOSITION OF MATHEMATICS
J7r 7Fsin(&)
-IT
cos(mt)dt = 0
Jk-lr
sin(&) sin(mt)dt = n&, (A.34)
cos(nt)
J-7r cos(mt)dt
= 7&,
as can be easily derived using the product formulas and direct integration
(see Appendix A.9).
Sometimes it is useful to expand trigonometric functions in series. The
two expansions
x3 x5 x7
sin(x) = x - 3 + gr - 7r + . . .
x;J x4 xfi
cos(x) = l-~+~-gr+... (A.35)
. . .
are important and easy to remember. For really small x you can usually get
away with the first x dependent term.
Using the trigonometric identities to simplify complex expressions is usu-
ally hard work. Its usually easier to replace real sinusoids with complex
exponentials; use the simpler math of eiZ and take the real part at the end.
Just as the trigonometric functions are circular functions in the sense
that x = cos(8), y = sin(e) trace out a circle when t9 goes from zero to 27r,
so we can define the hyperbolic functions sinh and cash that trace out a
hyperbola with its vertex at (0,l). Similarly to equation (A.8), we define
g - ,-e -8 sinh 9
sinh(8) = 2 cash(8) = ee , tanh(0) = a (A.36)
(A.39)
for real k in the range 0 2 k 5 1 and nonnegative real cp. This integral
arises in the determination of the length of an arc of an ellipse. There is a
special case of the Jacobian elliptical function, called the complete elliptical
integral
(A.40)
sir:(u) + cni(u) = 1
and that for k = 0 they are identical to the trigonometric sine and cosine.
Much less obvious is that for all k < 1 they remain periodic, but with period
4Kk, four times the complete elliptical integral. As k + 1 the elliptical sine
gets wider until at k = 1 where its period diverges, becoming equal to the
hyperbolic tangent function. As k increases from zero the elliptical cosine
at first becomes more like a triangle wave, but after k = $ it develops an
inflection, and at k = 1 it becomes ,&. We will return to the elliptical
functions in Appendix A.lO.
EXERCISES
A.8.1 Plot the circular, hyperbolic, and elliptical sines and cosines. Describe the
similarities and differences.
800 WHIRLWIND EXPOSITION OF MATHEMATICS
A.8.2 Prove:
l sinh(-x) = - sinh(x)
. cosh(-x) = cash(x)
l ( cash(x) + sinh(x)) n = cash nx + sinh nx
l sinh(z) = -isin
l cash(z) = cos(iz)
l sinh(x + 27&i) = sinh(x)
l cosh(x + 2rlci) = cash(x)
A.8.3 Prove that the derivative of sinh(x) is cash(x) and that of cash(x) is sinh(x).
A.8.4 Derive half-angle formulas for sine and cosine.
As.5 Use the half-angle formulas and the fact that sine(0) = 1, that is * -+ 1
when x + 0, to numerically calculate T. (Hint: sinc( I) + 1 when n + 00 so
n sin( E) -+ T in this same limit; start with known values for sine and cosine
when n = 4,5, or 6 and iteratively halve the argument.)
A.8.6 Prove equation (A.34).
A.8.7 Find sum and double angle formulas for the hyperbolic functions.
A.8.8 Derive expansions (A.35) and (A.38) from equation (A.17).
A.9 Analysis
We assume that the reader is familiar with the sigma notation for sums
N
c ai = ao + al + a2 + ... + aN (A.42)
i=o
and knows its basic properties:
i +bi)= Cai+Cbi
C(ai i i
c i
cai = CCai
cc aij = kc aij
i j j i
2
c ai =
cc
ail&j = 2ClZiaj + Cl.Zf
( i ) i j i<j i
A.9. ANALYSIS 801
When dealing with sums a particularly useful notation is that of the Kro-
necker delta
1 i=j
6.. =
X.3 0 i#j
{
which selects a particular term from a sum.
c a& = ak (A.44)
(A.47)
input change approaches the derivative when the input changes become
very small. It is assumed that the reader knows how to differentiate basic
functions. In particular we will need the following derivatives:
d = rip--l
2
d
zest = aeat
d
z sin@) = wcos(wt) (A.49)
d
z cos(wt) = -w sin(&)
The integral of a function is related to the area under its plot. As such
integrals can be approximated by Riemann sums
I f(t) dt x Cn fCtnlS
(A.50)
W(L) = IL X(x)dx
0
From the fundamental theorem of calculus it is clear that the density func-
tion X(x) is the derivative of the cumulative weight function W(L).
EXERCISES
A.9.1 Show that 1 + 2 + 3 + . . . n = in(n + 1) and that 1 + 3 + 5 + . . . = n2 (i.e.,
that every triangular number is a perfect square).
A.9.2 Show that 1 + $ + i + i + . . . diverges, but that 1 + 4 + a + g + . . . = 2.
A.9.3 What is the meaning of a continuous function of a complex variable? Of a
differentiable function?
A.9.4 The shortest way to get from point (0,O) to point (1,l) in the two-dimensional
plane is the straight line of length a. Another way is to go first along
the straight lines connecting the points (0,O) - (1,0) - (1, l), traversing a
path of length 2. Similarly, the paths (0,O) - (i,O) - (i,$) - (fr, 1) - (1, l),
(0,O) - (i,O) - (a,i) - ($,i) - ($,l) - (l,l), and indeed any path with
segments parallel to the axes have total path length 2. In the limit of an
infinite number of segments our path is indistinguishable from the straight
line and so we have proven that fi = 2. Whats wrong with this proof?
has the solution s(t) = A sin(wt + 4)) so sinusoids are the solutions of dif-
ferential equations of the second order.
There are many other equations that give birth to other named func-
tions. For example, Legendres differential equation
PO(t) = 1
PI(t) = t
Pz(t) = g3t2 - 1) (A.55)
Pa(t) = g5t3 - 3t)
The general form is
Pn(t) = g-&(t2 - 1)
.
showing that they are indeed polynomials of degree n. We can efficiently
compute the returned value for argument t using a recursion.
J -1
Pn (t>P, (t)dt = & b,m
an =-
2n+l
2 J s(t)Pn(t)dt
-1
l
(I-t)$-t$+n2s=0 (A.56)
which you can think of as a sneaky way of defining functions in it: = cos a.
These functions are only defined for x in the domain -1 5 x 5 1, and their
range is -1 _< Tk(x) 5 1, but they are exactly the functions defined above.
It can easily be seen from either definition that
To = 1
Tl(t) = t
but it is painful to derive (e.g., by using (A.31)) even the next few:
T2(t) = 2t2 - 1
T3(t) = 4t3 - 3t
T*(t) = 8t4 - 8t2 + 1
but the job is made manageable by a recursion that we shall derive below.
The functions TN(X) have a further interesting property. To(t), being
unity, attains its maximum absolute value for all t; Tl(t) starts at ITI (- 1) ] =
1 and ends at ]Tr(+l)] = 1; ITz(t)l = 1 at the three values t = -l,O, +l. In
general, all TN have N equally spaced zeros at positions
7r(k - $)
t = cos k = 1,2,. . . N
( N 1
and N + 1 equally spaced extrema where ITN (t) I = &l at
k = 0, 1,2,. . . N
which coupled with the explicit forms for To(t) and Tl (t) is a simple recursive
scheme that only generates polynomials. We can see from the form of the
recursion that the highest term is exactly N, and that its coefficient will be
precisely 2N-1 (at least for N > 0).
The eminent German astronomer Friedrich Wilhelm Bessel was the first
to measure distances to the stars. He was the first to notice that the brightest
star in sky, Sirius, executes tiny oscillations disclosing the existence of an
invisible partner (Sirius B was observed after his death). He also observed
irregularities in the orbit of Uranus that later led to the discovery of Neptune.
During his 1817 investigation of the gravitational three-body problem, he
derived the differential equation
t2d2s ds
-g + tz + (t2 - n2)s = 0 (A.62)
which doesnt have polynomial solutions. One set of solutions are the Bessel
functions of the first type Jn(t), which look like damped sinusoids. The first
few of these are plotted in Figure A.l.
Although we wont show this, these Bessel functions can be calculated
using the following recursions.
JO(t) = l - x2 + ---+...
x4 x6
22 2242 224262
x3 x5 x7
Jl@) = :--+--
2 224 22426 2242628 + l *
= 1 - 2(t) (A.63)
A.10. DIFFERENTIAL EQUATIONS 807
Figure A.l: Besselfunctions of the first type Jo(t),Jl(t), Jz(t), 53(t), and Jd(t)
EXERCISES
A.lO.l What differential equation do the hyperbolic functions obey?
A.10.2 Give an explicit formula for the k zeros and the k - 1 extrema of Tk.
A.10.3 Write a program to expand functions in Chebyshev polynomials. Test it by
approximating various polynomials. Expand cos(s) and tan(z) in Chebyshev
polynomials. How many terms do you need for 1% accuracy?
808 WHIRLWIND EXPOSITION OF MATHEMATICS
A.10.4 Show that all the zeros of the Chebyshev polynomials are in the interval
-15 t 5 +1.
A.10.5 How can differential equations be solved numerically?
O ii(t) dt = 1
J --00
which obviously cant be fulfilled by any normal function.
From this definition it is obvious that the integral of Diracs delta is
Heavisides step function
and conversely that the derivative of the unit step (which we would normally
say doesnt have a derivative at zero) is the impulse.
(A.68)
There are many useful integral relationships involving the delta. It can
be used to select the value of a signal at a particular time,
00
J s(t)b(t
-co
- 7)dt=S(T) (A.69)
and you dont get anything if you dont catch the singularity.
bs(t- 7)= o 1 a<r<b
J a else
(A.71)
A.12. APPROXIMATION BY POLYNOMIALS 809
As long as you only use them under integrals the following are true.
6(-t) = 6(t)
b(at) = +@I (A.72)
a
EXERCISES
A.ll.l Prove
where the sum is over all times when h(tn) = 0 but the derivative h(t,) # 0.
A.ll.4 Can you think of a use for the nth derivative of the delta?
A.11.5 Give an integral representation of Heavisides step function.
A. 12 Approximation by Polynomials
We are often interested in approximating an arbitrary but smooth continu-
ous function f(z) by some other function a(z) in some interval a 5 x 2 b.
The error of this approximation at each point in the interval
44 = f(x) - a(x)
defines the faithfulness of the approximation at a particular x. The variable
x will usually be either the time t or the frequency ~3.
810 WHIRLWIND EXPOSITION OF MATHEMATICS
but weighted sums of sinusoids and many other sets of functions can be
treated similarly. The important point is that the particular function in the
family is specified by some parameter or parameters, and that these param-
eters are themselves continuous. For polynomials of degree up to M there
are M + 1 parameters, namely the coefficients am for r-n = 0. . . M. These
parameters are continuous, and even a small change of a single coefficient
results in a different polynomial. Our job is to find the polynomial in the
family that best approximates the given function f(x).
Comparison of the overall quality of two different approximations neces-
sitates quantifying the accuracy of an approximation in the entire interval by
a single value. Two reasonable candidates come to mind. The mean square
error
1 b 1
c2 =- c2(x)dx = b - (A.75)
b-a sa
and the maximum error.
Although approximations with either low mean squared error or low maxi-
mum error are in some sense good approximations, these criteria are fun-
damentally different. Requiring small maximum error c2 guarantees that the
approximation error will be uniformly small; while with small mean squared
error, the pointwise approximation error may be small over most of the
interval but large at specific w.
We can thus define two different types of approximation problems. The
first is to find that function a(x) in a family according to the Least Mean
Squared (LMS) error criterion. The second is to find the function that has
minimal maximum error, called the minimux criterion. Since the function
a(x) is specified by its parameters in the family, both the LMS and minimax
problems reduce to finding the parameters that obey the respective criterion.
In this section we will limit ourselves to the family of polynomials of degree
M, as in equation (A.74); hence the question is simply how to find the best
M + 1 coefficients am (i.e., those coefficients that minimize either the LMS
or maximal error).
A.12. APPROXIMATION BY POLYNOMIALS 811
These two approximation types are not the only ones, but they are the
important ones when the problem is to minimize the error in an intemral.
Were we to want the best polynomial approximation in the vicinity of a single
point ~0, the best polynomial approximation would be the truncated Taylor
expansion. However, as we distance ourselves from ~0 the error increases, and
so the Taylor expansion is not an appropriate approximation over an entire
interval. Were we to want the best approximation at some finite number
of points xk for k = 1.. . K, the best approximation would be Lagranges
collocating polynomial of degree K - 1.
+ fCxd
Although the collocating polynomial has zero error at the K points, we have
no control over what happens in between these points, and in general it will
oscillate wildly.
We will first consider the LMS approximation, where we are looking
for the coefficients of (A.74) that minimize the mean squared error (A.75).
Substituting, we can explicitly write the squared error (the normalization is
irrelevant to the minimization) in terms of the coefficients a, to be found.
2
f(x) - 5 a,xm dx
m=O
c Im,lxl = Fm
I m,l IEE
b
a
xz+mdx (A.78)
Fm
JE abf(x)xmdx
These equations can be solved by any of the usual methods for solving
equations with symmetric matrices, but unfortunately often turn out to
812 WHIRLWIND EXPOSITION OF MATHEMATICS
and the normal equations are the same, but Fm and Im,l contain sums rather
than integrals. The Legendre polynomials are not orthogonal when the in-
ner product is a sum, but there are other polynomials, called the Szego
polynomials, that are.
The finding of the minimax polynomial is in general a more difficult prob-
lem, since there is no simple error expression to be differentiated. Chebyshev
proved a useful theorem, called the alternation theorem, that makes min-
imax polynomial approximation tractable. To understand the alternation
theorem, consider first the following simpler result. If a polynomial a(x) is
the minimax approximation to a function f(x) in the interval [a. . . b], and
the minimax error is emaz, then there are two points x1 and x2 in the in-
terval such that E(x~) = -cmaz and 4x2) = +emaz. Why is this true? By
the definition of emaz, the pointwise error is constrained to lie between two
parallel lines -emax 5 E(x) < +emax, and it must touch at least one of these
lines. In addition, were it not to touch the other we would be able to shift
the supposed minimax polynomial by a constant, thereby decreasing ~~~~~
What Chebyshev proved is that the pointwise error of the true minimax
polynomial touches the bounding lines many more times, alternating be-
tween the lower bound and the upper one. Once again, were it not to do so
A.12. APPROXIMATION BY POLYNOMIALS 813
there would be a way of reducing the maximum error without increasing the
degree. Therefore the minimax error is equiripple, i.e., oscillates between
lower and upper bounds touching first one and then the other.
(A.79)
where we have implicitly used the general multiple angle formula of equa-
tion (A.31). In particular, the alternation theorem still holds in terms of this
new representation in terms of trigonometric polynomials.
The Russian mathematician Evgeny Yakovlevich Remez enhanced the
practice of approximation by trigonometric polynomials, and rational func-
tions of cosines. His exchange algorithm is a practical method for finding
the coefficients in equation (A.79), based on the alternation theorem. The
idea is simple. We know that the error has M + 2 extrema and that the error
is maximal there. Were we to know the precise positions of the extrema &,
the following M + 2 equations would hold
it4
t(i) = f(ti) - C bmCOS(?X$i) = (-l)iQ for i = 1.. . M+2
m=O
814 WHIRLWIND EXPOSITION OF MATHEMATICS
and could be solved for the A4 + 1 coefficients b, and the maximal error eo.
Dont be confused; since the & are assumed to be known, Fi = f(&) and
Ci,m = cos(m&) are constants, and the equations to be solved are linear.
c
m=O
Ci,mbm - (-l)ieo = Fi
EXERCISES
A.12.1 Approximate the function f(z) = e on the interval [ - 1 5 z 5 +l] by
a polynomial of degree 4 using a Taylor expansion at x = 0, collocating
polynomials that touch the function at f z, LMS, and minimax polynomials.
Determine the maximum error for all the above methods.
A. 12.2 Give explicit formulas for the slope and zero crossing of the line that LMS
approximates N empirical data points. What are the expected errors for these
parameters?
A.12.3 How can we match y = Ae to empirical data using techniques of this
section? Does this technique truly find the minimum error?
A.12.4 Show that the normal equations for polynomial approximation become ill-
conditioned for high polynomial degree and large number of data points.
A.12.5 Find a set of digital orthogonal polynomials, p[](t), such that for ml # rn2:
CnN_op[mll(t,>p[l(tn) = 0. How can these polynomials be used for LMS
poly&omial approximation?
A. 13. PROBABILITY THEORY 815
I p(x) dx = 1
For example, if the event is the marking of a test, the marks probability
density will be approximately Gaussian, with its peak at the average mark.
The most important single piece of information about any random vari-
able is its expected value
the left form being used for discrete variables and the right form for continu-
ous ones. The terms expectation, average, and mean are also commonly
applied to this same quantity. For the simple case of N discrete equally-
probable values the expectation is precisely the arithmetic mean; for N
nonequally-probable values it is the weighted average. Even for the most
general case, if you have to make a single guess as to the value a ran-
dom variable you should probably pick its expectation. Such a guess will be
unbiased-half the time it will be too low and half the time too high.
Although the average is definitely important information, it doesnt tell
the full story; in particular we would like to know how wide the distribution
is. You may propose to compute the average deviation from the average
value,
(x- (x)) =o
but as we have just mentioned this is always zero. A better proposal is the
variance
Var= ((x- (z))) (AM)
which is always positive. If the expectation is zero then the variance is simply
(s2) , but even in general it is related to this quantity.
Since the units of variance are not those of length it is often more convenient
to define the standard deviation.
0 = 1/Var
Pk -
J xk p(x) dx
I
pk = xx:
n
P(G) (A.82)
and conversely the distribution is uniquely determined from the set of all
moments. The zeroth moment is unity by definition (normalization) and the
first moment is the expectation. From the second moment and higher we
will assume that the mean is zero; if it isnt for your distribution simply
define a new variable x - (x) . The second moment is precisely the variance.
A.13. PROBABILITY THEORY 817
The third moment divided by the standard deviation raised to the third
power is called the skew;
skew E -P3
03
it measures deviation from symmetry around zero. Normal random variables
have zero skew.
For the Gaussian distribution the fourth moment divided by the standard
deviation raised to the third power equals three; so to measure deviation
from normality we define the kurtosis as follows.
kurtosis E E - 3
04
Distributions with positive kurtosis have narrower main lobes but higher
tails than the Gaussian. The meaning of the first few moments is depicted
graphically in Figure A.2.
Frequently real-world objects have more than one characteristic; for ex-
ample, people have both height h and weight w. The obvious extension of
818 WHIRLWIND EXPOSITION OF MATHEMATICS
where the integrations are over the entire range of possible heights and
weights, p(h)dh is the percentage of people with height between h and h+dh
regardless of weight, and p(w)dw is the percentage of people with weight
between w and w + dw regardless of height. The integration over both height
and weight must give one.
Two random variables are said to be statistically independent if knowl-
edge of the value of one does not affect the knowledge of the other. For
example, we can usually assume that consecutive throws of a fair coin are
independent, and knowing what happened on the first 100 throws does not
help us to predict what will happen on the next. Two random variables
are said to be uncorrelated if their crosscorrelation (defined as the expec-
tation of their product) is zero. Statistically independent random variables
are necessarily uncorrelated, but the converse need not be true.
EXERCISES
A.13.1 Define p(BIA) to be the probability of event B occurring given that event
A occurred. Prove that the probability of both events A and B occurring
is P(A A B) = P(A)PUW; and if A and B are independent events that
PM A B) = P(A)PW
A.13.2 Prove that the probabilityof either of two events occurring is p(A V B) =
P(A) + P(B) - P(A A B), and that if A and B are mutually exclusive events
that p(A V B) = p(A) + p(B).
A.13.3 Prove Bayes theorem p(BIA) = p(AIB)p(B)/p(A) and explain how this en-
ables defining probabilities that can not be defined by our original definition.
A.13.4 Let the probability of an experiment succeeding be p. Show that the prob-
ability of exactly m successes out of n identical independent experiments is
given by the binomial distribution.
Pb-4 = ; pm (1 -p)-
0
Show that the binomial distribution approaches the normal distribution for
large n. (Hint: Use Stirlings approximation for the factorials in the binary
coefficient).
A.14. LINEAR ALGEBRA 819
A. 14 Linear Algebra
Linear algebra is the study of vectors. Vector actually means several radi-
cally different things that turn out,, almost by accident,, to be connected. If
your background is science, the word vector probably triggers the geometric
meaning, while computer scientists always think of n-tuples of numbers. The
technical mathematical meaning is more general than either of these, and
allows such entities as the set of all analog signals, or of all digital signals,
or of all periodic signals, to be vector spaces as well.
The abstract mathematical definition of vector is an element of a vector
space, a concept that we introduced in Appendix A.5. Compiling all the
requirements set forth there, a vector space must obey all of the following
rules.
Addition: For every two vectors z and -y , there is a unique vector x such
that -x = -x + -y; this addition& commutative and associative, -
Zero: There is a zero vector -0, such that -x + -0 = -x for every vector x,
Inverse: Every vector -x has an inverse vector -x- such that -x + -x- =-07
-
Multiplication: For every vector -x and number a there is a vector ax.-
In addition some vector spaces have further properties.
Inner Product: For every two vectors -x and -y , there is a unique number
a such that a = -x. -y,
Norm: For every vector x there is a unique nonnegative real number r such
- r = Oifandonlyifx=O, -
that r = 1x1; -
Metric: For every two vectors x and y, there is a unique nonnegative real
number d such that d = D~x,
-- y);d = 0 if and only if -x = 2.
From these basic definitions many interesting concepts and theorems can
be derived. We can make general linear combinations of vectors
N
c - = SlVl
sivi
- + S2V2
- + . . . + SNVN (A.83)
i=l
which must return a vector in the space. The set of all vectors that can be so
formed are called the span of -- VI, l-4, . . . VN. The span is itself a subspace of
the original space. It is not, difficult to prove this directly from the axioms.
For example, we must be able to create the zero vector, which can be done by
820 WHIRLWIND EXPOSITION OF MATHEMATICS
i=l
which by linear independence of the basis requires all the respective scalar
coefficients to be equal.
There are many different bases (for example, in two dimensions we can
take any two noncolinear vectors) but all have the same number N of vectors,
which is called the dimension of the space. The dimension may be finite
(such as for two- or three-dimensional vectors), denumerably infinite (digital
signals), or nondenumerably infinite (analog signals).
We defined two vectors to be orthogonal if their dot product is zero. A
set of three or more vectors can also be orthogonal, the requirement being
that every pair is orthogonal. If a set of unit-length vectors are orthogonal,
we call them orthonormal.
- Vi l 3 = &,j (A.84)
It is not hard to show that any finite number of orthonormal vectors are
linearly independent, and that if given a basis we can create from it an
orthonormal basis.
Given a vector and a basis how do we find the expansion coefficients?
By dotting the vector with every basis vector we obtain a set of equations,
called normal equations, that can be solved for the coefficients.
XT1 = X~V~.V~+X~V~*V~+...+XNVN*~
k*v, = xl~*~+x2~.~+...+xN~~
-- -m
-m -m
..
x*vN
-- = X~V~*VN+X~V~*VN+...+XNVN~VN
-- -- --
A.15. MATRICES 821
Now we see a useful characteristic of orthonormal bases; only for these can
we find the zth coefficient by dotting with -Vi alone.
-- (A.85)
j=l - - j=l
A.15 Matrices
A matrix is a rectangularly shaped array of numbers, and thus specification
of a particular matrix element Aij requires two indices, i specifying the
row and j specifying the column. In this book we symbolize matrices by
A, the double underline alluding to the twodimensionality of the array, just
=
as the single underline indicated that vectors are one-dimensional arrays.
When actually specifying a matrix we write it like this
11 12 13 14
21 22 23 24
31 32 33 34
41 42 43 44
this being a 4by-4 matrix with the numbers 11,12,13,14 residing on the
first row, 11,21,31,41 being in the first column, and 11,22,33,44 comprising
the diagonal.
The transpose At of a matrix is obtained by interchanging the rows
and columns At ==Aji. If A is N-by-M then At will be M by N. For
matrices with complex elemezs the corresponding?oncept is the Hermitian
transpose AH, where A: = A&
Actuallrvectors can be considered to be special cases of matrices with
either a single row or a single column. A row vector is thus a horizontal
array
( 11 12 13 14 )
and a column vector a vertical array.
11
21
31
i 41 i
822 WHIRLWIND EXPOSITION OF MATHEMATICS
Square matrices have the same number of rows as they have columns.
If a square matrix is equal to its transpose (i.e., aij = aji), then we say that
the matrix is symmetric.
( a
x
y
b
a
x
c
b
a
...
c
b
...
c ...
x y x a b ... (A.87)
. . . . . .
.. .. .. .. .. ..
I.41 2.4 . . . UN
= ulvl + u2212 + . . . UNV, (A.89)
If we place the vectors in the opposite order we obtain the outer product,
which is a N-by-N matrix.
u2
(A.90)
.
\ uN
The N-by-M zero matrix 0 is the matrix with all elements equal to
zero. It is obvious from the definkions that 0 + A = A + 0 = A. The set of
all N-by-M matrices with real elements is a?ield?verTheTeals\ith respect
to matrix addition and multiplication using this zero element.
The N-by-N square identity matrix =I is given by Iij = 6i,j
I A = =A and ==
l
.. . . 01
.
.
l
cos e sin 0
(A.91)
=Re = ( -sin8 cos0 >
824 WHIRLWIND EXPOSITION OF MATHEMATICS
It is easy to see that rotation matrices are orthogonal, and hence Rex has
the same length as x. It is also not difficult to prove that R,+p ==&R/j
= ==
(i.e., that rotations &n be performed in steps).
If we perform an orthogonal transformation R on a vector space, the
particular representation of a vector -x changes z x - = Rx,
-- but we can
think of the abstract vector itself as being unchanged. For instance, rotation
of the axes change the vector representation, but the vectors themselves have
a deeper meaning. Similarly, a matrix M that performed some operation
on vectors is changed by such changes of axes. The matrix in the new axes
that performs the same function is
M =RMR-=RMRt
= = = =
(A.92)
as can be easily seen. If the original effect of the matrix was y- = --M x then
in the new representation we have
,=!+ =RMR-lRx-R
= = =--=Y
as expected. Two matrices that are related by B = R A Rt where R is
= = =
orthogonal, are said to be similar.
There are four common tasks relating to matrices: inversion, diagonal-
ization, Cholesky decomposition, and singular value decomposition (SVD).
Inversion of A is the finding of a matrix A- such that AA-l -- = I. This is
--
closely relaterto the task of equation solzng that is discussed in=the next
section. Diagonalization of A means finding a diagonal matrix D that is
= =
similar to the original matrix.
A = RDR
= ---
---
(A.93)
where L (U) is lower (upper) diagonal with ones on the main diagonal, and
=
D is dizgoral. The singular value decomposition (SVD) of a (not necessarily
A.15. MATRICES 825
square) A is a representation
=
A=UDV (A.%)
= ---
where U and V are orthogonal (by column and by row respectively), and
D is diFgona1 %h nonnegative elements.
= There are many relationships between the above tasks. For example,
given either the diagonal, Cholesky, or SVD representations, it is simple to
invert the matrix by finding the reciprocals of the diagonal elements. Indeed
the Cholesky decomposition is the fastest, and the SVD is the numerically
safest, method for inverting a general square matrix. Numeric linear algebra
has a rich literature to which the reader is referred for further detail.
EXERCISES
A.15.1 Show that (AB)-1 = B-lA-.
A.15.2 The 2-by-2 Pauli spin matrices are defined as follows.
Show that these matrices are Hermitian and unitary. Find a: and aiaj.
A.15.3 The commutator, defined as [A, B] = A B - A B , can be nonzero since matrix
= - -- --
multiplication neednt be commzative. Find the commutators for the Pauli
matrices. Define the anticommutator as the above but with a plus sign. Show
that the Pauli matrices anticommute.
A.15.4 Find the Crout (LU) and Cholesky (LDU) decompositions of
by setting it equal to
and to
multiplying out and solving the equations. How many operations are needed?
Why is the Cholesky method better? Now solve the equations with right-hand
side (2,8,20).
826 WHIRLWIND EXPOSITION OF MATHEMATICS
Ax = b (A.96)
-
where A is a known N * N matrix, b is a known N-dimensional vector, and
x is thFN-dimensional vector we want, to find. Writing this out in full,
which means
b2 - h$&
A21331 + A2252 = b2 x2 =
a22
(A + ---
BCD)-1 = A-l - A-ll3 DA-l B + C-l)- DA- (A. 100)
= --- = = = ( --
-- = = --
--
828 WHIRLWIND EXPOSITION OF MATHEMATICS
Lets prove this last lemma by multiplying the supposed inverse by the
matrix,
(A + BCD)-(A + BCD) =
= --- ---
A-1 - A-1B ;DA-lB + c-)-DA-) (A + ---
BCD) =
I;( A&;- -- --
@B=(D;-B + C-l)- ; (I + &CD) =
=I+;-l,(C-i-),=
--- -- = = == = ---
= = == ==
where
x= =
=
EXERCISES
A.16.1 Assume that B is an approximation to A-, with error R = I - BA. Show
that A- = (I=+ R + R2 + R3 + . . .) B an; that this can bTuse=d tozzatively
= = =
imprge the iivege.
A.16.2 You know that x and y obey the equation x + 3y = 8 and determine numeri-
cally that they also obey 2x+6.OOOOly = 8.00001. What are x and y? Suppose
that the numerically determined equation is 2x + 5.99999y = 8.00002. What
are x and y now? Explain the discrepancy.
Bibliography
829
830 BIBLIOGRAPHY
[27] R.N. Bracewell. The Hartley Transform. Oxford U Press, New York,
NY, 1986.
[28] E.O. Brigham. The Fast Fourier Transform and its Applications.
Prentice Hall, Englewood Cliffs, NJ, 1988.
[29] C.S. Burrus. Index mappings for multidimensional formulation of the
DFT and convolution. IEEE Trans. ASSP, ASSP-25(3):239-242, June
1977.
[30] C.S. Burrus and P.W. Eschenbacher. An in-place, in-order prime factor
FFT algorithm. IEEE Trans. ASSP, ASSP-29:806-817, August 1981.
[31] C.S. Burrus and T.W. Parks. DFT, FFT and Convolution Algorithms.
Wiley, New York, NY, 1987.
[32] W.H. Calvin and G.A. Ojemann. Inside the Brain: An Enthralling
Account of the Structure and Workings of the Human Brain. New
American Library, New York, NY, 1980.
[33] J. Celko. Partitions. Dr. Dobbs Journal, pp. 116-117,140-141, Novem-
ber 1994.
[34] J-H. Chen, R.V. Cox, Y-C. Lin, N. Jayant, and M.J. Melchner. A low-
delay CELP coder for the CCITT 16 Kb/s speech coding standard.
IEEE Trans. Selected Areas Comm., 10(5):830-849, June 1992.
[35] J-H. Chen and A. Gersho. Adaptive postfiltering for quality enhance-
ment of coded speech. IEEE Trans. Speech and Audio Processing,
3(1):59-71, January 1995.
[36] W.Y. Chen. DSL Simulation Techniques and Standards Development
for Digital Subscriber Line Systems. MacMillan Technical Publishing,
Indianapolis, IN, 1998.
[37] D.G. Childers, D.P. Skinner, and R.C. Kemerait. The cepstrum: A
guide to processing. Proc. IEEE, 65(10):1428-1442, October 1977.
[38] D.S. Cochran. Algorithms and accuracy in the HP-35. Hewlett-
Packard Journal, pp. 10-11, June 1972.
[39] L. Cohen. Time-frequency distributions - a review. Proc. IEEE,
77(7):941-981, July 1989.
[40] Digital Signal Processing Committee, editor. Selected Papers in Digital
Signal Processing, II. IEEE Press, New York, NY, 1976.
[41] Digital Signal Processing Committee, editor. Programs for Digital
Signal Processing. IEEE Press, New York, NY, 1979.
832 BIBLIOGRAPHY
[42] J.H. Conway and N.J.A. Sloane. Sphere Packings, Lattices and
Groups. Springer-Verlag, New York, NY, third edition, 1998.
[43] J. W. Cooley. How the FFT gained acceptance. IEEE Signal Processing
Magazine, pp. 10-13, January 1992.
[44] J.W. Cooley, P.A.W. Lewis, and P.D. Welch. Historical notes on
the fast Fourier transform. IEEE Trans. Audio Electroacoust., AU-
15(2):76-79, June 1967.
[45] J.W. Cooley and J.W. Tukey. An algorithm for the machine cal-
culation of complex Fourier series. Mathematics of Computation,
19(90):297-301, April 1965.
[46] T.M. Cover and J.A. Thomas. Elements of Information Theory. Wiley,
New York, NY, 1991.
[47] R.V. Cox. Three new speech coders from the ITU cover a range of
applications. IEEE Comm. Magazine, pp. 40-47, September 1997.
[48] R.E. Crochiere and L.R. Rabiner. Interpolation and decimation of
digital signals-a tutorial review. Proc. IEEE, 69(3):300-331, March
1981.
[49] A.V. Dandawate and G.B. Giannakis. A triple cross-correlation ap-
proach for enhancing noisy signals. In Proc. Workshop on Higher
Order Signal Analysis (Vail, CO), volume II, pp. 212-216, June 1989.
[50] W.N. Dember and J.S. Warm. Introduction to Psychophysics. Holt,
New York, NY, 1981.
[51] A.M. Despain. Fourier transform computers using CORDIC itera-
tions. IEEE Trans. on Computers, C-23:993-1001, October 1974.
[52] A.M. Despain. Very fast Fourier transform algorithms for hardware
implementation. IEEE Trans. on Computers, C-28:333-341, May
1979.
[53] B.S. Dewitt and N. Graham, editors. The Many Worlds Interpretation
of Quantum Mechanics. Princeton University Press, Princeton, NJ,
1973.
[54] H. Dudley. The Vocoder. Bell Labs Record, 18(4):122-126, December
1939.
[55] H. Dudley. The carrier nature of speech. Bell Systems Technical J.,
19(4):495-515, October 1940.
BIBLIOGRAPHY 833
[115] L.B. Jackson. Roundoff noise analysis for fixed-point digital filters
realized in cascade or parallel form. IEEE lkzns. Audio Electroacoust.,
AU-18:107-122, June 1970.
[116] N.S. Jayant and P. Noll. Digital Coding of Waveforms. Prentice Hall,
Englewood Cliffs, NJ, 1984.
[117] G.D. Forney Jr., L. Brown, M.V. Eyuboglu, and J.L. Moran III. The
V.34 high-speed modem standard. IEEE Comm, Magazine, pp. 28-33,
December 1996.
[118] G.D. Forney Jr. and M.V. Eyuboglu. Combined equalization and cod-
ing using preceding. IEEE Comm. Magazine, pp. 25-33, December
1991.
[119] G.D. Forney Jr., R.G. Gallager, G.R. Lang, F.M. Longstaff, and S.U.
Qureshi. Efficient modulation for band-limited channels. IEEE Trans.
Selected Areas Comm., SAC-2(5):632-646, September 1984.
[120] J.N. Mitchell Jr. Computer multiplication and division using binary
logarithm. IEEE Trans. Electr. Camp., EC-11:512-517, 1962.
[121] J.P. Campbell Jr. and T.E. Tremain. Voiced unvoiced classification of
speech with application to the U.S. government LPC-1OE algorithm.
In Proc. ICASSP-86, pp. 473-476, 1986.
[122] J.P. Campbell Jr., T.E. Tremain, and V.C. Welch. The federal stan-
dard 1016 4800 bps CELP voice coder. Digital Signal Processing (Aca-
demic Press), l(3): 145-155, 1991.
[123] S.L. Marple Jr. Digital Spectral Analysis with Applications. Prentice
Hall, Englewood Cliffs, NJ, 1987.
[124] T.G. Stockham Jr. High speed convolution and correlation. In 1966
Spring Joint Conf. AFIPS, volume 28, pp. 229-233, 1966.
[125] E.I. Jury. Theory and Application of the z-Transform Method. Krieger
Publishing Co., Huntington, NY, 1973.
[126] N. Kalouptsidis. Signal Processing Systems-Theory and Design. Wi-
ley, New York, NY, 1997.
[127] S.M. Kay. Modern Spectral Estimation: Theory and Application. Pren-
tice Hall, Englewood Cliffs, NJ, 1988.
[128] S.M. Kay and S.L. Marple Jr. Spectrum analysis-a modern perspec-
tive. Proc. IEEE, 69:1380-1419, November 1981.
838 BIBLIOGRAPHY
[129] N.Y.S. Kiang and W.T. Peake. Physics and physiology of hearing. In
Stevens Handbook of Experimental Psychology, volume 1, pp. 227-326,
New York, NY, 1988.
[130] D.H. Klatt. Software for a cascade/parallel formant synthesizer. J.
Acoust. Sot. Amer., 67(3):971-995, March 1980.
[131] D.H. Klatt. Review of text-to-speech conversion for English. J. Acoust.
Sot. Amer., 82(3):737-797, September 1987.
[132] W.B. Kleijn. Encoding speech using prototype waveforms. IEEE
Trans. Speechand Audio Processing, 1(4):386-399, 1993.
[133] D.E. Knuth. The T&Xbook.Addison-Wesley, Reading, MA, 1984.
[134] D.E. Knuth. The METRFONTbook. Addison-Wesley, Reading, MA,
1986.
[135] D.E. Knuth. Fundamental Algorithms, volume 1 of The Art of Com-
puter Programming. Addison-Wesley, Reading, MA, third edition,
1997.
[136] D.E. Knuth. Seminumerical Algorithms, volume 2 of The Art of Com-
puter Programming. Addison-Wesley, Reading, MA, third edition,
1997.
[137] D.P. Kolba and T.W. Parks. A prime factor FFT algorithm using high
speed convolution. IEEE Trans. ASSP, ASSP-25(4):281-294, August
1977.
[138] I. Koren and 0. Zinaty. Evaluating elementary functions in a numer-
ical coprocessor based on rational approximations. IEEE Trans. on
Computers, C-39:1030-1037, August 1990.
[139] L. Lamport. M&X: A Documentation Preparation System, Users
Guide and Reference Manual. Addison-Wesley, Reading, MA, 1994.
[1401 P. Lapsley, J. B ier, A. Shoham, and E.A. Lee. DSP Processor Funda-
mentals. IEEE Press, Piscataway, NJ, 1997.
[141] P. Lapsley and G. Blalock. How to estimate DSP processor perfor-
mance. IEEE Spectrum, 33(7):74-78, July 1996.
[142] E.A. Lee. Programmable DSP architectures (I). IEEE ASSP Maga-
zine, pp. 4-19, October 1988.
[143] E.A. Lee. Programmable DSP architectures (II). IEEE ASSP Maga-
zine, pp. 4-14, October 1989.
BIBLIOGRAPHY 839
[188] S.J. 0 r f amd is. Optimum Signal Processing. McGraw-Hill, New York,
NY, second edition, 1988.
[189] A. Papoulis. Signals and systems. McGraw-Hill, New York, NY, 1977.
[190] A. Papoulis. Probability, Random Variables, and Stochastic Processes.
McGraw-Hill, New York, NY, third edition, 1991.
[191] T.W. P ar k s and C.S. Burrus. Digital Filter Design. Wiley, New York,
NY, 1987.
[192] T.W. P ar k s and J.H. McClellan. Chebyshev approximation for nonre-
cursive digital filters with linear phase. IEEE Trans. Circuit Theory,
CT-19:189-194, March 1972.
[193] G.E. Peterson and H.L. Barney. Control methods used in a study of
the vowels. J. Acoust. Sot. Amer., 24(2):175-184, March 1952.
[194] W.W. Peterson and E.J. Weldon Jr. Error Correcting Codes. MIT
Press, Cambridge, MA, second edition, 1972.
[195] J.O. Plc kl es. An Introduction to the Physiology of Hearing. Academic
Press, London, UK, 1982.
[196] V.F. Pisarenko. The retrieval of harmonics from a covariance function.
Geophysical Journal of the Royal Astronomical Society, 33:347-366,
1973.
[197] S.R. Powell and P.M. Chau. A technique for realizing linear phase IIR
filters. IEEE Trans. Signal Processing, 39( 11) ~2425-2435, November
1991.
[198] W.H. Press, B.P. Flannery, S.A. Teukolsky, and W.T. Vetterling. NU-
MERICAL RECIPES in C, The Art of Scientific Computing. Cambridge
University Press, Cambridge, U.K., second edition, 1992.
[199] J.G. Proakis. Digital Communications. McGraw-Hill, New York, NY,
second edition, 1989.
[200] J.G. P roa k is and D.G. Manolakis. Digital Signal Processing: Princi-
ples, Algorithms and Applications. Prentice Hall, Englewood Cliffs,
NJ, third edition, 1996.
[201] T.F. Quatieri and R.J. McAulay. Speech transformations based on a
sinusoidal representation. IEEE Trans. ASSP, ASSP-34(6):1449-1464,
December 1986.
[202] S. Qureshi. Adaptive equalization. IEEE Comm. Magazine, pp. 9-16,
March 1982.
BIBLIOGRAPHY 843
WI G. Schroeder and M.H. Sherif. The road to G.729: ITU &kb/s speech
coding algorithm with wireline quality. IEEE Comm. Magazine, pp.
48-54, September 1997.
WI M.R. Schroeder, B.S. Atal, and J.L. Hall. Optimizing digital speech
coders by exploiting masking properties of the human ear. J. Acoust.
Sot. Amer., 66(6):1647-1652, December 1979.
WI A. Schuster. On the investigation of hidden periodicities with appli-
cations to a supposed 26 days period of meterological phenomenon.
Terr. Magn., 3:13-41, 1898.
WI H.G. Schuster. Deterministic Chaos. VHC Publishers, 1988.
WI 0. Shalvi and E. Weinstein. New criteria for blind deconvolution of
nonminimum phase systems (channels). IEEE 13-ans. Inform. Theory,
IT-36(4):312-321, March 1990.
WI 0. Shalvi and E. Weinstein. Super-exponential methods for blind
deconvolution. IEEE Truns. Inform. Theory, IT-39(4):504-519, March
1993.
WI C.E. Shannon. A mathematical theory of communication. Bell Sys-
tems Technical J., 27(3):379-423,623-656, July 1948.
WI C.E. Shannon. Communication in the presence of noise. Proc. IRE,
37:10-21, January 1949.
WI M.R. Shroeder. Direct (nonrecursive) relations between cepstrum and
predictor coefficients. IEEE T+ans. ASSP, ASSP-29( 1):297-301, April
1981.
PO1R. Singleton. An algorithm for computing the mixed radix fast Fourier
transform. IEEE nuns. Audio Electroacoust., AU-17:93-103, June
1969.
WI S.K. Mitra SK and J.F. Kaiser, editors. Handbook for Digital Signal
Processing. Wiley, New York, NY, 1993.
PQI B. Sklar. Digital Communications, Fundamentals and Applications.
Prentice Hall, Englewood Cliffs, NJ, 1988.
WI M.I. Skolnik. Introduction to Radar Systems. McGraw-Hill, New York,
NY, second edition, 1980.
WI M.I. Skolnik. Fifty years of radar. Proc. IEEE, 73(2):182-197, Febru-
ary 1985.
WI M.I. Skolnik. Radar Handbook. McGraw-Hill, New York, NY, second
edition, 1990.
846 BIBLIOGRAPHY
[246] W.W. Smith and J.M. Smith. Handbook of Real-Time Fast Fourier
Transforms. IEEE Press, New York, NY, 1995.
[247] H.V. Sorensen, M.T. Heideman, and C.S. Burrus. On computing the
split radix FFT. IEEE nuns. ASSP, ASSP-34:152-156, June 1986.
12481H.V. Sorensen, D.L. Jones, M.T. Heideman, and C.S. Burrus. Real-
valued fast Fourier transform algorithms. IEEE nuns. ASSP, ASSP-
35:849-863, June 1987.
[249] T. Springer. Sliding FFT computes frequency spectra in real time,
EDN, 29:161-170, September 1988.
[250] M.D. Srinath and P.K. Rajasekaran. An Introduction To Statistical
Signal Processing with Applications. Wiley, New York, NY, 1979.
[251] T. Star, J.M. Cioffi, and P.J. Silverman. Understanding Digital Sub-
scriber Line Technology. Prentice Hall, Upper Saddle River, NJ, 1999.
[252] K. Steiglitz. A Digital Signal Processing Primer. Addison- Wesley,
Menlo Park, CA, 1996.
[253] K.N. Stevens. Acoustic Phonetics. MIT Press, Cambridge, MA, 1998.
[254] S.S. Stevens, J. Volkmann, and E.B. Newman. A scale for the mea-
surement of the psychological magnitude pitch. J. Acoust. Sot. Amer.,
8:185-190, January 1937.
[255] P. Stoica and A. Nehorai. Study of the statistical performance of the
Pisarenkos harmonic decomposition. IEE Proc. F, 135(2):161-168,
April 1988.
[256] Y. T ad okoro and T. Higuchi. Discrete Fourier transform computation
via the Walsh transform. IEEE Trans. ASSP, ASSP-26(3):236-240,
June 1978.
[257] Y. Tad ok oro and T. Higuchi. Conversion factors from Walsh coeffi-
cients to Fourier coefficients. IEEE Trans. ASSP, ASSP-31(1):231-
232, February 1983.
[258] H.D. Tagare and R.J.P. de Figueiredo. Order filters. Proc. IEEE,
73(1):163-165, January 1985.
[259] N. Tishby. A dynamical systems approach to speech processing. In
Proc. ICASSP-90 (Albuquerque, NM), pp. 365-368, April 1990.
[260] M. Tomlinson. New automatic equalizer employing modulo arithmetic.
Electronic Letters, 7:138-139, March 1971.
BIBLIOGRAPHY 847
849
850 INDEX