0:23

We will decompose sounds into these two parts.

The sinusoidal or harmonic one and the residual or

ideally a stochastical one if this residual is the stochastic signal.

So in this lecture,

we will be combining all the models we have been talking about until now.

We'll put together the sinusoidal or

harmonic modelling with the idea of residual component.

For that we will need to talk about the subtraction of the sinusoidal,

the harmonics in order to obtain the residual.

And we will talk about a system that puts these together into the harmonic

plus residual system.

Then we will introduce the stochastic model and

we'll put together the sinusoidal and or

harmonic models with the stochastic ones for the residual.

So in order to do that, we'll need to talk about how to model

the residual as in a stochastic component.

And finally we will make an example of this system that combines

the harmonic plus the stochastic analysis into an analysis synthesis system.

3:28

So let's show an example exactly how these will work.

This is a one frame of a sound and here we can show the different

steps involved these harmonic plus a residual analysis.

On the left top, we see our window frame of a flute sound, okay?

So it's just few periods of a flute sound.

And then below that, we see the harmonic analysis that we do from the spectrum.

So we do the spectral analysis the peaks, and we select the peaks with these

blue crosses that are really the harmonics of that particular sound.

And below it we see the actual phase of the spectrum with the crosses and

the phase of these harmonics, okay?

And then what we do is we have to synthesize these harmonics.

[COUGH] And this is what we see on the right side with the light

4:36

red and light cyan color.

So the light red is the synthesized harmonics of

the sound of that particular frame.

Of course this is a different FFT size,

the shapes of these lobes is different because the window is different.

This is a window using the synthesis.

So this is the synthesized spectrum and

then we have to subtract these spectrum from the original spectrum.

Strictly speaking, we don't subtract it from the spectrum on the left,

we subtract this from another generated spectrum that has the same parameters so

that we can subtract the two of the same size and the same window size.

And then it will subtract this synthesize sinusoids or

harmonics from the original one.

We get this dark red and dark cyan color.

Okay, and this is the residual spectrum in magnitude and phase representation.

And if we take the inverse of that, we see the residual signal in the tandem and

that's what we see on the top-right plot.

In which we see the original flute sound of course with

the right windowing and the right size that we have in the synthesis.

And we see the residual signal, this dark blue one.

And again, this is not just an error signal,

this in fact is a relevant component,

it's a relevant part of the sound that we want to recover.

So the whole system, if we put together all this analysis in a frame by

frame type of thing and put it together into a whole analysis synthesis system.

We get this block diagram in which we start from the signal x[n],

then we window it, we compute FFT, obtaining the magnitude and

phase spectrum, we detect the peaks.

Hold up those peaks, we find the fundamental frequency, and

once we have this fundamental frequency we can identify the harmonics of that sound.

And we can synthesize those harmonics with the window.

Okay, so we have another spectrum,

Yh, that can be subtracted from the original signal.

But in order to do that,

we need to recompute another spectrum of the original with a window and

a size that is comparable with the size that we use in the synthesis.

So we will choose a window size that normally will be 512 samples,

we'll use a window so that the shape of these X[k]

that we now compute can be easily subtracted From Yh.

So it's just a complex subtraction and we get capital Xr,

which is our residual spectrum, okay?

And then this residual spectrum can be added to the harmonic spectrum.

And then, we can compute the inverse FFT and

do the Overlap-add iterating over the whole sum.

We can see an example of the analysis of a particular sound using the harmonic

class residual model.

So here, we took the flute sound that we have heard before, and

so on top, we see the spectrogram of these flute sound and

superpose we see the harmonics that have been obtained.

So let's listen to these harmonic synthesis.

[SOUND] Okay, and then these harmonics are subtracted

from this background spectrogram and we obtain these

lower sort of a plot which are the spectrogram of the residual component.

So let's just now listen to this residual that has been obtained.

[NOISE] Okay, it's soft but it's clearly very relevant.

It's basically the breath noise of the instrument which is

an important part of the characteristics of the sound.

But this residual component is a complete sound.

9:45

So very similar to what we saw before.

So we have the signal to be the sum of

sinusoids plus the stochastic signal.

Now this is stochastic signal or stochastic component is not just

the subtraction of the sinusoids minus the original signal but

it's actually the result of a modeling approach.

So, below here, we see the equation of the modeling of this stochastic component.

The stochastic component is the result of filtering wide noise with the impulse

response of the approximation of these residual signal.

So we have the impulse response of every frame of this residual signal,

and we obtain this impulse response that approximates a spectral shape of that.

So in fact, it's much better to visualize this model in the frequency domain.

And so here we see on the top, the equation of the sum of the sinusoid,

the sum of the analysis windows plus the spectrum of the stochastic component.

And now this stochastic component is this idea of

a filter white noise but in the frequency domain,

is the product of the approximation of the absolute

value of the residual signal multiplied by e to the j and

the phase of the random, the white noise set of random numbers, okay?

So, the magnitude spectrum is the approximation of the residual and

the phase spectrum is the white noise.

Basically, the phase spectrum of the white noise, this is the concept of

the stochastic approximation that we saw in the previous lecture, okay?

So, with these, we can actually see how in a single spectrum,

we actually perform this stochastic approximation of the residual.

So we start on the top with the spectrum of a signal, the harmonics.

And then below the light red is

the synthesized spectrum, the mYh.

And then this is subtracted from the original spectrum,

again it's a spectrum that will have to be recomputed.

And then we obtained the next curve which is

the mXr which is the residual spectrum, okay?

And this residual spectrum can be approximated with a smooth

curve which is the mYst, which is this sort of

line approximation of this residual.

And this is going to be our stochastic model.

12:54

So, we can put it together into an analysis synthesis system and

it's very similar to what we saw before.

So we start from the signal.

We compute FFT, we find the peaks, we find the harmonics,

we synthesize them in the frequency domain and we subtract them from

another spectrum of the original signal recomputed to be able to subtract it.

And then what is new in this model is the stochastic approximation of the residual.

So we take this residual spectrum, we run it through the stochastic approximation,

approximation module and then we can synthesize.

And we can synthesize the stochastical by, basically,

the idea is filtering white noise, but in the implementation is basically taking

the phases of random numbers and applying

14:20

So let's now see an example of a complete analysis synthesis of a particular sound.

So we're taking this saxophone sound, let's listen to that.

[MUSIC]

Okay, and then below it we have the two representations that we have obtain

the harmonics and the stochastic component,

the spectragram of the stochastic component.

Let's listen to the harmonics.

[MUSIC]

Now we may not appreciate what is missing but

when we listen to this stochastic approximation, [NOISE] we have

to make it a little bit louder in order to actually listen what is going on.

Well, with these two components,

we basically have analyzed and modeled the original signal and

we can put them together and generate this synthesized sound,

[MUSIC]

That captures most of what is perceptually relevant in the sound.

So, for these topics that I discuss this lecture,

there is not that many references in terms of tutorial or

sort of more introductory material.

But, there is quite a bit of articles that have been proposing

different strategies to analyze sinusoids,

obtain residuals, approximate the residuals, etc.

So in this link that I put here on the website of the MTG,

I have kept some articles, well quite a bit of articles

that have been published related to these issues.

So feel free to go there and you can sort of find those articles.

And that's all basically in this lecture.

We have covered the most advanced models that we will be presenting in this course.

We basically combine all the previous models,

developing a variety of analysis and

synthesis techniques that can be applied to many sounds and for many applications.

In the next lecture,

we'll focus on how these models can be used to transform sounds.

So I think we're going to start having fun in doing some interesting new sounds.

So I hope to see you then, bye bye.