In this course you will learn about audio signal processing methodologies that are specific for music and of use in real applications. We focus on the spectral processing techniques of relevance for the description and transformation of sounds, developing the basic theoretical and practical knowledge with which to analyze, synthesize, transform and describe audio signals in the context of music applications. The course is based on open software and content. The demonstrations and programming exercises are done using Python under Ubuntu, and the references and materials for the course come from open online repositories. We are also distributing with open licenses the software and materials developed for the course.
STFT
Loading...
From the course by Universitat Pompeu Fabra of Barcelona
Audio Signal Processing for Music Applications
196 ratings
Explore Related Videos
At Coursera, you will find the best lectures in the world. Here are some of our personalized recommendations for you
From the lesson
Short-time Fourier transform
STFT equation; analysis window; FFT size and hop size; time-frequency compromise; inverse STFT. Demonstration of tools to compute the spectrogram of a sound and on how to analyze a sound using them. Implementation of the windowing of sounds using Python and presentation of the STFT functions from the sms-tools package, explaining how to use them.
Meet the Instructors
Xavier SerraFull Professor
Dept. of Information and Communication Technologies, UPF
Prof Julius O Smith, IIIProfessor of Music and (by courtesy) Electrical Engineering
CCRMA
Welcome to the course on Audio Single Processing for music applications.
This week is the week of the Short Time Free Transform.
And in the previous programming class, we talked about windows.
We talked about how to get a window,
how to display it, how to compute it's spectrum.
And how to visualize the kind of things that we talked about in the theory class
related to windows.
And now, in this second programming lecture,
I want to build the whole short time for your transform system and
talk about the implementation we have in the semestral's package.
So we will basically, implementing the whole STFT equation shown here.
So basically, we'll be iterating over a sound x and
we will be computing at every frame, so the frame number is l.
So we'll be computing frames of the sound at each
frame location we will be multiplying the fragment of sound with the window, w.
And therefore, at the end we'll have a sequence of spectra.
A sequence of in this case, complex spectra.
And we see it as a block diagram so we start from the signal, x of n,
we multiply one fragment with our window so we get a windowed signal.
That's what we compute the FFT of and then, of course,
we compute the magnitude and phase representation of the spectrum.
And then, we can undo the whole process by calling the inverse FFT,
therefore obtaining the windowed synthesize sound and
what we call overlap at strategy.
We will be adding together these fragments into a whole signal.
And hopefully, if we have done the things correctly,
especially if we have chosen the window and the hop size correctly,
these output signal will be identical to the original.
So let's look at the code of the STFT that we have in the SMS tools package.
Okay, so this is a filed called stft.py and in it we have three functions.
One there is a STFT, that does the whole analysis and synthesis.
And then, we just have the analysis and synthesis separate, so
sometimes it's useful.
So we have the STFT-anal function and the STFT-synth function for the inverse part.
But let's talk about the complete analysis, synthesis, so
let's talk about the STFT function.
As input parameter is it has the X, our array with the complete sound,
we need to do the sampling rate of that sound.
Then we need to pass it, the window we're going to be using W then the size
of the FFT capital N and the hop size, how are we going to hop through the sound.
And basically, the core of the function is a loop In this case,
a wild loop that keeps advancing through the sound by a pointer, this pin.
Pin is a pointer that keeps advancing by age, by capital age, by the hop size.
So, pin gets advanced at every iteration by capital
H until it reaches the end which is the end of the sound.
And within that, we basically choose the fragment of the sound
that we want to analyze and then we call the DFT anal function.
So this is the function that we already discussed last week
which is the DFT model, it comes from the package DFT model, okay.
So we don't have to worry now here about the zero phase windowing about
the zero padding, etc.
Everything is done within this DFT-anal function and
it receives the fragment of the sound, the window, and the FFT size.
And it returns the positive site of the magnitude and phase spectrum, okay.
After that, we can do the inverse DFT also calling the DFT-synth function.
And therefore, that will return a window fragment of the sound and
then we can put it all together, overlap and adding the sound and
creating the overall output array.
Okay, and apart from that the rest are basically initialization and
ways to better analyze the sound and
to make sure that the pointer Is can start at the very beginning of the sound.
And can end at the very end of the sound, so I'm not going to go into detail
of that but please feel free to go over that and understand the scope.
Okay, so now we have this function so
that returns sound and let's use it.
Let's write some code to use this function.
So this is a little script that I wrote to call the STFT aal,
in which I first have to get out the packages
required including the actual STFT function, so
this comes from the models and directory.
And then, I can get a file, define the window and then,
define the size of the window, the FFT size and the hop size.
And now, we can create the file with wavread,
get the window and call the STFT function.
And this will fill up two variables mX and pX which will
include the sequence of magnitude and phase spectrum.
Okay, so let's run, it let's run test3.
Okay, that's quite quick, so it analyse the sound and
it adapting all these variables.
So, let's go through them, so
maybe the first thing we can do is to listen to the sound.
So we can just do
the exclamation mark play and the name of the sound,
which is this flute sound, and we can listen to that.
[SOUND] Okay?
So this is the sound of the flute and now,
we can go through the different variables that we have computed.
So for example, the first one is the sound that we read, the X array,
so this is the whole sound, the array of the whole sound,
so it's more than 80,000 samples.
Okay, then of course we can plot the window that we
are computed which is a humming window.
And then, we have the output array of magnitude and
phases but those are very big variables.
They are very big matrixes of values, so in fact, if we put mx.shape it
will tell me the dimensions of the ray.
Basically, the number of frames and
the number of samples in the spectrum that we have.
So in fact this tells me that we have 238 frames and
each frame has 512 samples.
These 512 samples are the samples of the magnitude spectrum of half of
the magnitude spectrum.
So now, we can access individual frames of that, so if we put for example mX,
and we access for example the frame number 50.
And we plot the whole spectrum of that 50th frame,
these are the values of the magnitude.
And of course we can, if we call the function plot with that variable,
we will plot the magnitude spectrum of the particular location.
Okay, so this is the flute sound, we see the harmonics and
we see how the higher values are much softer and it's quite noisy.
Of course we can also plot the phase spectrum of that
same location, this is the phase.
The ungraph phase, so we can see all these a nice looking face shape, okay.
Of course we want to visualize the whole spectrum, so
in order to visualize the whole spectrum the plotting command
that can do that is the what is a the function is called pcolormesh.
Pcolormesh it plots a matrix as a kind of a three
dimensional shape, so if we give it p and
mX it will basically plot all the frames.
But it maybe plotted in a different way that we are accustomed to
because the vertical is the axis is the time.
So in fact, we see time going from bottom to top and
the frequency is on the horizontal axis.
In fact a better way to plot this with colormesh is transpose it first,
so we can call the transpose function.
So this will transpose the mX in a different way,
so that we can see it in a more meaningful way.
And so now we see time going from left to right and frequency going,
or the frame, yeah, frequency going from bottom to top.
Okay, so that's basically how we compute the magnitude and
the phase spectrum using STFT and how we can plot it.
And there is a function file that implements all that and
makes plots in a nice way and that's the STFT_function.
And this is the one that is call from the graphical interface that we also use and
here it has one function called main and it goes through the kinds
of things we did and it does both the analysis and synthesis.
Okay, so the main has its input values all the input parameters and
then it reads the sound file, it gets the window It performs analysis.
This is exactly what we did now but then it does the synthesis and
it output the synthesis file into a location with
a given name and then it writes it, okay?
By the way, here this file I just copied into the work-space but
I had to change the relative path of a number of these names.
So for example, I have to change the path of the software models or
I have to change the output file name because in here it.
It was using particular directory to store the sound files, so
here, I just write the directory on this directory.
Okay, so let's execute this, that's why I have it in this directory.
So, it's STFT function, so I can just run STFT function, okay.
And it plots, well, the input sound, the magnitude,
the derivative of the phase and the output sound.
And this has created a file which is
called what is it called is called piano stft.wav and
of course, from here again,
I can play so, I can say play stft,
I know piano, piano Stft.wav.
[MUSIC]
Okay, and it's basically the same than the original sound.
Of course now, from this file I can really change and
make the things differently if I want.
So feel free to change how the plots are shown but
these are quite nice way to plug the things with the correct access,
with the correct labels, with the correct scaling.
So that we can better visualize the analysis and
synthesis of using this TFT and that's all I wanted to say.
So basically, we have gone through the short-time Fourier transform,
the implementation of the short-time Fourier transform.
So make sure that you understand the actual equation and
what are the signal processing considerations that are involved in that.
And then, of course we use a number of packages from Python together with
the SMS tools that we have already developed an implement
the DFT and the SDFT so we have been calling those functions.
That's all, so with this programming class hopefully,
you have gotten an idea of how to implement the STFT and
how can we use the functions of the SMS tools to
actually adapt it to our own particular needs.
So please feel free to play around with that and make sure that you understand
the STFT also from this programming perspective and that's all.
Thank you very much.
Explore our Catalog
Join for free and get personalized recommendations, updates and offers.
Coursera provides universal access to the world’s best education,
partnering with top universities and organizations to offer courses online.
© 2019 Coursera Inc. All rights reserved.
