If you're watching this, hopefully this means that you've followed the course and are noticing and processing the applications until here. You've just reached the half point. So congratulations, and no need now to give up. Very interesting topics are still coming. Last week we talked about the sectional model. And now following in the path towards a more flexible and higher level representation of sounds we present the harmonic model. We will first introduce the actual model then talk about the difference between sinusoids, partials and harmonics. Then discuss the difference between polyphonic and monophonic signals. Then present the idea of how to detect these harmonics from sound, given that we know the fundamental frequency, which is going to be a topic of the next lecture. And then we'll put it together into an analysis synthesis system, the harmonic model system. So, the equation of the harmonic model is very close to the one of the sinusoidal model that we already explained. There is a single, but very significant difference. The frequencies of the sinusoids are all multiples of a fundamental frequency. Thus, it's only valid for singles for sounds produced by periodic or state of the periodic oscillations unlike the sounds produced by many musical instruments. So, if we look at the equation. Well, the output signal yh where we emphasize the idea that is a harmonic signal with the h That is, of course, the index n is a sum of our cosine function with a time varying amplitude and time varying frequency but here the frequency is not f sub r which would allow any frequency for any sinusoid But is r f(0). So this means that they are all multiples of f(0). R is an integer value, so it means that all the sinusoidal are either for different values of r, either the fundamental frequency, or any of its multiples. We can also express the model in the frequency domain. The time varying spectrum of a harmonic sound is the sum of all the time varying spectrum of the harmonics. So, in this case we are expressing the idea of a sinusoid being the transform of the window that we're using the analysis. So the spectrum, the overall harmonica spectrum is the sum of all those windows, which are scaling factor of an amplitude A, and a shifting factor of minus RF in which. We are placing these windows at the appropriate location of the sinusoid. Okay, this is basically what we talked about in the sinusoidal model, but now again here restricting the frequencies to the multiples of the fundamental frequency. Here we show spectrogram of the sinusoidal tracks of two sounds that have different characteristics. The top sound is enharmonic sound, a note of a vibraphone. And the bottom plot is a harmonic sound, voices singing a vowel. Let's listen to the top one, which is this vibraphone sound. [NOISE] Okay, this is a very simple sound, but it has lines that are non harmonic, so that means they are not multiples off from the mental frequencies. Some are related, in fact we listen pitch in this note, but there is some of the that are clearly not perfect multiples of this fundamental. Instead, the second sound which is a voice sound which varies so it's a little more complex. [MUSIC] Okay, so this is the sound of Vig Nitia singing. And here we see that the lines, the sinusoidal lines, are very much multiples of a fundamental. They're multiples in fact of the first line. And this is the typical structure of a harmonic sound. In order to better understand the concept of harmonics, it's important to understand the difference between sinusoids, partials and harmonics. Many people get confused with that. For a sinusoid is a mathematical function. So, it's a function we compute from an equation. And that is, it has an analytical representation, and therefore comes from a mathematical point of view. So the top plot is clearly the spectrum of twos of such mathematical function to sinusoids that we have computed with the sine function. A partial instead is a component of a signal that is periodic and stable. And that can be modeled as a slowly timed variant sinusoid. Okay. so, here we introduce the concept of a real sound. A partial is a concept that comes from analysing a real signal. And it has a certain behavior that can be approached and modeled with this mathematical concept that is the sinusoidal function. So, this second plot is one spectrum of the vibraphone, in which we see that there are several of these components in the spectrum that can be approach or can be model with idea of a Sinusoid. Others not so in here clearly there is some very clear sinusoidal components and some other aspect of the spectrum, which might is not so easily model with an ADF sinusoids. And finally a harmonic is a partial of a sound that is a multiple of a fundamental frequency. So it can also be modeled a slowly time carrying sinusoid, but it has this added restriction that has to be a multiple of a given frequency. So, in this last plot that is the oboe sound we see these very the peaks the top peaks these crosses these are clearly the multiples of the first peak. Which is the fundamental frequency, but there is other crosses which are not harmonic. They are not multiples of this fundamental frequency. These are maybe artifacts or maybe part of some signal that is underneath or maybe some side log. Okay, and looking at the face spectra of the same signals, it also helps to understand some of these concepts. So the top one is the phase of these two sinusoids. We see these very flat area in the main lopes of the two soundwaves. In the second plot with the vibraphone sounds there are clearly some areas that are quite flat where the cross is and that hopefully means that they are kind of sinusoidal and some other crosses are not in a stable region a phase region. So, that means that maybe they are not sinusoids, and finally on the oboe sound we see the same things. Some of the crosses are clearly in a flat region, and these clearly means that it's in a stable component, so it's a flat phase stable component. And some of the crosses are not in such areas though so that might give some indication that they're not sinusoidal, or they cannot be modelled easily as sinusoidal components. In order to use the harmonic model, we need to identify the sound sources that are harmonic. In the case of a polyphonic signal, there are several sound sources, for example there might be several instruments playing at the same time. And some might be harmonic. Some might be not. For example, the stop plot is the fragment of the kinetic music concert in which we hear a voice and some accompaniments. Let's listen to that. [MUSIC] So the goal would be to identify the harmonics of the voice sound. And looking at these sinusoidal tracks, well, it may be not that easy, but still, it can be done. We can try to track the harmonics of these voice sounds, or other harmonic components that might be present. A monophonic signal includes just one single source like the one below, which is just a single voice that we already heard, so it's Vignesh. [MUSIC] So he is, in fact, the same singer than the voice above, but clearly here we can see the time varying harmonics much easier. And these sinusoidal tracks that we have identified can show very clearly the harmonics of the voice. So the question is, how to identify the harmonics of sound source being in a monophonic a signal or in a polyphonic signal. We are just focusing on harmonic sounds. So, you'd like to find the harmonics in a monophonic signal or in one of the sound sources of a polyphonic signal. The measure problem is the identification of the fundamental frequency of a sound source. And this will be covered in the next lecture. So here we will just assume that we know the fundamental frequency of the sound. So we'll focus on the concept of a harmonic, and we'll define that as being a spectral peak whose frequency is close to a multiple of the fundamental frequency. And we can formulate this idea with this equation, which we define f p a peak the concept of it being a harmonic will be when this absolute value of the difference between f [INAUDIBLE] and a multiple of f sub 0 for a given frame l is smaller than a threshold that we give. And at the same time we defined this idea of being stable as having been lived for some time as having been in existent for some time. So, we also make the restriction that This f sub h, have had some existence for some number of frames before the corum frame. So we would define L as the number of frames, the minimum number of frames that we need to have in order for a harmonic to be defined as a harmonic. The implementation of the harmonic model, is a modification of the sinusoidal model that we saw last week. Here we introduce two models, one is the F0 detection algorithm, that again requires to be treated separately, and we'll discuss in the next lecture, and the idea of the harmonic detection towards to what we talked about. The idea that we select the peaks that are harmonic of the fundamental frequency. And the rest is exactly the same than the sinusoidal model. So, this is an example of an analysis synthesis of the sinusoidal model. That was the vignesh sound, the harmonics of that. Of course, that always might have some problems in the areas in which we don't have a clear harmonics, like in the transitions. In this case, even in the consonants of silences, we might have some problems. But In general it works. So from this sound that we already heard. [MUSIC] This is the synthesized sound, which is quite close to the original sound. So, for this sound the harmonic model works quite well. The concept of harmonic oscillations in the study of many natural phenomena is quite present in many books and references. And you will be able to find them in Wikipedia, you can find many of that. Of course, there is not much when we deal with specific music signals and sounds and the harmonicity of sound. But still you will find some things on Wikipedia and on some books. And again the sounds that you have heard that I played come from free sound, and you can find the code for all these plots that I showed in the GitHub repository. So, we have covered the harmonic model. So, we can use it to analyze and synthesize signals. The application facilities, we will see that are much larger than with the sinusoidal model. And we will show some examples of that. The difficult step is the detection of the fundamental frequency. And that's what we'll cover in the next lecture. So, see you in the next class.
