Bayesian methods are used in lots of fields: from game development to drug discovery. They give superpowers to many machine learning algorithms: handling missing data, extracting much more information from small datasets. Bayesian methods also allow us to estimate uncertainty in predictions, which is a really desirable feature for fields like medicine. When Bayesian methods are applied to deep learning, it turns out that they allow you to compress your models 100 folds, and automatically tune hyperparametrs, saving your time and money. In six weeks we will discuss the basics of Bayesian methods: from how to define a probabilistic model to how to make predictions from it. We will see how one can fully automate this workflow and how to speed it up using some advanced techniques. We will also see applications of Bayesian methods to deep learning and how to generate new images with it. We will see how new drugs that cure severe diseases be found with Bayesian methods.
Training GMM
Loading...
From the course by National Research University Higher School of Economics
Bayesian Methods for Machine Learning
140 ratings
National Research University Higher School of Economics
140 ratings
Course 3 of 7 in the Specialization Advanced Machine Learning
From the lesson
Expectation-Maximization algorithm
This week we will about the central topic in probabilistic modeling: the Latent Variable Models and how to train them, namely the Expectation Maximization algorithm. We will see models for clustering and dimensionality reduction where Expectation Maximization algorithm can be applied as is. In the following weeks, we will spend weeks 3, 4, and 5 discussing numerous extensions to this algorithm to make it work for more complicated models and scale to large datasets.
Meet the Instructors
Daniil PolykovskiyResearcher
HSE Faculty of Computer Science
Alexander NovikovResearcher
HSE Faculty of Computer Science
In the previous video,
we decided to use Gaussian Mixture Model to fit
our dataset and to solve the clustering problem, but how?
How can we do better than the generals that castigate and descent?
In this video, we will discuss some intuition that leads
to expectation maximization algorithm for this particular case.
To recall that the density of each data point in
our Gaussian Mixture Model is a weighted sum of three or,
in general, as many as you want, Gaussians divisions.
To start with, we will need to introduce a latent variable
here because it will make the reasoning about our model much easier.
What can we correlate more variable here?
Well, we can do something like this.
We can say that each data point was generated
by using some information from latent variable T, which exists.
It's like we have one latent variable T for each data point X and it causes X.
It explains something about X,
and the reasonable thing to assume about T here is that it takes 3 radius,
1, 2 or 3 and it shows us from which Gaussian this particular data point came from.
We actually don't know for any data point,
we don't know from which Gaussian came from.
It's a latent variable, right?
We doesn't observe this,
never nor in training nor in testing,
but these can be helpful so if we know the latent variables,
it can be helpful to understand something about our model.
Later, then with fit the Gaussian mixture model into our data.
We may find the distribution on the latent variable even the data.
We may ask how will the question,
how do you think,
what is the value for latent variable T for these particular data point?
The answer to this question will be basically the clustering, right?
It gives us the beliefs from which Gaussian these data point came from.
If we decided to use this kind of latent variable model,
then it is reasonable to assume that latent variable T has prior distribution pi,
so it's exactly the weights of our Gaussians.
The latent variable T equals to some cluster number,
for example one, with probability pi one and the likelihood.
The density of the data point X,
given that we know from which Gaussian came from,
is just Gaussian distribution because it came from the Gaussian.
Each data point, if we know that it came from the Gaussian number C,
it's density just these Gaussian things do with the parameters of
the Gaussian number C. When we introduce this kind of latent variable model,
we may now look at what will P of X given theta represent in this model.
We change our model and we now have to check that it still gives
you the same general results as our original model.
If you write down P of X given theta,
given parameters and this latent variable model, we will get this.
It's just a rule of sum for probabilities and it says
that a P of X given theta equals to sum with respect to T. So we are marginalizing T,
with the respect to all possible values for T, from 1-3,
and were assigning the join distribution P of X and T,
which equals to the likelihood times the prior.
If you substitute the definition of the prior and likelihood into this summation,
you can understand that this last formula on this slide
is exactly equal to the first formula.
We introduced some latent variable in
such a way that we when we marginalized out this variable,
we get exactly what we had before.
Which means that we now can use this latent variable model,
train it with observed data X,
and it will gives us exactly the same results
as we would get if we use the original model.
Let's now try to build some intuition about how to train this latent variable model.
So say we have a dataset and now,
say it's one dimensional.
Each data point from 1 to N is just one number for sublist of illustration.
How could our goal for finding the maximum likelihood estimation,
is finding the parameters, right?
How can we find the parameters?
Well, it turns out that if we know the sources,
if we know the values of the latent variables for each data point,
then finding the parameters sigma is easy.
Because you're going to say that all the blue data points here is the data points for
which we know that they came from the first Gaussian and
we can look at them separately from the older orange data points,
and just fit them into one Gaussian,
which we already know how to do because it's just
fitting some data set of blue point into a Gaussian distribution.
You can see the formulas on the bottom of the slide,
but it's something we have already done in week one,
which means that if we know the sources,
if we know the values of the latent variables,
it's easy to estimate the parameters data.
Actually, if we don't have this hard assignments,
but rather have soft assignments so some posterior distribution on T,
which means that for each data point.
We don't assume that it belongs to just one cluster,
but rather we assume that it belongs
to all clusters simultaneously, all Gaussian simultaneously.
What will some different probabilities being posterior P of T,
given X and parameters.
If we have these probabilities,
it is also easy to estimate the parameters.
So instead of just signing with respect to all blue points,
and averaging their position to get the location of the blue Gaussian,
we'll have to sum all the points but with weights.
If the posterior P of T = 1,
is 1 for some data point?
It means that this particular data point is completely blue.
It certainly belongs to the blue Gaussian,
and it will be used as just blue data point in the averaging with weight 1.
If for some data point,
P of T = 1 is 0,
it means that these data point is certainly orange,
and we will just don't use it in the computing the blue Gaussian mean at all,
but if the data point is for example,
P of T = 1 is 0.8,
it just means that this data point is kind of not certain.
It thinks that it belongs to the blue Gaussian but it's not sure.
It will highly affect the position of
the blue Gaussian and a little bit affect the position of the orange Gaussian.
We'll direct this kind of formulas later from more strict considerations,
but now, just believe me that if you know the posterior probably on T,
you could easily estimate the parameters this way.
The bottom line here is that if we know the sources,
no matter soft segments or hard segments,
then we can easily estimate the parameters of our Gaussian mixture model.
But on practice, we don't, right?
We don't know the sources,
so how can we estimate the sources?
Well, it turns out that if we know the parameters, so the Gaussians,
their locations and their variances,
then we can easily estimate the sources because we can use just the Bayes' rule to do it.
And by the Bayes' rule, the soft assignment,
the posterior probability of T equals to,
for example, blue Gaussian,
for some particle data point,
is just proportional to the join probability,
which is likelihood times prior.
And if we know the parameters theta,
we can easily compute these likelihood and prior at
any given point so we can easily to compute this posterior probability,
which is basically sources,
which is basically assignments for each data point for the clusters,
soft assignments, and we think that the normalization constant here can be problematic,
but it turns out that we have just two values here.
Two probabilities.
The distribution P of T given some data and theta,
can think just two possible values.
We can explicitly normalize this think by signing with
respect to two unnormalized probabilities. It's no big deal.
To summarize, we have kind of a chicken and egg problem.
If we know the Gaussian parameters,
we can easily estimate the sources.
If we know the sources,
we can easily estimate and Gaussian parameters.
What can we do in this case?
Well, the expectation maximisation algorithm,
in this particular case,
suggests us to do a very natural thing.
Let's, first of all,
internalize the parameter somehow randomly and then in iterations in the loop,
let's repeat two steps until convergence.
First of all, let's fix the parameters.
Assume that they are the true ones,
and they estimate the sources.
On the next sub-step,
let's use the sources to re-estimate the parameters,
to update the beliefs about the parameters.
This way, after repeating this two steps for long enough,
we will hopefully obtain a reasonable solution
to our probability distribution fitting problem.
We will be able to fit Gaussian mixture model into our data.
Explore our Catalog
Join for free and get personalized recommendations, updates and offers.
Coursera provides universal access to the world’s best education,
partnering with top universities and organizations to offer courses online.
© 2018 Coursera Inc. All rights reserved.
