Probabilistic graphical models (PGMs) are a rich framework for encoding probability distributions over complex domains: joint (multivariate) distributions over large numbers of random variables that interact with each other. These representations sit at the intersection of statistics and computer science, relying on concepts from probability theory, graph algorithms, machine learning, and more. They are the basis for the state-of-the-art methods in a wide variety of applications, such as medical diagnosis, image understanding, speech recognition, natural language processing, and many, many more. They are also a foundational tool in formulating many machine learning problems. This course is the first in a sequence of three. It describes the two basic PGM representations: Bayesian Networks, which rely on a directed graph; and Markov networks, which use an undirected graph. The course discusses both the theoretical properties of these representations as well as their use in practice. The (highly recommended) honors track contains several hands-on assignments on how to represent some real-world problems. The course also presents some important extensions beyond the basic PGM representation, which allow more complex models to be encoded compactly.
Independence of Causal Influence
Loading...
From the course by Stanford University
Probabilistic Graphical Models 1: Representation
994 ratings
From the lesson
Structured CPDs for Bayesian Networks
A table-based representation of a CPD in a Bayesian network has a size that grows exponentially in the number of parents. There are a variety of other form of CPD that exploit some type of structure in the dependency model to allow for a much more compact representation. Here we describe a number of the ones most commonly used in practice.
Meet the Instructors
Daphne KollerProfessor
School of Engineering
But we've previously talked about the importance of
allowing CPD representations that encode additional structure in the local
dependency model of a variable on its parents.
And we talked about the cases of tree CPDs.
Which allow us to depend on different variables in different contexts.
But none of that helps us deal with the situation that we used as a motivation
for this. Which is where we have a variable such
as, for example, cough. That depends on multiple different
factors: pneumonia, flu, tuberculosis, bronchitis, and so on and so forth.
This doesn't lend itself say, to a tree CPD.
Because it's not the case that you depend on one only in certain contexts, and not
and not in others. Really, you depend on all of them.
And all of them sort of contribute something to the probability of
exhibiting a cough. So one way for capturing that kind of
interaction is a model called the noisy OR CPD.
And the noisy OR can, is best understood by simply considering a slightly larger
graphical model where we have, where we're trying, where we break down the
dependencies of Y on its parents, X1 up to XK, by introducing a bunch of
intervening variables. So let's imagine that this is, again,
like cough variable. And this is different diseases for
example. And what we're doing here is we're
introducing a intermediate variable that's captures the event at this
disease, X1 if present, causes, a cough, by itself, so this is X1 by itself causes
a cough, or causes Y. You can think of it each of diseases is a
noisy transmitter. If you have the disease,
say if X one is true, then Z1 sort of says, fine X one succeeded in its intent
to make it Y true. X2 has its own little filter called Z2
and Z2 makes that same decision relative to X2.
So ultimately Y is true so Y is true. If, someone succeeded in making it true.
Which means that Y is a deterministic or of its parents, Z1 up to ZK,
Z0 to ZK. So now let's make this a little bit more
precise, so the probability of ZI being true given XI, there's two cases.
If XI0 equals zero well Xi isn't even trying to make ZI true, so in this case
the probably of ZI1 is zero, okay?
If Xi is not true Zi doesn't get turned on ever.
If Z, if Xi is true then there's some probability for Zi to get turned on.
And that probability is something called Lambda i.
And Lambda I is in the interval 01. And you can think of it as defining some
kind of penetrance of, how good is Xi at turning Y on?
So if we actually write this down as a probability, and we just consider the CPD
of Y equals zero so Y doesn't get turned on given some set of cases X1 up to XK.
So now we're asking what is the probability that all of these guys fail
to turn on the variable Y? So what I forgot to say is that Z0 is
what's called a leak probability and that's the probability that Y gets turned
on just by itself. And that has, and that happens with
probability lambda zero. So, when does y fail to get turned on?
First of all, when it doesn't get turned on by the leak.
So, that's one minus lambda zero, times the probability that none of the causes
that are on. So, none, of, causes, that are on.
Which are the x size that are equal to one turn y on.
And so that's a product of one minus Lambda I for all of the XI's that are on.
No, that gives us the probabililty of Y equals zero, and obviously the
probability of Y equals one is just the compliment of that, one minus.
sorry this should be
yeah, but os that's the noisy or CPD.
And you can generalize this to a much broader notion of independence of causal
influence. This is called independence of causal
influence because it assumes that you have a bunch of causes for a variable and
each of them acts independently to affect the truth of that variable.
And so, there's no interactions between the different causes.
They each have their own separate mechanism and ultimately it's all
aggregated together in in a single in a single variable, Z from which the truth
of Y is then is then determined from this aggregate effect of all of the.
All of the effects, ZI's of the different causes.
So, one example of this is, we, we've already seen the noisy orbit.
You can. Easily generalizes to a broad range of
other cases. There's noisy ands where the aggregation
function is an and. There's noisy maxes which apply in the
nonbinary case when causes might not just be turned or off but rather they have
different sort of extents of being turn on and then z is actually sort of the
maximal extent of of, of the, if, the independent effect of
each cause, and so on. So there's a lot, a large range of
different models all of which fit into this family, meesie order is probably the
one that's most commonly used but the other ones have also been used, in other
settings as well. One model that might not immediately be
seen to fit into this framework but actually does, is a model that
corresponds to the sigmoid CPD. So what's a sigmoid CPD?
A sigmoid CPD says that each XI induces a continuous variable which represents WI,
XI. So imagine if each XI is discrete, then
ZI is just a continuous value, WI, which parameterizes this edge, and it tells us,
sort of, how much force, XI is going to exert on making Y true.
So if WI is zero it tells us that XI exerts no influence whatsoever.
If WI is positive, XI is going to make Y more likely to be true and if WI is
negative it's going to make Y less likely to be true.
All of these influences are aggregated together in this expression for the
variables Z which effectively adds up all of these different influences plus an
additional bias term. W0.
And now we need to turn this ultimately into, the probability of the variable Y,
which is the variable that we care about. And in order to do that, what we're going
to do is we're going to pass this continuous quantity Z, which is a real
number between negative infinity and infinity, through a Sigmoid Function.
The Sigmoid Function is defined as follows, and it's a function that some of
you have seen before in the context of machine learning, for example.
So Sigmoid takes the value, the continuous value Z, exponentiates it, and
then divides by one plus that exponent of Z.
And. Since E of Z, since E to the power of Z
is a positive number, this gives us a number that is always in the interval of
0,1. And if we look at what this function
looks like. It looks like this.
So, this is the sigmoid function. The X axis here is the value Z.
And the Y axis is the sigmoid function. And you can see that as Z gets very
negative, the probability goes to zero. As Z gets, close, very high, the
probability gets close to one, and then there's this interval in the middle where
intermediate values are taken. You can.
So this is kind of like a squelching function that that sort of squashes the
function on both ends. Let's look at the behavior of the sigmoid
CPD as a function of different parameters.
So here is a case where all of the X Is have the same parameter W.
And so what we see here is the value of this parameter W, and over here is the
number of XI's that are true. So let's look at, first this access over
here, the more parents that are true, the more parents that are on, the higher.
The probability of Y to be true, okay. And this, it holds for any value of W
because these are all positive influences.'Kay.
So the more parents are true, the more things are pushing Y to take the value
true. This axis over here is the axis of the
weight and we can see that for low weights, you need an awful lot of X's to
get Y to be true but as W increases, Y becomes true with a lot of fewer positive
influences. This graph on the right now.
Is what we get when we basically just increase the amplitude of the whole
system. We multiply both W and W0 by a factor of
ten. And what happens is that, that means that
the exponent gets pushed up to extreme values much quicker.
Z gets dissect, effectively multiplied by a factor of ten.
And that means that the transition becomes considerably sharper.
That gives us a little bit of an intuition on how the sigmoid function and
how the sigmoid cpd behaves. So what are some examples of this kind of
a of an application of this. So I showed this network in an earlier
part of this course it's the CBCS network and it's used for the it was developed
for the here at Stanford Medical School for diagnosis of internal diseases.
And so, up here we have things that represent predisposing factors.
And there's actually a fairly eclectic range here.
So for example one tree disposing factor is intimate contact with small rodents.
because that's the contributing factor for the antivirus.
and so there's a whole range of predisposing factors.
Down here in the middle we have diseases. An down at the bottom, we have symptoms
and test results. Now as I previously mentioned there's
approximately 500 variables in this network and they take on network about
four values. So the total number of entries in a joint
distribution over this space would be approximately four to the 500 different
parameters, which is clearly an intractable number.
If we were to take this network, and just, if you take this distribution
represented with the network shown in this diagram we get considerable
sparsificaton and the factorized form. As approximately 134 million parameters,
which is still much too many that have a human estimate.
By using as in this case, they use a noisy max CPD.
They brought the number of parameters to about 8,000 total parameters for this
network. Which is a much more attractable number
to deal with.
Explore our Catalog
Join for free and get personalized recommendations, updates and offers.
Coursera provides universal access to the world’s best education,
partnering with top universities and organizations to offer courses online.
© 2019 Coursera Inc. All rights reserved.
