Chapter three, the I-Measure. In this chapter, we are going to study the set theoretic structure of Shannon's information measures. In particular, we will answer the question why Shannon's information measures can be represented by a form of Venn diagram called information diagram. We are going to learn how to use information diagrams to obtain information identities, and inequalities. And we are going to discuss some problem solving examples. We have seen before this set theoretical representation of Shannon's information measures for two random variables, X_1, and X_2. Entropy of X_1, entropy of X_2, the joint entropy of X_1 and X_2, the conditional entropy of X_1 given X_2, the conditional entropy of X_2 given X_1 and the mutual information between X_1 and X_2 So what is the precise relation between information measures and set theory? Here we introduce a formal substitution of symbols. On the left hand side is a set of symbols in the information theory. On the right hand side is a corresponding symbols in set theory. Specifically, H or I that represents entropy or mutual information in information theory, corresponds to mu^* in set theory, where mu^* is some signed measure of set-additive function. For a comma in information theory, it corresponds to the union in set theory. For a semi-colon in information theory, it corresponds to the intersection in set theory. For the conditioning bar in information theory, it corresponds to set minus in set theory. That is, A minus B is equal to A intersect B complement. Here are some examples. The entropy of X_1 given X_2, corresponds to mu^* of tilde{X_1} minus tilde{X_2}, where tilde{X_1} is a set variable that corresponds to the random variable X_1, and tilde{X_2} is a set variable that corresponds to the random variable X_2. The entropy of X_2 given X_1 corresponds to mu^* of tilde{X_2} minus tilde{X_1}. And the mutual information between X_1 and X_2 corresponds to mu^* tilde{X_1} intersect tilde{X_2}. Here is another example. In set-theory, we have the celebrated inclusion-exclusion formula, which says that mu^* tilde{X_1} union tilde{X_2} is equal to mu^* tilde{X_1} plus mu^* tilde{X_2} minus mu^* tilde{X_1} intersect tilde{X_2}. This corresponds to the identity in information theory. Entropy of X_1, X_2 equals the entropy of X_1 plus the entropy of X_2, minus the mutual information X_1 semicolon X_2. Section 3.1 are some preliminaries. Here we introduce some elementary concepts in measure theory. The field F_n generated by sets tilde{X_1}, tilde{X_2} up to tilde{X_n} is the collection of sets which can be obtained by any sequence of usual set operations, that is, union, intersection, complement, and difference, on set variables tilde{X_1} tilde{X_2} up to tilde{X_n}. The atoms of the field F_n are sets of the form intersection i equals 1 up to n Y_i, where Y_i is either tilde{X_i} or tilde{X_i} complement. Here is an example for F_2, the field generated by the sets tilde{X_1} and tilde{X_2}. There are four atoms in F_2, tilde{X_1} intersect tilde{X_2}, tilde{X_1} compliment intersect tilde{X_2}, tilde{X_1} intersects tilde{X_2} compliment and tilde{X_1} compliment intersects tilde{X_2} compliment. There are a total of 2 to the power 4 equals 16 sets in F_2 formed by the unions of the above 4 atoms. A real function field defined on F_n is called a signed measure if it is set additive. That is for disjoint A and B in F_n, we have mu(A union B) is equal to mu(A) plus mu(B). The set additivity property is satisfied by the weight of objects. For example, the weight of two distinct objects is equal to the weight of the first object plus the weight of the second object. For any set-additive function mu, it can be shown that mu of the empty set is equal to zero, because for any set A, we have mu(A) is equal to mu(A) union the empty set, by set additivity is equal to mu(A) plus mu of the empty set. So now we have mu(A) equals mu(A) plus mu empty set. After cancelling mu(A), we see that mu on the empty set is equal to zero. A signed measure can take positive or negative values. If a signed measure takes only positive values, it is simply called a measure. Here is an example. A signed measure mu on F_2 is completely specified by the values on the four atoms that we have seen before. The value of mu on the other sets in F_2 are obtained by set-additivity. For example, mu on the set tilde{X_1} is equal to mu on the atom tilde{X_1} intersect tilde{X_2} union to atom tilde{X_1} intersect tilde{X_2} compliment. By set additivity, this is equal to mu on tilde{X_1} intersect tilde{X_2} plus mu on tilde{X_1} intersect tilde{X_2} compliment.