Explicit Formula for
Partial Information Decomposition
Abstract
Mutual information between two random variables is a well-studied notion, whose understanding is fairly complete. Mutual information between one random variable and a pair of other random variables, however, is a far more involved notion. Specifically, Shannon’s mutual information does not capture fine-grained interactions between those three variables, resulting in limited insights in complex systems. To capture these fine-grained interactions, in 2010 Williams and Beer proposed to decompose this mutual information to information atoms, called unique, redundant, and synergistic, and proposed several operational axioms that these atoms must satisfy. In spite of numerous efforts, a general formula which satisfies these axioms has yet to be found. Inspired by Judea Pearl’s do-calculus, we resolve this open problem by introducing the do-operation, an operation over the variable system which sets a certain marginal to a desired value, which is distinct from any existing approaches. Using this operation, we provide the first explicit formula for calculating the information atoms so that Williams and Beer’s axioms are satisfied, as well as additional properties from subsequent studies in the field.
I Introduction
Since its inception by Claude Shannon [1], mutual information has remained a pivotal measure in information theory, which finds extensive applications across multiple other domains. Extending mutual information to multivariate systems has attracted significant academic interest, but no widely agreed upon generalization exists to date. For instance, the so-called interaction information [2] emerged in 1960 as an equivalent notion for mutual information in multivariate systems, and yet, it provides negative values in many common systems, contradicting Shannon’s viewpoint of information measures as nonnegative quantities.
Arguably the simplest multivariate setting in which Shannon’s mutual information fails to capture the full complexity of the system is that of a three variable system, with two source variables, and one target variable. Mutual information between the source variables and the target variable does not provide insights about how the source variables influence the target variable. Specifically, in various points of the probability space the value of the target variable might be computable either:
-
(a)
exclusively from one source variable (but not the other);
-
(b)
either one of the source variables; or
-
(c)
both variables jointly (but not separately).
In 2010 William and Beer [3] proposed to formalize the above fine-grained interactions in a three variable system111William and Beer formulated their notions for any number of source variables, and yet herein we focus on three variables for simplicity. Extending our methods to multiple source variables will be addressed in future version of this paper. using an axiomatic approach they called Partial Information Decomposition (PID). They proposed decomposing said mutual information to four constituent ingredients called information atoms, which capture the above possible interactions between the variables:
-
(a)
two unique information atoms, one for each source variable, which capture the information each source variable implies about the target variable, that cannot be inferred from the other;
-
(b)
one redundant information atom, which captures the information that can inferred about the target variable from either one of the source variables; and
-
(c)
one synergistic information atom, which captures the information that can be inferred about the target variable from both source variables jointly, but not individually.
Ref. [3] proposed a set of axioms that the above information atoms should satisfy in order to provide said insights, and follow-up works in the field identified several additional properties [4, 5, 6, 7]. Yet, in spite of extensive efforts [8, 9, 10, 11, 12], a comprehensive definition of information atoms which satisfies all these axioms and properties is yet to be found.
In spite of limited understanding of the information atoms, PID has already found multiple applications in various fields. As a simple example [13, Fig. 1], one can imagine the two source variables being education level and gender, and the target variable being annual income. An exact formula for computing the information atoms would shed insightful information about the extent to which annual income is a result of education level, gender, either one, or both.
Beyond this simple example, PID has broad applications in a wide range of fields. In brain network analysis, PID (or similar ideas) has been instrumental in measuring correlations between neurons [14] and understanding complex neuronal interactions in cognitive processes [15]. For privacy and fairness studies, the synergistic concept provides insights about data disclosure mechanisms [16, 17]. In the field of causality, information decomposition can be used to distinguish and quantify the occurrence of causal emergence [18], and more.
In this paper, we propose an explicit PID formula that satisfies all of Williams and Beer’s axioms, as well as several additional desired properties. We do so by introducing the do-operation, which is inspired by similar concepts in the field of causal analysis [19, 20, 21]. Intuitively, based on the understanding that unique information is “ideal conditional mutual information,” our method first adjusts the entire probability distribution by using the do-operation in order to make the target variable identical to its conditional distribution given one source variable, and then calculates the expectation of mutual information between it and the other source variable under different conditions. It is worth noting that our method is not based on any of the point-wise, localized, or optimization approaches that existing methods use.
We begin in Section II by introducing the PID framework, and its axioms and properties. We continue in Section III by introducing our do-operation and the definition of unique information, from which all other definitions follow, and prove that all axioms and properties are satisfied. We discuss the intrinsic meaning of our definition in Section IV, and provide all proofs in the appendix.
II Framework, axioms, and properties
The following notational conventions are observed throughout this article: (similarly etc.) denote a random variable, its corresponding (finite) alphabet, and an element of that alphabet, respectively. The distribution of is denoted by , the joint distribution of and is denoted by , and the distribution of given is denoted by .
For random variables , the quantity captures the amount of information that one target variable shares with the source variables , but provides no further information regarding finer interactions between the three variables. To gain more subtle insights into the interactions between and , [3] proposed to further decompose into information atoms. Specifically, the shared information between and should contain a redundant information atom, two unique information atoms, and one synergistic information atom (see Figure 1).
The redundant information atom (also called “shared”) represents the information which either oder imply about . The unique information atom represents the information individually contributed to by , but not by (similarly ). The synergistic information atom (also called “complementary”), represents the information that can only be known about through the joint observation of and , but cannot be provided by either one of them separately. Together, we must have that
(1) |
We refer to (1) as Partial Information Decomposition (PID).
Moreover, since the redundant atom together with one of the unique atoms constitute all information that one source variable implies about the target variable, it must be the case that their summation equals the mutual information between the two, i.e., that
(2) |
In a similar spirit, the synergistic information atom and one of the unique information atoms measure shared information between the target variable and one of the source variables, while excluding the other source variable. Therefore, the summation of these quantities should coincide with the well-known definition of conditional mutual information, i.e.,
(3) |
Eqs. (2), and (3) are the foundation of an axiomatic approach towards an operational definition of the information atoms. These equations form the first in a series of axioms, presented next, which were raised in previous works on the topic [3, 22, 23]. Such axiomatic approach was also taken in the past in order to shed light on Shannon’s mutual information [24].
Axiom 1 (Information atoms relationship).
Notice that it suffices to specify the definition of any one of the information atoms, and the definitions for the remaining atoms follow from Axiom 1. Consequently, [3, 25] chose to specify , and provided three additional axioms which should satisfy.
The first additional axiom is commutativity of the source variables, which implies that the order of the source variables must not affect the value of the redundant information.
Axiom 2 (Commutativity).
Partial Information Decomposition satisfies .
The second is monotonicity, which implies that the redundant information is non-increasing when adding a source variable, since the newly added variable cannot increase the redundancy between the original variables. We sidestep the discussion about monotonicity with more than two variables, which is not our focus in this paper, even though it can be easily obtained by extending our definition to more than two source variables.
The third is self-redundancy, which defines the redundant information from one source variable to the target variable (i.e., ) as the mutual information between them. In the case of two source variables considered herein, monotonicity and self-redundancy merge into the following single axiom.
Axiom 3 (Monotonicity and self-redundancy).
Partial Information Decomposition satisfies .
Notice that Axiom 3, alongside Axiom 1 (specifically (2)), imply that is a nonnegative quantity. The nonnegativity of is stated in [3, 25] as a separate axiom, shown next.
Axiom 4 (Nonnegativity).
Partial Information Decomposition satisfies .
The nonnegativity of is normally not listed as an axiom, since it is debatable if it should or should not be nonnegative; we will show that our method yields nonnegative under the closed-system assumption (i.e., ) in Section III, and further discussion is given in Section IV.
Besides, subsequent to [3, 25], studies suggested two additional properties, additivity and continuity [12, 26]. Additivity implies that whenever independent variable systems are considered, the joint information measures should be the sum of the information measures of each individual system. This is the case, for instance, in joint entropy of two independent variables.
Property 1 (Additivity).
Partial Information Decomposition of two independent systems and satisfy
for every .
Continuity implies that small changes in the probability distribution lead to small changes in the value of the information measure. It ensures that the measure behaves predictably and is a key property in information theory, particularly for measures like entropy and mutual information.
Property 2 (Continuity).
, , and are continuous functions from the underlying joint distributions to .
In addition, another well-known property is independent identity [9], which asserts that in a system of two independent source variables and a target variable which equals to their joint distribution, the redundant information should be zero.
Property 3 (Independent Identity).
If and , then .
We mention that several important properties can be inferred from the above. For example, the non-negativity of can be obtained from Axiom 1 and Axiom 3 as mentioned earlier; the commutativity of follows from Axiom 1 and Axiom 2; the difference between (2) and (3) is often called consistency [12], etc.
Finally, we emphasize once again that none of the existing operational definitions of the information atoms satisfy all of the above. A comprehensive list of violations is beyond the page limit of this paper, and yet we briefly mention that Axiom 4 (nonnegativity) is violated by [8, 9, 27] (although some sources do not refer to non-negativity as a requirement); Property 1 (additivity) is violated by all works except [12], [28], [8], and [29] according to [26]; Property 3 (independent identity) is violated by [3]; Property 2 (continuity) is violated by [11, 8], [28], [29], etc.
III Proposed information decomposition definition
In this section, we present our operational definition of , from which the definitions of the remaining information atoms follow. Then, we explain the logic behind this definition, and prove that it satisfies all the axioms and properties proposed in Section II.
III-A Definition of Information Atoms
Given a system define a new random variable over the alphabet via its conditional joint distribution as follows:
(4) |
meaning, for every ,
The variable is well-defined since all probabilities are non-negative, and since
Definition 1 (Unique Information).
The unique information from to given is .
The definitions for the remaining information atoms are then implied by Axiom 1 as follows.
Definition 2 (Redundant Information).
The Redundant Information from and to is defined as:
Definition 3 (Synergistic Information).
The synergistic information from and to is defined as:
III-B Intuitive Explanation of Definition 1.
Our definition of unique information is derived from a newly defined do-operation.
Definition 4 (Do-operation).
Given and such that the support of is contained in the support of , let , where
(5) |
for all .
In Lemma 8, which is given and proved in Appendix 8, it is shown that in Definition 4 is well-defined in the sense that the right marginal of is identical to . Therefore, there is no ambiguity in referring to both the input distribution and the right marginal of the output distribution by the same letter . This operation receives and , and outputs a joint distribution whose right marginal is , and for all . Using the do-operation, can be defined equivalently as follows:
Definition 5 (Unique Information, equivalent definition).
For let be a random variable with distribution , and let . The unique information from to given is defined as:
The proof of equivalence is simple, and is given in Appendix -B. The idea behind Definition 5 is that by setting for some in Definition 4, and then by averaging the resulting mutual information values over all , we eliminate the effect of from the directional dependence between and , without changing the directional dependence itself. A similar idea exists in Judea Pearl’s do-calculus [20] (also [21]), where a node in a Bayesian network is set to a certain value while removing all incoming dependencies to that node, thereby distilling the causal relationship between that value and the remainder of the network.
III-C Satisfaction of axioms and properties
To show that our definition satisfies the axioms and properties mentioned in Section II, we require the following technical lemma. The proof is given in Appendix -C.
Lemma 1.
Following the notations of Definition 1, we have that , and that .
Corollary 1.
Unique information (Def. 1) can be written as:
Corollary 2.
By Lemma 1, and since conditioning reduces entropy, we have that .
Based on the above lemmas and corollaries, we are in a position to prove that our definition of satisfies the required axioms.
III-C1 Proof of Axiom 1, Information atoms relationship
III-C2 Proof of Axiom 2, Commutativity
First, Definition 1 and Definition 2 provide the following equivalent way for computing , which is proved in Appendix -D.
Lemma 2.
Redundant information (Def. 2) can alternatively be written as .
Similarly, by switching between and in Definition 2 we have that ; based on Lemma 2, this equals to , where is defined analogously to in (III-A), i.e.,
(6) |
Then, we can conclude the commutativity of redundant information through the following lemma, which is proved in Appendix -E.
Lemma 3 (Commutativity of Redundant Information).
For and as above, we have that .
III-C3 Proof of Axiom 3, Monotonicity and self-redundancy
III-C4 Proof of Axiom 4, Nonnegativity
We begin by showing that is nonnegative, for which we require the following lemma, proved in Appendix -F.
Lemma 4.
Unique information (Def. 1) is bounded from above by mutual information, i.e.,
Then, nonnegativity of follows from Lemma 4.
Corollary 3 (Nonnegativity of Redundant Information).
Redundant information (Def. 2) is nonnegative, i.e.,
Finally, we remark that even though it is not a required axiom, non-negativity of can be proved under an additional assumption as follows.
Remark 1 (Nonnegativity of Synergistic Information).
Suppose (closed system assumption). Since , it follows from Definition 3 that .
III-C5 Proof of Property 1, Additivity
The following lemma is proved in Appendix -G.
Lemma 5 (Additivity of Unique Information).
For two independent sets of variables and , unique information (Def. 1) is additive:
(7) |
III-C6 Proof of Property 2, Continuity
We begin by showing that is continuous, for which we require the following lemma proved in Appendix -H.
Lemma 6 (Continuity of Redundant Information).
The redundant information (Def. 2) is a continuous function of the input distribution to .
III-C7 Proof of Property 3, Independent Identity
The following lemma is proved in Appendix -I.
Lemma 7.
The operator satisfies Property 3.
IV Discussion
In this paper, we proposed an explicit operational formula for PID, which is distinct from any existing approach, and proved that it satisfies all axioms and properties. In this section we provide further intuitive explanation for our approach.
First, we wish to elucidate the role that our do-operation plays in the definition of (Definition 5). In a sense, the do-operation can be understood as adjusting the marginal distribution of the variable of , while impacting its connections with other variables as little as possible. This understanding can be confirmed by Lemma 1, which shows that the expected value of the conditional entropy after the do-operation retains its original value. This resembles the invariance implied in Shannon’s communication model [1], where the conditional entropy of the output given the input is not affected by the input distribution. From this perspective, and can be regarded as the input and output of the channel, respectively, that indicates their “relationship.” The do-operation changes the distribution of the input , but does not change the channel’s characteristic (i.e., ).
Based on this, Definition 5 realizes the intuition that unique information should represent the relationship between source variable and target variable given the other source variable. So, we use the do-operation to control the marginal distribution of the target variable to its conditional distribution given the value of some source variable , and then use the expectation of mutual information to capture the “connection” between the specific source variable and target variable given after the do-operation.
The reason this method can partition to and (Def. 3), is that the do-operation eliminates high-order relations between and , i.e. . Specifically, conditional mutual information relies on the joint conditional probability , in expectation over all . This distribution includes both the conditional influence of on , but also has a simultaneous influence on the relationship between and .
However, Definition 5 of unique information retains the relationship between and without influence from by using the conditional probability , in expectation over all , to perform the do-operation, which only reflects the conditional influence of on . Therefore, the expectation of mutual information can accurately quantify the unique information, which represents the pure conditional mutual relationship.
In addition to the above analysis of do-operations in unique information, Lemma 2 also brings another perspective worth discussing. Redundant information can be understood as the mutual information (or ) obtained by changing the joint probability distribution according to without changing the marginal distribution (or ) according to Lemma 1 ().
As mentioned earlier, our definition of might be negative, unless the system is closed (i.e., , Remark 1). While and represent the information shared by one or two source variables with the target variable, represents the information provided to the target variable by the “cooperation” of source variables. It is an accepted aphorism that cooperation does not necessarily increase outcome, and hence it might be the case that negative values of conform with intuition. However, the reason why this explanation is no longer necessary in a closed system, as well as alternative interpretations of that are nonnegative, remain to be studied.
Acknowledgments. The authors would like to thank an anonymous reviewer whose suggestions greatly simplified the paper. This research was supported by AFOSR grants FA9550-22-1-0054 and FA9550-23-1-0208.
References
- [1] Claude Elwood Shannon. A mathematical theory of communication. ACM SIGMOBILE mobile computing and communications review, 5(1):3–55, 2001.
- [2] Satosi Watanabe. Information theoretical analysis of multivariate correlation. IBM Journal of research and development, 4(1):66–82, 1960.
- [3] Paul L Williams and Randall D Beer. Nonnegative decomposition of multivariate information. arXiv preprint arXiv:1004.2515, 2010.
- [4] Robin AA Ince. The partial entropy decomposition: Decomposing multivariate entropy and mutual information via pointwise common surprisal. arXiv preprint arXiv:1702.01591, 2017.
- [5] Pedro AM Mediano, Fernando Rosas, Robin L Carhart-Harris, Anil K Seth, and Adam B Barrett. Beyond integrated information: A taxonomy of information dynamics phenomena. arXiv preprint arXiv:1909.02297, 2019.
- [6] Aobo Lyu, Bing Yuan, Ou Deng, Mingzhe Yang, Andrew Clark, and Jiang Zhang. System information decomposition. arXiv preprint arXiv:2306.08288, 2023.
- [7] Thomas F Varley. Generalized decomposition of multivariate information. arXiv preprint arXiv:2309.08003, 2023.
- [8] Virgil Griffith, Edwin KP Chong, Ryan G James, Christopher J Ellison, and James P Crutchfield. Intersection information based on common randomness. Entropy, 16(4):1985–2000, 2014.
- [9] Robin AA Ince. Measuring multivariate redundant information with pointwise common change in surprisal. Entropy, 19(7):318, 2017.
- [10] Nils Bertschinger, Johannes Rauh, Eckehard Olbrich, and Jürgen Jost. Shared information—new insights and problems in decomposing information in complex systems. In Proceedings of the European conference on complex systems 2012, pages 251–269. Springer, 2013.
- [11] Malte Harder, Christoph Salge, and Daniel Polani. Bivariate measure of redundant information. Physical Review E, 87(1):012130, 2013.
- [12] Nils Bertschinger, Johannes Rauh, Eckehard Olbrich, Jürgen Jost, and Nihat Ay. Quantifying unique information. Entropy, 16(4):2161–2183, 2014.
- [13] Pradeep Kr Banerjee, Johannes Rauh, and Guido Montúfar. Computing the unique information. In 2018 IEEE International Symposium on Information Theory (ISIT), pages 141–145. IEEE, 2018.
- [14] Elad Schneidman, William Bialek, and Michael J Berry. Synergy, redundancy, and independence in population codes. Journal of Neuroscience, 23(37):11539–11553, 2003.
- [15] Thomas F Varley, Maria Pope, Maria Grazia, Joshua, and Olaf Sporns. Partial entropy decomposition reveals higher-order information structures in human brain activity. Proceedings of the National Academy of Sciences, 120(30):e2300888120, 2023.
- [16] Borzoo Rassouli, Fernando E Rosas, and Deniz Gündüz. Data disclosure under perfect sample privacy. IEEE Transactions on Information Forensics and Security, 15:2012–2025, 2019.
- [17] Faisal Hamman and Sanghamitra Dutta. Demystifying local and global fairness trade-offs in federated learning using partial information decomposition. arXiv preprint arXiv:2307.11333, 2023.
- [18] Fernando E Rosas, Pedro AM Mediano, Henrik J Jensen, Anil K Seth, Adam B Barrett, Robin L Carhart-Harris, and Daniel Bor. Reconciling emergences: An information-theoretic approach to identify causal emergence in multivariate data. PLoS computational biology, 16(12):e1008289, 2020.
- [19] Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669–688, 1995.
- [20] Judea Pearl. Causality. Cambridge University Press, 2009.
- [21] Erik P Hoel, Larissa Albantakis, and Giulio Tononi. Quantifying causal emergence shows that macro can beat micro. Proceedings of the National Academy of Sciences, 110(49):19790–19795, 2013.
- [22] Paul L Williams. Information dynamics: Its theory and application to embodied cognitive systems. PhD thesis, Indiana University, 2011.
- [23] Virgil Griffith and Christof Koch. Quantifying synergistic mutual information. In Guided self-organization: inception, pages 159–190. Springer, 2014.
- [24] Imre Csiszár. Axiomatic characterizations of information measures. Entropy, 10(3):261–273, 2008.
- [25] Joseph T Lizier, Benjamin Flecker, and Paul L Williams. Towards a synergy-based approach to measuring information modification. In 2013 IEEE Symposium on Artificial Life (ALIFE), pages 43–51. IEEE, 2013.
- [26] Johannes Rauh, Pradeep Kr Banerjee, Eckehard Olbrich, Guido Montúfar, and Jürgen Jost. Continuity and additivity properties of information decompositions. International Journal of Approximate Reasoning, 161:108979, 2023.
- [27] Conor Finn and Joseph T Lizier. Pointwise partial information decompositionusing the specificity and ambiguity lattices. Entropy, 20(4):297, 2018.
- [28] Virgil Griffith and Tracey Ho. Quantifying redundant information in predicting a target random variable. Entropy, 17(7):4644–4653, 2015.
- [29] Artemy Kolchinsky. A novel approach to the partial information decomposition. Entropy, 24(3):403, 2022.
-A Proof of the completeness of Definition 4.
In this part, we will show that do-operation’s output is a probability distribution with the same marginal distribution as its input.
Lemma 8.
Furthermore, the marginal distribution of the output is equal to the input (call it in this section), i.e.,.
-B Proof of the equivalence of Definitions 1 and 5.
Proof.
For the purpose of the proof, let
(10) | ||||
(11) |
where . By Definition 4 we have , i.e.,
Therefore, we have
-C Proof of Lemma 1.
Proof.
By Definition 1, for every , we have
(12) |
and therefore
which can be simplified to
Therefore, we have that for every , which implies that .
-D Proof of Lemma 2.
-E Proof of Lemma 3.
Proof.
To prove that , it suffices to show that for all . We have
(15) |
in which and are symmetric. The proof is concluded by following similar steps in reversed order, i.e.,
-F Proof of Lemma 4.
-G Proof of Lemma 5.
Proof.
By Definition 1, we have:
(16) |
where
(17) |
Since are independent from , we have
which implies that and are also independent for every . Therefore, we have