Explicit Formula for
Partial Information Decomposition

Since its inception by Claude Shannon [1], mutual information has remained a pivotal measure in information theory, which finds extensive applications across multiple other domains. Extending mutual information to multivariate systems has attracted significant academic interest, but no widely agreed upon generalization exists to date. For instance, the so-called interaction information [2] emerged in 1960 as an equivalent notion for mutual information in multivariate systems, and yet, it provides negative values in many common systems, contradicting Shannon’s viewpoint of information measures as nonnegative quantities.

Arguably the simplest multivariate setting in which Shannon’s mutual information fails to capture the full complexity of the system is that of a three variable system, with two source variables, and one target variable. Mutual information between the source variables and the target variable does not provide insights about how the source variables influence the target variable. Specifically, in various points of the probability space the value of the target variable might be computable either:

1. (a)

   exclusively from one source variable (but not the other);

2. (b)

   either one of the source variables; or

3. (c)

   both variables jointly (but not separately).

In 2010 William and Beer [3] proposed to formalize the above fine-grained interactions in a three variable system¹¹1William and Beer formulated their notions for any number of source variables, and yet herein we focus on three variables for simplicity. Extending our methods to multiple source variables will be addressed in future version of this paper. using an axiomatic approach they called Partial Information Decomposition (PID). They proposed decomposing said mutual information to four constituent ingredients called information atoms, which capture the above possible interactions between the variables:

1. (a)

   two unique information atoms, one for each source variable, which capture the information each source variable implies about the target variable, that cannot be inferred from the other;

2. (b)

   one redundant information atom, which captures the information that can inferred about the target variable from either one of the source variables; and

3. (c)

   one synergistic information atom, which captures the information that can be inferred about the target variable from both source variables jointly, but not individually.

Ref. [3] proposed a set of axioms that the above information atoms should satisfy in order to provide said insights, and follow-up works in the field identified several additional properties [4, 5, 6, 7]. Yet, in spite of extensive efforts [8, 9, 10, 11, 12], a comprehensive definition of information atoms which satisfies all these axioms and properties is yet to be found.

In spite of limited understanding of the information atoms, PID has already found multiple applications in various fields. As a simple example [13, Fig. 1], one can imagine the two source variables being education level and gender, and the target variable being annual income. An exact formula for computing the information atoms would shed insightful information about the extent to which annual income is a result of education level, gender, either one, or both.

Beyond this simple example, PID has broad applications in a wide range of fields. In brain network analysis, PID (or similar ideas) has been instrumental in measuring correlations between neurons [14] and understanding complex neuronal interactions in cognitive processes [15]. For privacy and fairness studies, the synergistic concept provides insights about data disclosure mechanisms [16, 17]. In the field of causality, information decomposition can be used to distinguish and quantify the occurrence of causal emergence [18], and more.

In this paper, we propose an explicit PID formula that satisfies all of Williams and Beer’s axioms, as well as several additional desired properties. We do so by introducing the do-operation, which is inspired by similar concepts in the field of causal analysis [19, 20, 21]. Intuitively, based on the understanding that unique information is “ideal conditional mutual information,” our method first adjusts the entire probability distribution by using the do-operation in order to make the target variable identical to its conditional distribution given one source variable, and then calculates the expectation of mutual information between it and the other source variable under different conditions. It is worth noting that our method is not based on any of the point-wise, localized, or optimization approaches that existing methods use.

We begin in Section II by introducing the PID framework, and its axioms and properties. We continue in Section III by introducing our do-operation and the definition of unique information, from which all other definitions follow, and prove that all axioms and properties are satisfied. We discuss the intrinsic meaning of our definition in Section IV, and provide all proofs in the appendix.

The following notational conventions are observed throughout this article: $X,\mathcal{X},x$ (similarly $Y,\mathcal{Y},y$ etc.) denote a random variable, its corresponding (finite) alphabet, and an element of that alphabet, respectively. The distribution of $X$ is denoted by $\mathcal{D}_{X}$, the joint distribution of $X$ and $Y$ is denoted by $\mathcal{D}_{X,Y}$, and the distribution of $X$ given $Y=y$ is denoted by $\mathcal{D}_{X|Y=y}$.

For random variables $X,Y,Z$, the quantity $I((X,Y);Z)$ captures the amount of information that one target variable $Z$ shares with the source variables $(X,Y)$, but provides no further information regarding finer interactions between the three variables. To gain more subtle insights into the interactions between $Z$ and $(X,Y)$, [3] proposed to further decompose $I((X,Y);Z)$ into information atoms. Specifically, the shared information between $Z$ and $(X,Y)$ should contain a redundant information atom, two unique information atoms, and one synergistic information atom (see Figure 1).

The redundant information atom $\operatorname{Red}(X,Y\to Z)$ (also called “shared”) represents the information which either $X$ oder $Y$ imply about $Z$. The unique information atom $\operatorname{Un}(X\to Z|Y)$ represents the information individually contributed to $Z$ by $X$, but not by $Y$ (similarly $\operatorname{Un}(Y\to Z|X)$). The synergistic information atom $\operatorname{Syn}(X,Y\to Z)$ (also called “complementary”), represents the information that can only be known about $Z$ through the joint observation of $X$ and $Y$, but cannot be provided by either one of them separately. Together, we must have that

We refer to (1) as Partial Information Decomposition (PID).

Moreover, since the redundant atom together with one of the unique atoms constitute all information that one source variable implies about the target variable, it must be the case that their summation equals the mutual information between the two, i.e., that

In a similar spirit, the synergistic information atom and one of the unique information atoms measure shared information between the target variable and one of the source variables, while excluding the other source variable. Therefore, the summation of these quantities should coincide with the well-known definition of conditional mutual information, i.e.,

Eqs. (2), and (3) are the foundation of an axiomatic approach towards an operational definition of the information atoms. These equations form the first in a series of axioms, presented next, which were raised in previous works on the topic [3, 22, 23]. Such axiomatic approach was also taken in the past in order to shed light on Shannon’s mutual information [24].

Notice that it suffices to specify the definition of any one of the information atoms, and the definitions for the remaining atoms follow from Axiom 1. Consequently, [3, 25] chose to specify $\operatorname{Red}$, and provided three additional axioms which $\operatorname{Red}$ should satisfy.

The first additional axiom is commutativity of the source variables, which implies that the order of the source variables must not affect the value of the redundant information.

The second is monotonicity, which implies that the redundant information is non-increasing when adding a source variable, since the newly added variable cannot increase the redundancy between the original variables. We sidestep the discussion about monotonicity with more than two variables, which is not our focus in this paper, even though it can be easily obtained by extending our definition to more than two source variables.

The third is self-redundancy, which defines the redundant information from one source variable to the target variable (i.e., $\operatorname{Red}(X\to Z)$) as the mutual information between them. In the case of two source variables considered herein, monotonicity and self-redundancy merge into the following single axiom.

Notice that Axiom 3, alongside Axiom 1 (specifically (2)), imply that $\operatorname{Un}$ is a nonnegative quantity. The nonnegativity of $\operatorname{Red}$ is stated in [3, 25] as a separate axiom, shown next.

The nonnegativity of $\operatorname{Syn}$ is normally not listed as an axiom, since it is debatable if it should or should not be nonnegative; we will show that our method yields nonnegative $\operatorname{Syn}$ under the closed-system assumption (i.e., $H(Z|X,Y)=0$) in Section III, and further discussion is given in Section IV.

Besides, subsequent to [3, 25], studies suggested two additional properties, additivity and continuity [12, 26]. Additivity implies that whenever independent variable systems are considered, the joint information measures should be the sum of the information measures of each individual system. This is the case, for instance, in joint entropy of two independent variables.

Continuity implies that small changes in the probability distribution lead to small changes in the value of the information measure. It ensures that the measure behaves predictably and is a key property in information theory, particularly for measures like entropy and mutual information.

In addition, another well-known property is independent identity [9], which asserts that in a system of two independent source variables and a target variable which equals to their joint distribution, the redundant information should be zero.

We mention that several important properties can be inferred from the above. For example, the non-negativity of $\operatorname{Un}$ can be obtained from Axiom 1 and Axiom 3 as mentioned earlier; the commutativity of $\operatorname{Syn}$ follows from Axiom 1 and Axiom 2; the difference between (2) and (3) is often called consistency [12], etc.

Finally, we emphasize once again that none of the existing operational definitions of the information atoms satisfy all of the above. A comprehensive list of violations is beyond the page limit of this paper, and yet we briefly mention that Axiom 4 (nonnegativity) is violated by [8, 9, 27] (although some sources do not refer to non-negativity as a requirement); Property 1 (additivity) is violated by all works except [12],  [28],  [8], and [29] according to [26]; Property 3 (independent identity) is violated by [3]; Property 2 (continuity) is violated by [11, 8], [28], [29], etc.

In this section, we present our operational definition of $\operatorname{Un}$, from which the definitions of the remaining information atoms follow. Then, we explain the logic behind this definition, and prove that it satisfies all the axioms and properties proposed in Section II.

In this paper, we proposed an explicit operational formula for PID, which is distinct from any existing approach, and proved that it satisfies all axioms and properties. In this section we provide further intuitive explanation for our approach.

First, we wish to elucidate the role that our do-operation plays in the definition of $\operatorname{Un}$ (Definition 5). In a sense, the do-operation can be understood as adjusting the marginal distribution of the $Z$ variable of $\mathcal{D}_{X,Z}$, while impacting its connections with other variables as little as possible. This understanding can be confirmed by Lemma 1, which shows that the expected value of the conditional entropy after the do-operation retains its original value. This resembles the invariance implied in Shannon’s communication model [1], where the conditional entropy of the output given the input is not affected by the input distribution. From this perspective, $Z$ and $X$ can be regarded as the input and output of the channel, respectively, that indicates their “relationship.” The do-operation changes the distribution of the input $Z$, but does not change the channel’s characteristic (i.e., $H(X|Z)$).

Based on this, Definition 5 realizes the intuition that unique information should represent the relationship between source variable and target variable given the other source variable. So, we use the do-operation to control the marginal distribution of the target variable $Z$ to its conditional distribution given the value $y$ of some source variable $Y$, and then use the expectation of mutual information $\sum_{y}\Pr(Y=y)I(A_{y};C_{y})$ to capture the “connection” between the specific source variable $X$ and target variable $Z$ given $Y$ after the do-operation.

The reason this method can partition $I(X;Z|Y)$ to $\operatorname{Syn}$ and $\operatorname{Un}$ (Def. 3), is that the do-operation eliminates high-order relations between $Y$ and $X,Z$, i.e. $\operatorname{Syn}$. Specifically, conditional mutual information relies on the joint conditional probability $\mathcal{D}_{(X,Z)|y}$, in expectation over all $y\in\mathcal{Y}$. This distribution includes both the conditional influence of $Y$ on $X,Z$, but also has a simultaneous influence on the relationship between $X$ and $Z$.

However, Definition 5 of unique information retains the relationship between $X$ and $Z$ without influence from $Y$ by using the conditional probability $\mathcal{D}_{Z|y}$, in expectation over all $y\in\mathcal{Y}$, to perform the do-operation, which only reflects the conditional influence of $Y$ on $Z$. Therefore, the expectation of mutual information $\sum_{y}\Pr(Y=y)I(A_{y};C_{y})$ can accurately quantify the unique information, which represents the pure conditional mutual relationship.

In addition to the above analysis of do-operations in unique information, Lemma 2 also brings another perspective worth discussing. Redundant information can be understood as the mutual information $I(X^{\prime};Y)$ (or $I(X;Y^{\prime})$) obtained by changing the joint probability distribution $\mathcal{D}_{X,Y}$ according to $\mathcal{D}_{X,Y,Z}$ without changing the marginal distribution $\mathcal{D}_{X}$ (or $\mathcal{D}_{Y}$) according to Lemma 1 ($H(X^{\prime})=H(X)$).

As mentioned earlier, our definition of $\operatorname{Syn}$ might be negative, unless the system is closed (i.e., $H(Z|X,Y)=0$, Remark 1). While $\operatorname{Un}$ and $\operatorname{Red}$ represent the information shared by one or two source variables with the target variable, $\operatorname{Syn}$ represents the information provided to the target variable by the “cooperation” of source variables. It is an accepted aphorism that cooperation does not necessarily increase outcome, and hence it might be the case that negative values of $\operatorname{Syn}$ conform with intuition. However, the reason why this explanation is no longer necessary in a closed system, as well as alternative interpretations of $\operatorname{Syn}$ that are nonnegative, remain to be studied.