1. Introduction
The Renormalization Group (RG) is used in physical settings to deduce how a theory changes when it is viewed at different
scales. In Wilsonian RG, one regards the set of possible theories as a space coordinatized by a collection of coupling constants specifying all of the possible contributions to a classical action. An RG flow can then be understood as a one-parameter family, or
flow, in the space of theories generated by a vector field on theory space referred to as the
Beta Function [
1]. The flow is completely specified once the requisite initial data are provided, for example, fixing a theory in the ultraviolet (UV) from which the RG flow begins (by fixing a theory in the UV, we mean a theory that is a priori valid at all energy scales).
A typical approach to RG is to deduce the beta function by sequentially coarse-graining the theory—that is, integrating out degrees of freedom that become suppressed at lower energy scales. In field theoretic contexts, this approach makes use of perturbative techniques in order to perform the requisite functional integrals. From a formal perspective, however, it is possible to study the properties of renormalization as an abstract flow equation without immediately concerning ourselves with any complicated or even intractable calculations that may be required to realize the flow explicitly. This approach to RG goes under the name of the Exact Renormalization Group (ERG) and will be the primarily focus of our paper.
In our approach to ERG, we shall regard a Quantum Field Theory (QFT) as equivalent to the specification of a probability distribution on the space of fields included in the theory, which we denote by and refer to as the sample space in standard probability theoretic nomenclature (to accommodate this perspective, we shall consider only Euclidean QFT). From this perspective, the space of theories is isomorphic to the space of probability distributions on the sample space , denoted by . An ERG flow is therefore equivalent to a one-parameter family of probability distributions on .
From this point of view, we can still make contact with the Wilsonian picture by regarding a Wilsonian Effective Action, written in terms of a collection of coupling constants, as specifying a parameteric family of probability distributions. In other words, Wilsonian RG corresponds to a particular coordinatization of the space .
Following the lead of [
2,
3,
4] and others, it is natural to interpret the one-parameter family of probability distributions generated by an ERG flow as being governed by a functional
convection–diffusion equation on the sample space
. Regarding ERG as a flow on the space of probability distributions over a given sample space contextualizes renormalization in a language that is amenable to applications outside of the usual realm of physical theory. In particular, it suggests a manifestly information-theoretic interpretation for renormalization. Uncovering and presenting the details of this interpretation is the primarily objective of this note.
In seeking an answer to the question, “What is the information-theoretic Interpretation of an ERG flow?” we find it useful to consider a related question: “What does it mean to
invert an ERG flow?”. Here, our understanding of ERG as being governed by a diffusion equation provides us with a direction. A powerful method for formally inverting a diffusion process is
Bayesian Inversion [
5]. Bayesian Inversion is a probabilistic approach used to determine the initial data that was fed into a partial differential equation and subsequently generated an observed sequence of outcomes. As the name implies, the main tool employed in Bayesian Inversion is Bayes’ Law. This lends some credence to the idea that Bayesian Inference may serve as an “inverse process” to ERG.
In our previous publication [
6], we introduced
Dynamical Bayesian Inference, which recasts the Bayesian Inference as a dynamical system. Like ERG, Dynamical Bayesian Inference generates a one-parameter family of probability distributions satisfying a flow equation. Whereas in ERG, the flow equation is deduced by sequential application of a coarse-graining law, in Dynamical Bayes the flow equation is obtained by the sequential application of Bayes’ law with respect to a continuously growing set of observed data. In this respect, an ERG flow continuously loses information (hence, it is
diffusive), while a Dynamical Bayesian flow continuously gains information.
In this note, we shall argue in favor of this interpretation. In particular, we will demonstrate that the Dynamical Bayesian Flow equation gives rise to a convection–diffusion equation describing the evolution of an associated posterior predictive distribution backwards against the collection of additional data. We refer to the process described by this equation as Bayesian Diffusion, or Backward Inference. We interpret the equation governing Bayesian Diffusion as defining an ERG flow, with a coarse-graining procedure given by the continuous discarding of observed data. By construction, this ERG is inverted by the forward Dynamical Bayesian inference flow that is obtained by reincorporating the lost data back into the model. Alternatively, starting from an ERG flow we identify the Dynamical Bayesian flow for which it is the Backward Inference process by drawing a correspondence between the partial differential equation governing ERG and the partial differential equation governing Bayesian diffusion. Ultimately, this correspondence suggests a fascinating information-theoretic interpretation for ERG: it can be regarded as the one-parameter family of probability distributions obtained by starting from a data-generating model and continuously throwing away information in the form of observed data.
The organization of the paper is given as follows: in
Section 2, we review the formulation of ERG as a functional differential equation, specifically in the Wegner–Morris form. We also demonstrate that the Wegner–Morris equation is equivalent to a Fokker–Planck equation with a given potential function, establishing the correspondence between ERG and optimal transport. In
Section 3, we introduce Stochastic Differential Equations (SDEs) and discuss the relationship between SDEs and partial differential equations of the type appearing in ERG. In
Section 4, we review Dynamical Bayesian Inference and derive the Bayesian Diffusion equations. Finally, in
Section 5, we establish the correspondence between Bayesian Diffusion and ERG explicitly. We conclude with
Section 6, in which we review our findings and discuss future research directions.
4. Dynamical Bayesian Inference and the Backward Equation
The formulation of the Exact Renormalization Group flow as an Optimal Transport problem, particularly in its relationship to the extremization of relative entropy, already suggests a deep connection between renormalization, diffusion, and information theory. We now turn our attention to the task of fleshing out this relationship and, in doing so, providing an information-theoretic conceptualization of renormalization through the intermediary of statistical inference. To accomplish this task, we will build on the language of
Dynamical Bayesian Inference introduced in [
6]. We will show that the dynamics of an inferred probability model defined by a continuously updating Bayesian scheme give rise to an
inverse process governed by a diffusion equation that can be brought into correspondence with ERG. Using this fact, we advocate for the perspective that renormalization can be understood as the
inverse process of statistical inference.
We should emphasize that inversion is understood in the statistical sense as a ‘reverse time’ stochastic process relative to the Kolmogorov forward process defined by ERG [
36]. Such stochastic processes have recently appeared in diffusion learning, where the reverse time Stochastic Differential Equation relative to a forward diffusion process is modeled via a score-based generative algorithm [
37]. From the point of view developed in this section, a score-based generative algorithm can be interpreted as implementing a form of Bayesian inference. We will revisit this and other related points in
Section 6.
4.1. Bayesian Inversion
Bayesian inference is a probabilistic, self-consistent approach to adjusting one’s beliefs in the presence of new information. It originates from Bayes’ Law, which encodes the probabilistic relationship between two, potentially co-varying, random variables
and
:
In Bayesian Inference, the variables
H and
E are respectively identified with the
Hypothesis and the
Evidence. In this language, one can attach a compelling geometric interpretation to Bayes’ law as measuring the relative volumes of the region
, for which both
H and
E are realized, and the region
, for which the observed evidence is true. Thus, the interpretation of Bayesian Inference is that it measures the volume of hypotheses that is consistent with a given set of observed data, which we denote as the region
.
A particularly interesting form of Bayesian Inference arises in the context of so-called Bayesian Inversion problems [
5,
38,
39,
40,
41,
42,
43]. There, we are concerned with a pair of random variables,
and
, which are related by the following process:
We regard
as a deterministic function, denoted by
noise, which can either be associated with the process itself or the process of measuring its output.
The goal of the Bayesian Inversion problem is to determine the signal, u, that led to the measured output, y. However, it is often impractical to truly invert the process; thus, it is more prudent to seek to place a probability distribution over the inputs.
To this end, we regard
as having a prior measure
and treat the noise
N as a random variable independent from
Y also possessing of a measure
. Then, we can define the conditional random variable
as:
If we regard
as a map from output to noise, such that
we can obtain a conditional measure for
by pulling back the measure
via the map
:
We regard the pulled-back density
as the
likelihood function for an inference problem and often denote it by
. We therefore obtain a joint measure:
for the random variable
.
Provided the marginalization:
is greater than zero and finite, we can define the Bayesian posterior measure as:
which is just a recapitulation of Bayes’ law as written in the form (
91). The notation
is meant to remind us that the posterior distribution depends on the realization of observed data
. We can also recognize
as the marginal density for the random variable
Y modulo the prior distribution
.
Next, we define the
Bayesian Potential as:
which is nothing but the negative log likelihood. With this potential in hand, we can write the Bayesian Inference condition suggestively as:
On the left hand side, we have the
Radon–Nikodym derivative of the posterior measure with respect to the prior measure. On the right hand side, we have what we would like to interpret as the stationary distribution modulo observations
y. This equation can be interpreted as follows: given any measurable function
meaning that the expectation value with respect to the posterior measure is the same as the expectation value with respect to the prior provided the function is augmented by the evidence in the form of the stationary distribution
.
4.2. Dynamical Bayes
Dynamical Bayesian Inference is an extension of the conventional approach to Bayesian Inference in which one implements Bayesian inference as an iterative process where swathes of evidence are collected, and the posterior distribution at the end of a given iteration is used as the prior distribution in the following iteration [
6].
Using such an approach, one can regard Bayesian Inference as a dynamical system governed by a first-order differential equation. To begin, we define a timelike variable,
T, which essentially corresponds to the total number of data point observed. Inference up to the “time”
T therefore leverages evidence from the set of observed data
, which we can regard as a continuous time stochastic process coming from the data-generating measure
. In what follows we shall regard the data-generating measure as belonging to the parametric family
, associated with the true underlying signal value
. That is,
. The equation satisfied by
is then given by:
where
is the KL-Divergence between a distribution
p on
and the data-generating distribution
. As an aside, we note that (
102) is equivalent to the replicator dynamic which appears in evolutionary game theory provided the fitness of a particular model
is taken to be minus its KL-Divergence with the data-generating model. We refer the reader to [
44,
45] for more detail.
Among the most significant insights of this approach is that, if we consider models that are within an
neighborhood of the true underlying signal, the
-point functions described by
are very well approximated by power laws:
Here,
is the inverse of the Fisher Metric arising from the family of distributions
, evaluated at the data-generating parameter
.
4.3. Bayesian Diffusion for Normal Data
To illustrate the Dynamical Bayesian Inference dynamic, it will be beneficial to work through the problem of performing Bayesian inference on the mean of normally distributed data with known variance,
. In this case, we take:
where
n is noise distributed according to a distribution
:
and
is a random draw from the distribution over means for the data
y so that
(for notational simplicity, we shall denote this distribution by
). The data-generating distribution is taken to belong to the same parametric family of distributions, only with a fixed but unknown “true” underlying mean parameter
-
.
The governing equation of Dynamical Bayesian Inference can be solved formally as:
Here, we have used an abbreviated notation in which
is the KL-divergence between two distributions of the same paramteric form with given parameter values
u. Notice that only the first exponential depends on the variable
u; hence, we conclude that the role of the second exponential is simply to maintain the normalization of
as a probability distribution. Thus, we can write:
where
In the case of the normal model with fixed variance, the KL-divergence is given by
. The
T-Posterior is therefore given by:
It is straightforward to determine the normalization of this distribution by performing the requisite integral that is now Gaussian. When all is said and done, the
T-dependent posterior density is of the form:
Let us now compare (
111) to the standard density of a length
t increment of a Weiner process with diffusivity parameter
:
Recall that the density
solves the heat equation:
Following the analysis of
Section 3, one can also recognize the stochastic process
as a martingale adapted to the Weiner process
We therefore recognize (
111) as describing a shifted Brownian motion for the mean parameter with diffusivity
in the “time” parameter
:
This observation lends credence to the idea that Bayesian inference can be associated with a diffusion process. Moreover, it provides an important insight: Bayesian Diffusion ensues backwards with respect to the performance of Bayesian Inference, in the timelike parameter, , which is the inverse of the Bayesian time, T, originally introduced.
Given the
-posterior
, and terminal data such as the parametric form of the data-generating distribution, one can obtain the
path of the posterior predictive distribution for future data by marginalizing. For the normal model, this means:
which we can recognize as the convolution of
and
and interpret as the action of the Green Function of the Heat Operator translating the model forward in
(and backward in
T).
4.4. Bayesian Drift and Scheme Independence
We have now shown that the solution to the Dynamical Bayesian inference equation, (
111), is also the solution to the standard diffusion equation when viewed as transforming
backwards relative to the update time,
T. As we shall now discuss, we can promote (
111) to a solution to a
drift-diffusion equation by an analogous argument to the one that appeared in
Section 2.3. In particular, we argue that drift in the context of Bayesian diffusion is also associated with a redundancy of description, in this case related to the specific sequence with which data are observed.
The reasoning behind this argument is most easily understood through an example. Suppose again that one is performing Bayesian inference on a system is tthataken to follow a normal distribution with known variance,
, but an unknown mean. To deduce the mean of the distribution governing the system, we observe a sequence of
N-independent, identically distributed random draws from the true underlying distribution,
. Starting with a normal prior, one finds that the mean of the posterior distribution after observing the first
n pieces of data shall be given by:
Let
be a permutation, and let
denote the same set of evidence but in a new order defined by
. We interpret this transformation as changing the
sequence in which the data are obtained. Crucially,
is still a set of
N-independent, identically distributed random variables drawn from the same data-generating distribution. If we compute the maximum likelihood estimate based on the first
n piece of data appearing in
, we find:
and hence it is true that
. However, if we compute the maximum likelihood estimate on account of all of the data contained in either set, we find:
That is, the terminal maximum likelihood estimate is
invariant with respect to the
sequence in which the data are incorporated into the model.
We extrapolate this observation to the statement that the posterior distribution can be assigned an arbitrary path through the space of probability models, provided the terminal distribution remains consistent with the large observation limit, that is, the central tendency towards the data-generating distribution. This is completely analogous to the role of drift in defining an ERG scheme: the invariant definition of an ERG flow is given in terms of the IR fixed point it describes, and the path through the space of theories by which the theory moves from the UV to the IR is scheme-dependent.
Operationally, we make use of the sequencing freedom in the Bayesian inference to “seed” the flow with information about the individual inference trajectory by specifying the
path of the maximum a posteriori estimate (MAP). In the general case where we are given a set of signal parameters
, we specify the trajectory of the MAP as a flow on the manifold
generated by a vector field
. That is,
such that
or, in more standard notation,
The trajectory of the MAP arises from maximizing the log-likelihood of the data-generating model. Thus, in many cases the MAP path,
will realize a gradient descent:
where
are the matrix components of the inverse Fisher metric, and we have implemented the chain rule to evaluate the derivative. It is very crucial to note that the gradient descent here ensues in the direction of
increased data observation, that is, as
,
approaches the data-generating parameter value, as desired.
4.5. Generic Bayesian Diffusion at Late T
The Normal Model is significant because it arises as the late
T limit of
any Dynamical Bayesian Inference scheme for which the parameters of the data-generating distribution are equal to some fixed values. This observation arises naturally as an asymptotic limit of the solution to (
102) and is a statement of the Central Limit Theorem. At late
T, the KL-Divergence can be approximated by the quadratic form:
meaning we can write the unnormalized posterior distribution as:
Provided
does not depend on
u, the normalization is obtained by performing a Gaussian integral, and we can write the posterior distribution as:
Here, we have changed the variables to
in order to observe that this is the distribution of a multi-variate Brownian motion.
Using the gauge freedom discussed in the last section, we can promote this solution to one of the forms:
where
is the trajectory of the MAP. As advertised, both (
126) and (
127) are consistent with the late
T statistics (
104).
4.6. A Partial Differential Equation for Bayesian Inference
In light of the previous sections, we shall now show that one can derive a convection–diffusion equation describing the evolution of the posterior predictive distribution when it is updated according to Bayes’ law. As we have come to recognize, the sense in which Bayesian inference describes a diffusion process is in moving
backwards relative to the observation of new information. We will therefore work in the time parameter
, defined as the inverse to the time parameter
T that tracks the amount of data observed. Given the time
posterior distribution, which we have argued is of the form of a modified heat kernel, we can define the time
posterior predictive model for future data
by marginalizing over the likelihood model:
In
Section 3, we reviewed the relationship between the solution to a convention-diffusion equation and the convolution of given boundary value data with a heat kernel. Comparing (
128) with (
86), it is natural to regard the posterior distribution as a Markovian transition kernel measuring the probability of going from
to
. From this perspective, we define the set of operators
such that:
Following the approach outlined in (
87), we can now deduce the diffusion equation generated by the semi-group
. By Taylor Expansion, we can compute:
Here, we regard
as corresponding to the infinitesimal flow of the parameter
in terms of the vector field
generating the flow of the MAP. That is,
. Following this interpretation:
We can now use (
104) to compute the expectation values explicitly to the order specified in the expansion. Because we are describing the diffusion in terms of the variable
and we shall eventually be taking the limit as
, we can use the
results described in Equations (
104) and (
127). In particular, we make use of the fact that
Meaning, we can write:
We can rewrite these terms in a slightly more illuminating way by writing:
Moreover, if we assume the MAP follows a gradient flow with respect to the log-likelihood, we can also write:
Putting everything together, we have therefore shown that:
Thus, taking the limit
we obtain the result:
This is precisely of the form of the Kolmogorov equation for a diffusion process with potential
!
It is tempting to interpret the matrix
as being associated with the induced Fisher metric in the space of probability distributions along the path of the MAP. From this perspective, we can regard
, the random variable associated with the
T-dependent posterior predictive distribution, as a stochastic process on a curved space specified by the Stochastic Differential Equation:
The potential is given by
, which is minus the log-likelihood associated with the time
maximum likelihood estimate for the generating distribution. Thus, we have shown that a Dynamical Bayesian inference induces a gradient flow with respect to the log-likelihood of the data-generating distribution.
5. The ERG Flow/Dynamical Bayesian Inference Correspondence
We are now prepared to build the dictionary relating ERG flow and Bayesian Inference. For simplicity, we shall consider one-parameter families of probability distributions on finite dimensional sample spaces; however, it is a simple exercise to generalize these insights to the infinite dimensional case as well. To begin, let us review the work we have presented in the previous sections.
In
Section 2, we recalled the Wegner–Morris formulation for ERG and demonstrated that it is equivalent to a one-parameter family of probability distributions described by a gradient flow with respect to the relative entropy:
In this equation, we interpret the time parameter
t as a logarithm of the scale –
. The distribution
, with
for the potential
V, can be regarded as specifying the ERG scheme through the fixing of a stationary point along the flow generated by (
139). The choice of
V defines the functional,
, and the reparameterization kernel,
; hence, it is equivalent to a choice of scheme in the standard Wegner–Morris sense. As we have reviewed in
Section 3, the Fokker–Planck equation is equivalent to the Kolmogorov Forward equation for the Stochastic Process governed by the Stochastic Differential Equation:
This is intuitively satisfying since such an equation describes a stochastic gradient descent of the potential function
V. Once initial data are supplied in the form of a UV theory,
, (
139) completely describes an ERG flow terminating at an IR fixed point.
In
Section 4, we introduced the notion of Dynamical Bayesian Inference. Dynamical Bayesian Inference describes a one-parameter family of probability distributions obtained by implementing Bayes’ law using data collected from a continuous time stochastic process. To quantify this family of distributions, we introduced the “time” parameter,
T, which corresponds to the number of data incorporated into the model. In the direction of increasing
T, the inferred probability model converges onto the distribution generating the observed data. In this respect, a Dynamical Bayesian flow has the complexion of an
inversion in the ERG sense because it begins with an uninformed prior distribution but eventually converges to an informed distribution. In the language of ERG, this describes a flow from an IR theory to a UV theory. The pair consisting of an uniformed prior, along with the specification of a sufficiently complete set of data, therefore defines a flow terminating with a UV theory. We take this as our definition of a Dynamical Bayesian Flow.
This picture suggests that Dynamical Bayesian Inference and ERG flow are inverse of each other. If we can find a Dynamical Bayesian inference that begins with a prior distribution equal to the IR fixed point of an ERG flow, and which terminates at a data-generating distribution equal to the UV initial data of said ERG flow, we will have obtained an inversion of the ERG flow. Let us now describe a strategy for determining pairs of ERG flows with Dynamical Bayesian Flows.
5.1. Dynamical Bayesian Flow → ERG Flow
Suppose we are given a Dynamical Bayesian flow and asked to determine an ERG flow that is inverse to it. Recall, we have defined a Dynamical Bayesian Flow as the pair, , where is an uninformed prior distribution and is a set of data generated by a distribution . Using these data, it is straightforward to define the corresponding ERG flow as the one-parameter family of probability distributions obtained from the Dynamical Bayesian flow when viewed as evolving backwards with respect to the time parameter T. In other words, one takes the terminal distribution of the Dynamical Bayesian flow, , as defining the UV initial data of the ERG flow and obtains subsequent probability distributions along the ERG flow by removing items of data from the inferred model.
In
Section 4.6, we derived a partial differential equation governing just such a situation, in which the posterior predictive distribution evolves
backwards against the collection of new data. The resulting partial differential equation describes a diffusion process that we dubbed
Bayesian Diffusion: that is, one obtains a one-parameter family of probability distributions
such that
and for which:
This differential equation describes the evolution of a probability model governing the stochastic process
, which itself is governed by the Stochastic Differential Equation
Here,
is a trajectory in the model space of the theory describing the path of the maximum a posteriori parameter estimate generated by the sequence of observed data,
K is the pullback of the Fisher Information Metric on the space of models defined in (
134), and
is the log-likelihood function associated with the Bayesian Inference scheme.
Together, the aforementioned items constitute a
choice of scheme for the Dynamical Bayesian flow. Bayesian diffusion can therefore be associated with an ERG flow governed by the Wegner–Morris equation:
where now the stationary distribution, scheme function, and reparameterization kernel are, respectively, given by:
These data, along with the distribution
, define an ERG flow in the Wegner–Morris sense, which, by construction, is the inverse of the Dynamical Bayesian flow we began with.
5.2. Dynamical Bayesian Flow ← ERG Flow
Conversely, suppose that we are given an ERG flow in a Wegner–Morris form and asked to determine a Dynamical Bayesian flow that is inverse to it. To do so, we first identify the Bayesian Diffusion process that the ERG flow corresponds to. Since we have shown that ERG and Bayesian Diffusion are governed by the same equations, this is as simple as translating between the ERG scheme and the Dynamical Bayesian scheme. Comparing (
139) and (
143), we deduce that the ERG associated with an optimal transport with potential
V and sample space metric
g is equivalent to a Bayesian diffusion in which
V is taken as the log-likelihood function, and
g is identified with the metric
K. These data are sufficient to define a Bayesian inference problem. Notice, if we go all the way back to (
13), we can finally provide a conceptual understanding of the seed action: the seed action sets the log-likelihood for the Bayesian Inference scheme related to the ERG flow it defines.
5.3. A Dictionary
We summarize the analysis of this final section in
Table 3. It provides a dictionary translating between the Wegner–Morris equations relevant to finite sample space ERG, infinite dimensional sample space ERG, and Bayesian Diffusion.
5.4. Renormalizability and Scale
The dictionary in
Table 3 provides a blueprint for interpreting ERG in the language of Statistical Inference. In this role, our work suggests new approaches to understanding and resolving many interesting problems inside and outside of field theory. As a demonstration, we shall use this final section to discuss the meaning of renormalizability in the context of an ERG flow related to the Bayesian Inference paradigm.
An important observation in (
3) is the correspondence between the Fisher Metric in the inference context and
in the ERG context. In the exact renormalization of a free field theory,
is the regulated two-point function and therefore sets a running momentum scale for operators in the theory. The Fisher Metric,
, plays an analogous role in the Bayesian Inference scheme as a generalized two-point function encoding a notion of
scale through the covariance between operators.
The interpretation of the Fisher Metric as defining an energy scale is made very clear when we consider the inverse Bayesian flow as a diffusion process. Let
denote the manifold of probability distributions over a sample space
S. Then, a Bayesian diffusion, or equivalently an Exact Renormalization Group flow, can be described by a drift-diffusion process generated by an operator
, such that:
Given initial data (for example, in an ERG we would provide a
theory)
, we can write the solution to (
145) symbolically as:
where
is the heat kernel of
L. To give more concrete meaning to (
146), let us assume that
L is a positive definite operator that can be diagonalized as
Here,
is a countably infinite set forming a basis for
, and
is non-negative but may be equal to zero. An arbitrary element
can be expanded as a series:
, where
is the coordinate of
p in the eigendirection
.
The spectrum of
L defines an
emergent energy scale in the following sense: consider the action of (
146) as given by:
(
148) dictates that the projection of
onto each mode,
, is damped over time with a strength determined by on the “energy”
. At time
t, the effective description exponentially suppresses modes that have large eigenvalues relative to the operator
L and thus sequentially removes these modes in a generalized integration over effective “momentum shells”. From Equation (
137), we can see that the operator
L is generically a convection–diffusion operator with a diffusion matrix given by the Fisher Information. Through further comparison to (
6), we see that in ERG for physical theories the analogous role is played by the regulated two-point function. This leads to the important conclusion that, in physical contexts, the emergent energy scale is in fact equivalent to a physical one.
More generally, interpreting the two-point function, or the covariance matrix, in a statistical inference problem as generating an emergent energy scale provides the foundation for an information-theoretic interpretation of the conditions for renormalizability. In Wilsonian RG, a theory is said to be non-renormalizable if the divergences present in higher-order Feynman diagrams can only be canceled by the introduction of an infinite number of arbitrarily high-energy couplings [
46]. By contrast, a theory is said to be renormalizable if arbitrarily higher-order Feynman diagrams can be computed by introducing only a finite number of operator sourced counterterms.
In the language of statistical inference, the operator content of a theory is related to the problem of model selection [
47,
48]. In the context of parametric statistics (as we have mentioned in the introduction, the relationship between Wilsonian RG and ERG is analogous to the relationship between Parametric and Non-Parametric statistics), model selection can be reduced to determining the set of sufficient parameters needed to form a model that can accurately compute the expectation values of any observable associated with the system of interest. A natural framing of this problem is given in terms of
n-point correlation functions for the random variable,
Y, observed throughout the inference. It is sensible to restrict our attention to
n-point functions since arbitrary observables can be constructed from them using a Taylor expansion. Put differently, a probability distribution can be reconstructed with knowledge of all its moments.
In view of the previous discussion, higher n-point functions can be interpreted as encoding information at higher values of the emergent energy scale. This inspires the interpretation that an inference model is “renormalizable” if a finite N exists such that for any , the n-point function can be computed with the information contained only in m-point functions with . In other words, an energy as measured through L, exists, above which all of the information in the theory is actually encoded in lower energy operators. This happens, for example, in a Gaussian theory in which all n-point functions higher than can be formulated as sums of products of 2-point functions using Wick’s theorem. If no such finite N exists, the inference problem is “nonrenormalizable”. A nonrenormalizable theory can therefore be understood as a theory in which an infinite number of n-point functions will be required to compute the expectation values of arbitrary observables. In other words, there is no energy scale above which information becomes encoded in the energy scales below it—every energy scale contributes, in some sense, independently to the theory.
6. Discussion
In this note, we have demonstrated that an ERG flow can be identified with a diffusion process that is
inversely related to a Dynamical Bayesian Inference scheme. In particular, we have argued that ERG flow can be understood as a one-parameter family of probability distributions arising where data are continuously removed from the inferred probability model. We have motivated this interpretation by illustrating that the equations governing ERG and Bayesian Diffusion can be brought into direct correspondence with one another, as outlined in
Section 5. The resulting dictionary provides a novel, fully information-theoretic language for understanding ERG flow. It also provides an operational answer to the question of what it means to “invert” an ERG flow.
From a very general perspective, the solution to this problem can be framed in the following way. Given a preliminary probability distribution,
, we imagine running our model through a noisy channel generated by a diffusion operator
B. In other words, we produce a probability distribution,
, which solves the differential equation:
with initial data
. After a given period of time,
t, we obtain a new probability distribution,
, which has lost some of the information previously contained in
to diffusion. In the case of an ERG flow viewed from the functional diffusion perspective, we can regard this loss of information as being generated by a coarse-graining scheme encoded in the operator
B.
We then ask the question, can this diffusion process can be “inverted”? Since exact inversion may not be possible, we frame this problem in the form of an optimization scheme. Consider the set
consisting of all operators
F, generating a one-parameter flow of probability distributions,
, such that:
subject to an initial condition
. If we take the initial data of this process to be the terminal distribution of the diffusion process given by (
149),
, we can interpret the solution
as a
reconstruction algorithm for the initial data,
. It is natural to identify the optimal reconstruction algorithm as the operator
, for which the relative entropy between the reconstructed distribution,
, and the initial data,
, is minimal:
In this language, we interpret the main result of our paper as dictating that, given an ERG generated by a diffusion operator B, the optimal reconstruction operator corresponds to a continuous Bayesian Inference scheme in which the information lost to coarse-graining is re-learned and hence reincorporated into the model. This clarifies the sense in which an ERG is “invertible” as long as we allow for the reconstruction of information ostensibly destroyed by diffusion.
One can visualize this process as follows: imagine an experimenter performing a statistical inference experiment in which they observe a collection of data , generated from the distribution . Next, imagine we can place each of the observations along the real axis, distinguishing a series of points, each of which we label by a probability distribution . The probability distribution at the Tth point, , is obtained by incorporating all of the data to the left of T into a model using Bayes’ law. Moving to the left along this axis corresponds to disincorporating data from the model and therefore induces a diffusion process and by extension an ERG scheme. Conversely, moving to the right along this axis corresponds to reincorporating lost data and therefore inverts the ERG flow.
Framing the relationship between ERG and Statistical Inference in terms of the reconstruction problem (
151) suggests several interesting paths for future study. Firstly, the reconstruction problem is equivalent to a common problem encountered in Machine Learning when one wishes to sample data from an analytically intractable distribution,
. An approach to this problem goes by the name of
Diffusion Learning [
49,
50,
51,
52]. Diffusion learning is a two-step process: first, one uses a diffusion operator,
B, with a known fixed point to transform the initial data
into an analytically tractable form. Then one identifies a second diffusion operator,
F, which optimally reconstructs the initial data without sacrificing analytic tractability. This routine is equivalent to (
151), provided we restrict the set of allowed reconstruction algorithms to operators that generate diffusion processes. More generally, the information-theoretic formulation of ERG constructed in this paper renders renormalization in a form that is amenable to applications outside of pure physics. We hope this will catalyze continued work, especially at the intersection of physics and data science, geared towards constructing and better understanding machine learning algorithms like diffusion learning. Since the original drafting of this note, some work to this end has been undertaken. In [
53], the approach introduced in this paper was adapted into a practical renormalization scheme for generic statistical inference models including neural networks. As a proof of concept, this scheme was subsequently applied to construct renormalization group flows for autoencoder neural networks.
A second fascinating implementation of (
151) appears in the study of Holography [
54]. There, one is interested in
reconstructing a bulk spacetime from the data contained in a quantum field theory on its conformal boundary [
55,
56]. The relationship between our work and bulk reconstruction is very natural. In modern literature, bulk reconstruction is often interpreted through the language of Quantum Error Correction as the inversion of a quantum channel associated with the propagation of bulk data into a subregion of the conformal boundary [
57,
58,
59,
60]. Placed in this context, the bulk reconstuction problem is a non-commutative generalization of (
151) in which one replaces probability distributions by density operators, and the maps
B and
F by Quantum Channels [
61,
62] (see
Appendix B for a short discussion of the relationship between this paper and error correction). This suggests that by studying a quantum version of the correspondence introduced in this paper, one might be able to shed light on some of the mysterious aspects of the
correspondence. Beyond this, there are also interesting questions that pertain to the generalization of our correspondence into the general language of non-commutative probability theory,
operator algebras. These include understanding statistical inference for non-commutative operator algebras [
63,
64,
65], the study of non-commutative diffusion processes [
66,
67,
68], and the formulation of renormalization techniques that work beyond the scope of Euclidean QFT in the vein of entanglement renormalization and tensor networks [
69,
70,
71,
72,
73,
74].
Finally, our picture of ERG can provide a powerful tool for constructing and interpreting theorems about renormalization. By formulating ERG flows as a
related statistical inference problem, one obtains a stark accounting of the information/degrees of freedom contained in the renormalized theory at any point over the course of its flow. Such knowledge is of great use in computing the information lost between various points along an RG flow and, especially, in constructing and interpreting RG monotones [
75,
76,
77,
78,
79]. We are hopeful that our conceptual approach to the ERG can expand renormalization in its role as a toolset for studying the space of Quantum Field Theories.