Bayesian renormalization

Following the lead of [41], we take the perspective that an ERG flow can be understood as a one parameter family of probability distributions, $\{P_{\Lambda}[\phi]\}_{\Lambda \in \mathbb{R}}$. Here Λ is a physically meaningful RG scale (typically associated with a momentum cutoff), and $\phi \in \mathcal{F}$ corresponds to the field configuration relevant to a given theory. $P_{\Lambda}[\phi]$ should therefore be read as the probability density assigned to the field configuration φ at the scale Λ. Schematically, we regard

$$\begin{equation} P_{\Lambda}\left[\phi\right] \propto \mathrm{e}^{-S_{\Lambda}\left[\phi\right]}, \end{equation} \tag{ 1 }$$

where $S_{\Lambda}[\phi]$ is the renormalized action at scale Λ. The guiding principle of the ERG is that the flow $P_{\Lambda}[\phi]$ must be chosen in such a way that the partition function is preserved ⁴ :

$$\begin{equation} \frac{\mathrm{d}}{\mathrm{d}\ln \Lambda} \int_{\mathcal{F}} \mathcal{D}\phi \; P_{\Lambda}\left[\phi\right] = 0. \end{equation} \tag{ 2 }$$

The ERG principle (2) ensures that all correlation functions below the scale Λ are preserved over the course of the ERG flow.

The most familiar form of exact renormalization is the so-called Polchinski scheme [10]. In Polchinski's ERG, one writes the probability distribution over fields in the form

$$\begin{equation} P_{\Lambda}\left[\phi\right] \propto \mathrm{e}^{-\frac{1}{2} \int \frac{\mathrm{d}^d p}{\left(2\pi\right)^d} \; \phi\left(p\right) G\left(p^2\right) K^{-1}_{\Lambda}\left(p^2\right) \phi\left(-p\right)} \mathrm{e}^{-S_{\mathrm{int},\Lambda}\left[\phi\right]}. \end{equation} \tag{ 3 }$$

We recognize the first term as the Gaussian distribution associated with a free field theory with propagator $G(p^2)$, but with the incorporation of a function $K^{-1}_{\Lambda}(p^2)$ which plays the role of a smooth cutoff function in momentum space. In words, $K_{\Lambda}^{-1}(p^2)$ suppresses the contribution of momentum modes above the cutoff scale Λ. The second term in (3) is the exponential of the renormalized interacting action at the scale Λ.

In Polchinski's picture, $K_{\Lambda}(p^2)$ has a prescribed dependence on Λ, thus Polchinski's ERG equation arises by determining the equation which must be obeyed by $S_{\mathrm{int},\Lambda}[\phi]$ in order to satisfy the principle (2). By a straightforward computation, one can show that the resulting equation can be put into the form:

$$\begin{align} \frac{\mathrm{d}}{\mathrm{d}\ln\Lambda} P_{\Lambda}\left[\phi\right] & = \int_{M \times M} \mathrm{d}^dx \mathrm{d}^dy \; \left\{{C}^\mathrm{Pol.}_{\Lambda}\left(x,y\right) \frac{\delta^2 P_{\Lambda}\left[\phi\right]}{\delta\phi\left(x\right) \delta\phi\left(y\right)} + \frac{\delta}{\delta \phi\left(x\right)}\left(P_{\Lambda}\left[\phi\right]{C}^\mathrm{Pol.}_{\Lambda}\left(x,y\right) \frac{\delta V^\mathrm{Pol.}_{\Lambda}\left[\phi\right]}{\delta \phi\left(y\right)}\right)\right\} \end{align} \tag{ 4 }$$

$$\begin{align} &\equiv \Delta P_{\Lambda}\left[\phi\right] + \text{div}\left(P_{\Lambda}\left[\phi\right] \text{grad}_{{C}^\mathrm{Pol.}_{\Lambda}} V^\mathrm{Pol.}_{\Lambda}\left[\phi\right]\right), \end{align} \tag{ 5 }$$

where

$$\begin{equation} {C}^\mathrm{Pol.}_{\Lambda}\left(p^2\right) = \left(2\pi\right)^d G\left(p^2\right)^{-1} \frac{\partial K_{\Lambda}\left(p^2\right)}{\partial \ln \Lambda}; \;\;\; V^\mathrm{Pol.}_{\Lambda}\left[\phi\right] = \int \frac{\mathrm{d}^dp}{\left(2\pi\right)^d} \phi\left(p\right) G\left(p^2\right) K^{-1}_{\Lambda}\left(p^2\right) \phi\left(-p\right). \end{equation} \tag{ 6 }$$

One might recognize (4) as the Fokker–Planck equation with diffusion governed by ${C}^\mathrm{Pol.}_{\Lambda}(p^2)$ and drift governed by the potential $V^\mathrm{Pol.}_{\Lambda}[\phi]$. This is the first indication of a deep relationship between exact renormalization and diffusion. Note that the equivalence between (4) and (5) is just a rewriting in terms of the functional (infinite dimensional) equivalent of vector operators. This is so one can identify (4) as a functional version of Fokker–Plank.

We refer to the Polchinski approach as an ERG scheme because it corresponds to a particular choice on how to renormalize the theory described by (3). The nexus of this choice can be traced back to the way Polchinski decided to regulate the action in (3) by introducing a smooth cutoff function $K^{-1}_{\Lambda}(p^2)$. This choice manifests itself in the particular form of the diffusion and drift aspects of (6) specifying the Fokker–Planck equation associated with the Polchinski ERG (4). Choosing different regulating functions corresponds to different ERG schemes, and as a result different Fokker–Planck equations specified by the data (6).

More abstractly, we can define an ERG directly by specifying the data $({C}_{\Lambda},V_{\Lambda})$ corresponding to the diffusivity and drift of a Fokker–Planck equation. Equation (4) still holds but with $({C}_{\Lambda},V_{\Lambda})$ replacing ${C}^\mathrm{Pol.}_{\Lambda}(p^2), V^\mathrm{Pol.}_{\Lambda}[\phi]$. This is then an ERG corresponding to a different scheme.

The Fokker–Planck equation corresponds to a bonafide ERG because it satisfies the ERG principle (2). To see that this is the case, let us now show that we can rewrite (4) in the form

$$\begin{equation} -\frac{\mathrm{d}}{\mathrm{d}\ln\Lambda} P_{\Lambda}\left[\phi\right] = \int_M \mathrm{d}^dx \; \frac{\delta}{\delta \phi\left(x\right)} \left(\Psi_{\Lambda}\left[\phi;x\right] P_{\Lambda}\left[\phi\right]\right), \end{equation} \tag{ 7 }$$

where M is the spacetime manifold on which the theory is defined [12]. Hopefully it is clear that any one parameter family $P_{\Lambda}[\phi]$ satisfying (7) also satisfies (2). This is because (7) specifies a divergence flow, that is the right hand side of (7) is a divergence in the space of field configurations. We can therefore employ the divergence theorem to observe that

$$\begin{equation} \frac{\mathrm{d}}{\mathrm{d}\ln\Lambda} \int_{\mathcal{F}} \mathcal{D} \phi \; P_{\Lambda}\left[\phi\right] = - \int_{\mathcal{F}} \mathcal{D}\phi \; \int_M \mathrm{d}^dx \; \frac{\delta}{\delta \phi\left(x\right)} \left(\Psi_{\Lambda}\left[\phi;x\right] P_{\Lambda}\left[\phi\right]\right) = 0. \end{equation} \tag{ 8 }$$

In order to write (4) in the form (7) we take

$$\begin{equation} \Psi_{\Lambda}\left[\phi;x\right] = \int_M \mathrm{d}^dy \; {C}_{\Lambda}\left(x,y\right) \frac{\delta \Sigma_{\Lambda}\left[\phi;P_{\Lambda}\right]}{\delta \phi\left(y\right)}, \end{equation} \tag{ 9 }$$

as has appeared previously in [12, 16, 17, 41, 45]. Here $C_{\Lambda}(x,y)$ is the ERG kernel appearing in the Fokker–Planck equation associated to the ERG, and $\Sigma_{\Lambda}[\phi;P_{\Lambda}]$ is called the scheme functional which is determined through the ERG potential $V_{\Lambda}$ via the equation

$$\begin{equation} \Sigma_{\Lambda}\left[\phi;P_{\Lambda}\right] = -\ln\left(\frac{P_{\Lambda}\left[\phi\right]}{\mathrm{e}^{-V_{\Lambda}\left[\phi\right]}}\right) = S_{\Lambda}\left[\phi\right] - V_{\Lambda}\left[\phi\right]. \end{equation} \tag{ 10 }$$

Plugging (9) back into (7), we reconcile (4) with the diffusion and drift aspects given by $(C_{\Lambda}, V_{\Lambda})$, as desired. Together $(C_{\Lambda},V_{\Lambda})$ therefore specify a consistent scheme for regulating the high energy degrees of freedom of the field theory, in analogy with the regulating function $K^{-1}_{\Lambda}(p^2)$ appearing in (3).

It is worth noting that (7) defines an approach to ERG that is more general than diffusion. All divergence flows, which can generically be written in the form (7), specify ERG flows as evidenced by (8). However, only the subset of divergence flows in which the reparameterization kernel $\Psi_{\Lambda}$ is taken to be of the form (9) result in Fokker–Planck equations. Equation (7) is called the Wegner–Morris equation, and ERG schemes satisfying the Wegner–Morris equation are called Wegner–Morris schemes. The choice of Wegner–Morris scheme is encapsulated entirely in the reparameterization kernel. We shall refer to reparameterization kernels which are in the form of (9) as Fokker–Planck schemes to highlight their relationship with diffusion. A Fokker–Planck scheme is specified entirely by the data $(C_{\Lambda},V_{\Lambda})$.

To conclude this section, let us briefly explore an alternative way to recognize that the Wegner–Morris equation specifies an ERG flow which allows us to supply a more concise and intuitive interpretation of the ERG. The effect of (7) on $P_{\Lambda}$ can be absorbed by continuously reparameterizing the fields φ at each new scale according to the rule

$$\begin{equation} \phi^{\prime}\left(x\right) = \phi\left(x\right) + \left(\delta \ln\Lambda\right) \Psi\left[\phi;x\right]. \end{equation} \tag{ 11 }$$

Equation (11) should be regarded as the integral curve of $\Psi_{\Lambda}[\phi;x]$ in the space of field configurations, where we are regarding $\Psi_{\Lambda}[\phi;x] \in T\mathcal{F}$ as a tangent vector to this space. Solving (11) results in a one parameter family of field configurations $\{\phi_{\Lambda}\}_{\Lambda \in \mathbb{R}}$, in which $\phi_{\Lambda}$ can be thought of as describing the relevant field degrees of freedom at scale Λ. In this way, exact renormalization can be connected with more familiar Wilsonian renormalization schemes by interpreting equation (11) as specifying a coarse graining procedure. The coarse graining map (11) is a diffeomorphism in the space of field configurations, which means it must leave the integral

$$\begin{equation} \int_{\mathcal{F}} \mathcal{D}\phi \; P_{\Lambda}\left[\phi\right] \end{equation} \tag{ 12 }$$

invariant. Thus, again, we find that the equation (7) specifies a meaningful renormalization scheme in the usual sense of satisfying (2).

Specializing to Fokker–Planck ERG schemes, we can expand on this discussion. As was introduced in detail in [18], a (functional) Fokker–Planck equation of the form (4) is associated with a (functional) stochastic differential equation (SDE):

$$\begin{equation} \mathrm{d}\phi\left(x\right) = -\text{grad}_{{C}_{\Lambda}} V_{\Lambda}\left[\phi\right] \left(d\ln\Lambda\right) + \sqrt{2} \int_{M} \mathrm{d}^dy \; \sigma_{\Lambda}\left(x,y\right)\mathrm{d}W_{\Lambda}\left(y\right). \end{equation} \tag{ 13 }$$

Here, $W_{\Lambda}(x)$ is a function valued Weiner process, and $\sigma_{\Lambda}$ is the diffusivity kernel defined by the property that it 'squares' to the covariance ${C}_{\Lambda}$:

$$\begin{equation} \int_{M} \mathrm{d}^dz \; \sigma_{\Lambda}\left(x,z\right) \sigma_{\Lambda}\left(z,y\right) = {C}_{\Lambda}\left(x,y\right). \end{equation} \tag{ 14 }$$

Equation (13) is the stochastic differential equation that arises from the deterministic gradient flow defined by (11) subject to noise with covariance governed by ${C}_{\Lambda}$. Thus, we have arrived at the punchline: an ERG flow specified by the data $(\mathcal{F},{C}_{\Lambda}(x,y),V_{\Lambda}[\phi])$ can be understood as the stochastic coarse graining of the field degrees of freedom arising from the reparameterization of the fields under the gradient of the potential $V_{\Lambda}$ subject to noise governed by ${C}_{\Lambda}$.

In section 2.1, we illustrated how ERG flows a la Wegner–Morris correspond to a subclass of functional diffusion equation (7). These Fokker–Planck schemes are specified by the pair $({C}_{\Lambda}, V_{\Lambda})$ which set, respectively, the diffusion and drift of a stochastic process underyling the Fokker–Planck equation associated with the ERG. The stochastic process (13) explicitly describes a coarse graining of the field degrees of freedom, while the related diffusion equation encodes the impact of such a coarse graining on the renormalized action, resulting in an effective field theory at each scale. One of the main insights of [18] is that viewing exact renormalization as a diffusion process suggests an approach to renormalization which is applicable to a wider class of systems modeled by probability distributions, beyond merely those which posses a physical interpretation. In this section we will outline this approach.

Consider a random variable Y with sample space S. For simplicity, let us assume that Y is a continuous random variable, and S is a Riemannian manifold. A general class of diffusion equations on S are then specified by a one parameter family of probability distributions $\{p_t(y)\}_{t \in \mathbb{R}}$ along with a metric ⁵ $g_t: TS \times TS \rightarrow \mathbb{R}$ and a potential function $V_t: S \rightarrow \mathbb{R}$ such that

$$\begin{equation} \frac{\mathrm{d}p_t\left(y\right)}{\mathrm{d}t} = \Delta p_t\left(y\right) + \text{div}\left(p_t\left(y\right) \text{grad}_{g_t} V_t\left(y\right)\right). \end{equation} \tag{ 15 }$$

Equation (15) is the Fokker–Planck equation associated with the stochastic differential equation

$$\begin{equation} \mathrm{d}Y^i_t = -\left(\text{grad}_{g_t} V_t \right)^i \mathrm{d}t + \sqrt{2} \left(\theta_t\right)^i_j \mathrm{d}W^j_t \end{equation} \tag{ 16 }$$

where ⁶

$$\begin{equation} \delta^{kl} \left(\theta_t\right)^i_k \left(\theta_t\right)^j_l = g_t^{ij}. \end{equation} \tag{ 17 }$$

It is worth noting here that the g_t plays a double role. From one perspective it is the metric in this process but interpreted in terms of the underlying statistics it is the inverse of the covariance matrix. Then equations (15) and (16) should be compared with (4) and (13) from the exact renormalization context.

As was the case in section 2.1, we can formally solve for the gradient flow of the potential V_t to determine a one parameter family of renormalized degrees of freedom. To be precise, let $\gamma: \mathbb{R} \rightarrow S$ be a one parameter family of points in S solving the gradient flow problem

$$\begin{equation} \frac{\mathrm{d}\gamma_t}{\mathrm{d}t} = \text{grad}_{g_t} V\rvert_{\gamma_{\tau}}. \end{equation} \tag{ 18 }$$

In terms of γ_t we can now write (16) in the form

$$\begin{equation} \mathrm{d}Y^i_t = -\dot{\gamma_t}^i \mathrm{d}t + \sqrt{2} \left(\theta_t\right)^i_j \mathrm{d}W^j_t. \end{equation} \tag{ 19 }$$

where $\dot{\gamma}_t$ and θ_t correspond to the drift and diffusion, respectively.

The Fokker–Planck equation (15) has a schematic solution:

$$\begin{equation} p_t\left(y\right) = \int_S \mathrm{d}^d y_0 \; \pi\left(y,y_0;t\right) p_0\left(y_0\right). \end{equation} \tag{ 20 }$$

Here, $p_0(y)$ is the initial data, and $\pi(y,y_0;t)$ is the heat Kernel. Provided the drift and diffusion are constant over the full sample space the diffusion kernel will be a Gaussian of the form ⁷

$$\begin{equation} \pi\left(y,y_0;t\right) = \mathcal{N}\left(\gamma_t, tg_t^{-1}\right)\left(y-y_0\right) \end{equation} \tag{ 21 }$$

Explicitly, $\pi(y,y_0;t)$ is the transition probability density for a sample point to start from y₀ and diffuse to y in a time t. We shall interpret the heat kernel as a stochastic map encoding the implicit coarse graining scheme associated with the diffusive RG flow.

This picture of renormalization is closely related to information theoretic approaches to renormalization from the high energy physics community that have been employed in the study of holography and operator algebras [46-48]. To see this, note that we can encode (20) as a semigroup of conditional expectation operators acting on the space of functions on S, $E_{t}: \Omega^0(S) \rightarrow \Omega^0(S)$. The conditional expectation operator E_t acts as ⁸

$$\begin{equation} E_{t}\left(f\right) = \mathbb{E}_{\pi_{t}}\left(f\left(Y_0\right)\right) = \int_{S} \text{Vol}_S\left(y_0\right) \; \pi\left(y,y_0;t\right)f\left(y_0\right), \end{equation} \tag{ 22 }$$

so that, for example, the posterior predictive distribution is given by

$$\begin{equation} p_t\left(y\right) = E_t\left(p_0\right)\left(y\right) = \mathbb{E}_{\pi_t}\left(p_0\left(Y_0\right)\right). \end{equation} \tag{ 23 }$$

The set of operators $\{E_t\}_{t \in \mathbb{R}}$ form a semigroup in the sense that

$$\begin{equation} E_{t_2} \circ E_{t_1} = E_{t_1 + t_2}. \end{equation} \tag{ 24 }$$

In a more general context, a conditional expectation on a von Neumann algebra M is a projection of M to a subset $N \subset M$, $E: M \rightarrow N$, which retains the normalization of states such that $E(\unicode{x1D7D9}_M) = \unicode{x1D7D9}_N$ [49]. A very broad class of renormalization schemes accessible to quantum probabilities can subsequently be formulated as a semigroup of conditional expectation operators $\{E_{\Lambda}\}_{\Lambda \in \mathbb{R}}$ acting on the space of operators affiliated with a given system. In a recent work [48], this form of renormalization was given the name code subspace renormalization to reflects its relationship with error correction ⁹ .

The relationship between renormalization and diffusion is very satisfying because we can think of a diffusion process as destroying some of the fine grained information stored in a very complicated, 'UV complete' probability distribution. Indeed, this is the point of view which is advocated for in the influential diffusion learning paradigm [42]. In diffusion learning, one begins with a highly complex and intractable distribution p₀ which is run through a forward diffusion process for a time t_f in order to arrive at an analytic distribution, p_f . One can then sample data from p₀ by first sampling data from p_f and subsequently solving a 'reverse' stochastic differential equation derived from (16) [53]. To be more precise, given a generic forward diffusion process specified by the SDE

$$\begin{equation} \mathrm{d}Y_t^i = \mu^i_t\left(Y_t\right) \mathrm{d}t + \sigma_t{}^i{}_j\left(Y_t\right) \mathrm{d}W_t^j, \end{equation} \tag{ 25 }$$

the associated reverse SDE is of the form [54]

$$\begin{equation} \mathrm{d}Y_{s}^i = \left\{\mu^i_{s} - \frac{1}{2} \partial_j \left(\delta^{kl} \sigma_{s}{}^i{}_k \sigma_{s}{}^j{}_l\right) - \delta^{kl} \sigma_{s}{}^i{}_k \sigma_{s}{}^j{}_l \partial_j \ln p_{s} \right\}\bigg\rvert_{Y_{s}} \mathrm{d}s + \sigma_s{}^i{}_j\left(Y_{s}\right) \mathrm{d}W_{s}^j \end{equation} \tag{ 26 }$$

Here, t, which runs from 0 to $t_\mathrm{f}$, is the forward time, while s, which runs from $t_\mathrm{f}$ to the initial time 0, is the reverse time. The reverse SDE is determined by all of the same data as the forward SDE with exception of the score functions—$\partial_j \ln p_{s}(y)$. Here, $p_{s}(y)$ is probability distribution obtained by diffusing p₀ according to the forward SDE up to the reverse time s. Simulating the reverse SDE therefore amounts to efficiently reconstructing the scores. In the machine learning community, a resolution to this problem has been presented in terms of the training of a score based generative algorithm (see [53] for a comprehensive review). Interestingly, such a score-based model can be interpreted as a statistical inference problem in which the distribution p_s is learned by observing draws from intermediate diffused distributions. We will revisit this in section 3 where we provide an alternative point of view on the same fact by demonstrating that inverting a diffusion process can be thought of as a Bayesian inference experiment.

Based on the observations of the preceding paragraph, one may be tempted to say that diffusion learning can be thought of as a form of exact renormalization for data science models. In light of the correspondence between ERG and diffusion, such a statement is technically correct. However, naive diffusion lacks a very crucial feature which is at the core of a good RG flow; namely a meaningful notion of scale. Indeed, unlike physical renormalization which coarse grains hierarchically over length/energy scales, naive diffusion removes information in an essentially unstructured way. Thus, while basic diffusion can technically be regarded as a form of renormalization, it fails to provide real control over how a probability model changes as a function of any meaningful scale. More plainly, in physics based renormalization we know that the information lost to coarse graining correspond to high energy modes via the construction of the ERG kernel. By contrast, naive diffusion of a probability distribution apparently discards information indiscriminately. Ultimately, this is problem that the Bayesian approach to diffusion/renormalization will overcome.

The first step in Bayesian inference is model selection. This corresponds to choosing a parametric family of probability distributions on S, $p_{Y \mid \Theta}(y \mid \theta)$. A priori, we should consider any allowed distribution for Y as a candidate for the data generating distribution. Thus, we might take

$$\begin{equation} p_{Y \mid \Theta}\left(y \mid \theta\right) = \mathrm{e}^{-\theta^i S_i\left(y\right)} \end{equation} \tag{ 27 }$$

where here $\{S_j(y)\}_{j \in \mathcal{J}}$ is the set of log-likelihoods corresponding to allowed probability models for Y. In principle the space of models might be infinite dimensional, however for simplicity let us assume that it is finite ¹⁰ .

Let $\mathcal{M} = \{p_{Y\mid \Theta}(y \mid \theta) \; | \; \theta \in \mathbb{R}^n\}$. In words, $\mathcal{M}$ is a space whose points correspond to probability distributions for Y. By construction, this space has a local coordinate system given in terms of the parameters θ. A natural basis for the tangent space of $\mathcal{M}$, is given to us in terms of the score vectors $\underline{\ell}_i = \frac{\partial}{\partial \theta^i} \ln(p_{Y\mid\Theta}(y\mid\theta))$. It is easy to see that these vectors are linearly independent and spanning insofar as they are isomorphic to the coordinate basis $\underline{\partial}_i$. Moreover, when viewed as functions on S the score vectors have zero expectation value due to the normalization condition on p:

$$\begin{equation} \mathbb{E}_\theta\left(\underline{\ell}_i\left(Y\right)\right) = \int_S \mathrm{d}^dy \; p_{Y\mid\Theta}\left(y \mid \theta\right) \frac{\partial \ln\left(p_{Y\mid\Theta}\left(y\mid\theta\right)\right)}{\partial \theta^i} = \frac{\partial}{\partial \theta^i} \int_S \mathrm{d}^dy \; p_{Y\mid\Theta}\left(y\mid\theta\right) = 0. \end{equation} \tag{ 28 }$$

In terms of this basis, we define the Fisher information matrix:

$$\begin{equation} \mathcal{I}_{ij}\left(\theta\right) = \mathbb{E}_{\theta}\left(\underline{\ell}_i \underline{\ell}_j\right) = \mathbb{E}_{\theta}\left(\frac{\partial \ln\left(p_{Y\mid\Theta}\left(Y\mid\theta\right)\right)}{\partial \theta^i} \frac{\partial \ln\left(p_{Y\mid\Theta}\left(Y\mid\theta\right)\right)}{\partial \theta^j}\right). \end{equation} \tag{ 29 }$$

The Fisher matrix (29) should be interpreted as the components of a metric, the Fisher metric, on $\mathcal{M}$ in the basis $\{\underline{\ell}_i\}$.

The Fisher metric provides an infinitesimal measure of the similarity between two models in $\mathcal{M}$. This can be seen most clearly through the relationship between the Fisher metric and the KL-divergence. Recall,

$$\begin{equation} D_\mathrm{KL}\left(\theta \parallel \theta^{\prime}\right) = \mathbb{E}_{\theta}\left(\ln\left(\frac{p_{Y\mid\Theta}\left(Y\mid\theta\right)}{p_{Y\mid\Theta}\left(Y\mid\theta^{\prime}\right)}\right) \right). \end{equation} \tag{ 30 }$$

measures the relative entropy between two distributions. $D_\mathrm{KL}$ is an information divergence, which means that $D_\mathrm{KL}(\theta \parallel \theta^{\prime}) \unicode{x2A7E} 0$ and it is equal to zero if and only if $\theta = \theta^{\prime}$. This makes $D_\mathrm{KL}$ a good measure of the distinguishability between models. In general, however, $D_\mathrm{KL}$ is not a symmetric function of θ and $\theta^{\prime}$ and therefore cannot be regarded itself as an inner product. Nonetheless, in the immediate neighborhood of a point $\theta \in \mathcal{M}$ the KL-divergence can be expanded to quadratic order as

$$\begin{equation} D_\mathrm{KL}\left(\theta \parallel \theta^{\prime}\right) = \frac{1}{2} \mathcal{I}_{ij}\left(\theta\right) \delta \theta^i \delta \theta^j + \mathcal{O}\left(\delta \theta^3\right). \end{equation} \tag{ 31 }$$

The Fisher metric plays a significant role in parameter estimation because it encodes the sensitivity of a model's output to changes in parameter values [40]. This is closely related to the task of distinguishing between model parameters that are 'sloppy' and 'stiff' [69, 70]. Strict parameters covary strongly with the model output and therefore correspond to large eigenvalues of the Fisher metric. Sloppy parameters, on the other hand, covary only weakly with the model output and therefore correspond to small eigenvalues of the Fisher metric. From a geometric perspective, this means that we can think of sloppy directions—that is directions coordinatized by sloppy parameters—as being highly compact in the space of models. In particular, two models that differ only along sloppy directions will be very hard to distinguish. As a consequence, sloppy parameters require an extensive amount or quality of observed data in order to be fit [71]. However, because these parameters only weakly impact the predictive power of the model it may also be possible to systematically remove them in favor of a reduced model that depends only a sufficient set of strict parameters.

This state of affairs strongly resembles the typical use case of the RG in physical theories. Our ability to meaningfully distinguish between theories is capped by our capacity to perform experiments which measure physics beyond particular scales. In this sense, two theories which yield equivalent predictions except beyond scales that cannot be experimentally probed must be regarded as practically equivalent. If we view a physical theory as parameterized by the Fourier modes of field degrees of freedom, it is precisely the high energy modes which correspond to the 'sloppy parameters' since one requires an extensive or even impossible set of measurements to distinguish between theories that differ only at very high energy scales. From this point of view, we recognize renormalization as a scheme for systematically regulating sloppy model parameters in order to arrive at an equivalence class of models that cannot be distinguished at the level of accuracy admitted by our present ability to observe data.

In recent work [34] it has been suggested that a similar approach would be very useful in a more broad data science context. Quite serendipitously, this can be thought of as a 'UV regularization scheme' from the perspective of the Fisher geometry—it is a consistent method for dealing with the fact that we cannot resolve up to the arbitrarily small distances in model space necessary to identify between models that differ along directions coordinatized by sloppy parameters. As we introduce the Bayesian renormalization scheme, we will highlight how it automatically performs an information geometrically natural regularization to this end, that is oriented directly toward intelligently removing these sloppy degrees of freedom.

The upshot of section 3.1 is that the starting point of a Bayesian inference experiment can be understood as the specification of a Riemannian information geometry $(\mathcal{M},\mathcal{I})$, where $\mathcal{M}$ consists of all possible probability models for Y, and $\mathcal{I}$ is an infinitesimal measure of the distinguishability between models. As we have discussed, adopting an information geometric approach to Bayesian inference already allows us to begin formulating the correspondence between learning in the space of models and renormalization. The next step in Bayesian inference is updating, which we regard as a specification of dynamics in the space of models.

In conventional Bayesian inference, the updating phase starts by specifying a prior distribution $\pi_0: \mathcal{M} \rightarrow \mathbb{R}$ which acts as the initial data for the dynamical system. In light of observed data, the prior distribution is updated to the posterior distribution, $\pi_{T}(\theta)$, by using Bayes' law. That is

$$\begin{equation} \pi_T\left(\theta\right) \propto \left(\prod_{t = 1}^T p_{Y\mid\Theta}\left(y_t \mid \theta\right)\right) \pi_0\left(\theta\right). \end{equation} \tag{ 32 }$$

The constant of proportionality can be deduced by enforcing that π_T be a normalized probability distribution on $\mathcal{M}$. One should interpret (32) as specifying that the probability the data generating model lives in a small neighborhood of the point $\theta \in \mathcal{M}$ is proportional to the probability that one would have observed the existing sample conditional on the underlying parameter value being in such a neighborhood, multiplied by the prior weight given to that neighborhood. In this way the posterior distribution is slowly trained around the region in model space where the data generating distribution lives.

In [4], we asked the question of what Bayesian inference would look like as a continuous time dynamical system. In other words, we think of data as being continuously observed so that the sample $\{y_t\}_{t = 1}^T$ is growing as a function of the 'time' parameter T. We subsequently showed that the posterior distribution is governed by an integro-differential equation

$$\begin{equation} \frac{\partial \pi_T\left(\theta\right)}{\partial T} = -\left(D_\mathrm{KL}\left(\theta^* \parallel \theta\right) - \mathbb{E}_{\pi_T}\left(D_\mathrm{KL}\left(\theta_* \parallel \Theta\right)\right) \right) \pi_T\left(\theta\right). \end{equation} \tag{ 33 }$$

Here $\theta_*$ is the parameter corresponding to the location of the data generating distribution in $\mathcal{M}$, and we have assumed that this value is unique ¹¹ . The equation (33) has a schematic solution

$$\begin{equation} \pi_T\left(\theta\right) \propto \mathrm{e}^{-TD_\mathrm{KL}\left(\theta_* \parallel \theta\right)} \end{equation} \tag{ 34 }$$

which we interpret as a Boltzmann distribution with 'energy' given by the KL-divergence between the data generating model and a model at $\theta \in \mathcal{M}$. This observation is intimately related with the idea of treating the distinguishability of models as a kind of 'energy' scale.

At sufficiently late T the posterior distribution will be of the form

$$\begin{equation} \pi_T\left(\theta\right) = \mathcal{N}\left(\mu_T, \frac{1}{T} \mathcal{I}\left(\mu_T\right)^{-1}\right)\left(\theta\right). \end{equation} \tag{ 35 }$$

Here µ_T is the T-path of the maximum a posterior (MAP) estimate. We regard the specification of µ_T as defining a new dynamical element. Once the full time path of the MAP has been observed, one can construct a potential function $V: \mathcal{M} \rightarrow \mathbb{R}$ for which µ_T is defined to be a gradient flow

$$\begin{equation} \frac{\mathrm{d}}{\mathrm{d}T} \mu_T = \text{grad}_{\mathcal{I}} V \rvert_{\mu_T}. \end{equation} \tag{ 36 }$$

One can consider V as encoding the details of the sequence with which data was observed in relation to the sequential evolution of the answer to the question of which single model best approximates the data generating distribution. Eventually, as $T \rightarrow \infty$ we expect that $\mu_T \rightarrow \theta_*$.

One may interpret (35) as specifying that the posterior distribution π_T is localized around the MAP, µ_T , with a characteristic width given by $\frac{1}{T} \mathcal{I}(\mu_T)^{-1}$. Notice, the width of this distribution is shrinking as a function of T. In other words, as more and more data is observed, the posterior trains around a smaller and smaller neighborhood of the MAP. This means that the posterior predictive model at time T will become more and more precisely attuned to the specific characteristics of the data generating distribution.

We conclude that a dynamical Bayesian inference scheme is specified by the triple $(\mathcal{M}, \mathcal{I}, V)$, where $\mathcal{M}$ specified the set of allowed models, $\mathcal{I}$ endows this set with a notion of scale in terms of the distinguishability of nearby models, and V encodes the sequence with which data is observed and the trajectory of the MAP as it converges toward the data generating model. This set of data is cosmetically very similar to the data defining an ERG flow, $(\mathcal{F},{C}_{\Lambda}, V_{\Lambda})$. We will now demonstrate how one can define an information theoretic ERG flow in terms of the dynamical Bayesian inference scheme.

The posterior predictive distribution is obtained by taking the convolution of the posterior and the likelihood model:

$$\begin{equation} p_T\left(y\right) = \mathbb{E}_{\pi_T}\left(p_{Y\mid\Theta}\left(y \mid \Theta\right)\right) = \int_{\mathcal{M}} \text{Vol}_{\mathcal{M}}\left(\theta\right) \pi_T\left(\theta\right) p_{Y\mid\Theta}\left(y\mid\theta\right). \end{equation} \tag{ 37 }$$

Taking the posterior to be of the form (35), we can see that (37) is a weighted sum of probability models for Y in which the set of models outside a small neighborhood of the MAP are heavily suppressed. Notice that the posterior distribution is playing a very similar role to the cutoff function in (3), only in the opposite direction. The smooth cutoff suppresses high momentum modes or, in other words, information at short scales. Conversely, as T increases, the posterior distribution suppresses information at large distances as measured by the information geometry on $\mathcal{M}$. This makes sense, as we have dictated renormalization is a form of diffusion in which information is removed from the probability model. By contrast, Bayesian inference incorporates new information with the observation of each new data point. This discrepancy motivates the idea of considering Bayesian inference in reverse in which, rather than incorporating new data, an experimenter sequentially removes data from the reconstructed model. We refer to this process as backward inference.

Formally, backward inference corresponds to flowing along the inverse 'time' parameter $\tau = \frac{1}{T}$. In terms of this parameter,

$$\begin{equation} \pi_{\tau}\left(\theta\right) = \mathcal{N}\left(\mu_{\tau}, \tau \mathcal{I}\left(\mu_{\tau}\right)^{-1}\right)\left(\theta\right). \end{equation} \tag{ 38 }$$

As τ increases the width of the posterior distribution increases and a larger set of models are meaningfully incorporated into the posterior predictive distribution ¹² . Over time the set of models which live in a small neighborhood of the data generating model will receive less and less weight in the posterior predictive distribution, as the reconstructed model becomes a weighted sum of a more diverse set of models. Inputting (38) into (37), we find the τ-dependent predictive distribution:

$$\begin{equation} p_{\tau}\left(y\right) = \int_{\mathcal{M}} \text{Vol}_{\mathcal{M}}\left(\theta\right) \; \mathcal{N}\left(\mu_{\tau}, \tau \mathcal{I}\left(\mu_{\tau}\right)^{-1}\right)\left(\theta\right) p_{Y \mid \Theta}\left(y \mid \theta\right), \end{equation} \tag{ 39 }$$

which defines a semigroup of conditional expectation values acting in the space of models. Hence we regard (39) as defining an RG flow directly in the space of models.

Operationally, (39) defines a perfectly reasonable renormalization scheme which coarse grains in model space. However, there is utility to translating (39) so that it can also be understood as explicitly coarse graining in the space of data realizations, as is more standard in typical renormalization schemes. To accomplish this task, we will restrict our attention to spaces of models that are realized in the context of Bayesian inversion [72].

The goal of a Bayesian inversion problem is to deduce the signal, θ, that predicated a measured output or data, y. The data and signal are related by a map, $y = G(\theta)$, which we regard as a deterministic model. In practice, either due to explicit stochasticity or limitations to measurement precision, we must regard the realized output as a random variable which depends conditionally on the signal as

$$\begin{equation} Y \mid \Theta = G\left(\Theta\right) + N \end{equation} \tag{ 40 }$$

where N is a random variable that is conditionally independent of Y and encodes the aforementioned noise. One may therefore read (40) as dictating that Y and Θ are related by a 'law' $Y = G(\Theta)$ but subject to some random fluctuations governed by N. Provided that the noise is distributed with some density $p_N(n)$, we can form the conditional density of Y given Θ by pulling this measure back by (40) to obtain

$$\begin{equation} p_{Y \mid \Theta}\left(y \mid \theta\right) = p_N\left(y - G\left(\theta\right)\right). \end{equation} \tag{ 41 }$$

The procedure described above is familiar from conventional approaches to Bayesian inference for modeling complex systems and constructing neural networks. For example, in the most simple case of a feed-forward neural network, the deterministic function $G(\theta)$ is given by $f(x;W,b)$ where f is the neural network architecture specified by its set of weights, W, and biases, b. In the simplest case of L² loss, we take

$$\begin{equation} p_{Y \mid \Theta}\left(y \mid \theta\right) = \mathcal{N}\left(0,\sigma^2\right)\left(y-G\left(\theta\right)\right) \end{equation} \tag{ 42 }$$

where σ² is a hyperparameter that sets a scale for the tolerated prediction error.

Given a parametric family of probability distributions for Y which can be written in the form (41), we are motivated to rewrite the posterior predictive distribution (39) as an integral over S by forming the pushforward measure via the mapping G ¹³ .

$$\begin{equation} p_{\tau}\left(y\right) = \int_{S} \text{Vol}_S\left(y_0\right) \; \mathcal{N}\left(\gamma_{\tau}, \tau K_{\tau}^{-1}\right)\left(y-y_0\right) \; p_N\left(y_0\right). \end{equation} \tag{ 43 }$$

Here $\gamma_{\tau} = G(\mu_{\tau})$ is the data prediction associated with the MAP estimate, and

$$\begin{equation} \left(K_{\tau}^{-1}\right)^{ab} = \frac{\partial G^a}{\partial \theta^i} \frac{\partial G^b}{\partial \theta^j} \mathcal{I}^{ij}_{\tau} \end{equation} \tag{ 44 }$$

is the pushforward of the inverse Fisher metric by the map G to give the induced Fisher metric on the data space itself.

We can now compare (43) with (20) and recognize that the posterior distribution, when pushed forward into the sample space by the map G, is the heat kernel of an associated convection–diffusion equation. In particular, (43) can be read as the solution to a Fokker–Planck equation

$$\begin{equation} \frac{\mathrm{d}p_{\tau}}{\mathrm{d}\tau} = \Delta p_{\tau} + \text{div}\left(p_{\tau} \text{grad}_{K_{\tau}} V_{\tau}\right) \end{equation} \tag{ 45 }$$

or equivalently, as describing a stochastic process

$$\begin{equation} \mathrm{d}Y^a_{\tau} = -\left(\text{grad}_{K_{\tau}} V_{\tau}\right)^a \mathrm{d}\tau + \sqrt{2} \left(\theta_{\tau}\right)^a_{b} \mathrm{d}W^{b}_{\tau}. \end{equation} \tag{ 46 }$$

As discussed in (36), V_τ is a potential function on the sample space for which γ_τ is a gradient flow, and $\delta^{cd}(\theta_{\tau})^a_c (\theta_{\tau})^b_d = (K^{-1}_{\tau})^{ab}$.

Equations (43), (45) and (46) complete an explicit mapping between a dynamical Bayesian inference scheme and an ERG flow which we can now represent schematically as the identification:

$$\begin{equation} \left(\mathcal{M}, \mathcal{I}, V\right) \leftrightarrow \left(S, K, V\right). \end{equation} \tag{ 47 }$$

The mapping (47) provides a novel information theoretic interpretation for exact renormalization as the inverse process dual to a Bayesian inference procedure governed by (33). In particular, we have realized our goal of demonstrating how performing the inverse of statistical inference, that is by discarding data rather than collecting data, implements an information theoretic renormalization scheme with emergent scale given by the distinguishability of probability distributions in model space. The role of this scale in coarse graining can be seen through the relationship between the ERG kernel, $K^{-1}_{\tau}$, and the Fisher information metric.

For clarity, it is helpful to compare the governing equations of Bayesian renormalization and DB with the forward and reverse diffusion processes introduced in section 2.2. From the diffusion learning perspective, one begins by running the data generating distribution through a forward diffusion process specified by an SDE of the form (25). This results in a more tractable distribution which can subsequently be used to generate samples from the data generating distribution by appealing to the associated reverse SDE (26). To accomplish this step one must implement an algorithm which reconstructs the score functions; a task which can be formulated as a Bayesian learning problem. By contrast, our derivation of the Bayesian renormalization scheme has ensued in exactly the opposite direction. We began with a Bayesian learning problem governed by the dynamical Bayesian inference equation (33). Then, we constructed the reverse process (relative to Bayes) by studying how the learned distribution changes when data is removed as opposed to incorporated. The result was a diffusion process governed by the SDE (46).

One should identify the forward diffusion, (25), with the Bayesian diffusion process, (46). Similarly, one should identify the reverse diffusion process, (26), with the dynamical Bayesian learning process, (33). In summary, one way of interpreting Bayesian renormalization is as an information theoretic formulation of the observation that the reverse SDE associated with a diffusion process corresponds with a statistical learning problem. The advantage of this approach is that, opposed to a naive diffusion scheme as would be implemented by (20), the Bayesian diffusion scheme intelligently discards information according to a hierarchy of importance as dictated by the Fisher metric. In this respect we hope that Bayesian renormalization can inspire new approaches to information theoretic optimal diffusion learning while also providing insights into the underlying information theoretic character of both diffusion learning and generic renormalization.

To this end, although we have introduced Bayesian renormalization through its relationship with Bayesian inference, we should stress that it motivates a methodology that can be undertaken without the need to first perform a Bayesian inference experiment. In particular, we can renormalize a family of probability distributions by diffusing it through the space of possible models with the diffusion kernel $\mathcal{N}(\gamma_{\tau}, \tau K^{-1}_{\tau})$; one can understand this procedure as information theoretic form of diffusion learning where equation (46) provides an explicit proposal for a forward diffusion process which coarse grains information in a hierarchy of importance as specified by the Fisher metric. In turn, the reverse SDE associated with (46) through the identification (26) has an explicit interpretation as encoding the Bayesian learning of the data generating distribution in the time T corresponding to the reintegration of removed data. Here, $K^{-1}_{\tau}$ should be chosen by first computing the Fisher information matrix of the model space of interest, and subsequently pushing this matrix forward in the sample space by selecting a function $G: \mathcal{M} \rightarrow S$. The potential function V_τ can be chosen arbitrarily, and will govern the mean tendency of the diffusion process. Thus, the Bayesian renormalization scheme corresponds to the choices of G and V_τ , with $\mathcal{I}$ being fixed by the system. This is significant because $\mathcal{I}$ is what encodes the information theoretic scale of the problem.

The identification of the mapping G should be thought of as a form of data compression in which a deterministic model for the data realizations in terms of the parameters is postulated. Over the course of the Bayesian renormalization, there will be a flowing in these parameters resulting in the formulation of an effective model. This procedure is related to a variety of different data compression techniques such as the information bottleneck [37] and variational autoencoding [56]. In this way, the Bayesian renormalization scheme makes direct contact with previous insights into the relationship between model building/dimensional reduction for complex systems and the RG such as [19–21, 36].

An autoencoder is a type of (artificial) neural network trained using unsupervised learning which is used to create efficient encodings of a set of unlabeled input data. By definition, autoencoders are composed of two blocks: an encoder and a decoder; typically both blocks are trained simultaneously.

Assuming the encoded and decoded message spaces are Euclidean, let

$$\begin{align} \chi & = \mathbb{R}^n& \tilde \chi & = \mathbb {R}^m. \end{align} \tag{ 48 }$$

where n denotes the dimension of the input data, and m the dimension of the latent space chosen when the network is initialized. For each data instance $y \in \chi$, we denote the encoded latent space representation as $\tilde y \in \tilde \chi$. Let E_θ be the encoder function, parameterized by a set of variables $\{\theta_i\}$ and mapping from the space of decoded (source) messages χ to the space of encoded messages $\tilde \chi$,

$$\begin{equation} E_\theta: \chi \to \tilde \chi. \end{equation} \tag{ 49 }$$

Similarly, D_φ is the decoding function, parameterized by a second set of variables $\{\phi_j\}$ and mapping from the space of encoded messages $\tilde \chi$ to the space of decoded messages χ,

$$\begin{equation} D_\phi: \tilde \chi \to \chi. \end{equation} \tag{ 50 }$$

As a choice of autoencoder architecture, we consider a simple dense, (linear) multi-layer perceptron network with no biases, n input neurons, and m latent space neurons (where the number of output neurons is thus constrained also to n). Selecting the architecture as a multi-layer perceptron network is intentional: since each node is fully connected with an associated weight parameter, one may expect the network to have many weights that covary comparatively weakly with the output of the network. A schematic summary of the network architecture is presented in figure 1.

Zoom In Zoom Out Reset image size

In the case of the aforementioned network, we choose the element-wise encoder and decoder layers to be

$$\begin{align} \left(E_\theta\right)_i & = \alpha_\text{Enc}\left(\theta_{ij} x^j\right)& \left(D_\phi\right)_j & = \alpha_\text{Dec}\left(\phi_{ij} x^j\right), \end{align} \tag{ 51 }$$

where $\alpha_\text{Enc}, \alpha_\text{Dec}$ are the encoder and decoder activation functions ¹⁴ . Each parameter of the encoder and decoder networks are elements of the corresponding weight matrix θ_ij —the machine learner may be more familiar with the usual form $\alpha(Wx + b)$ where W is the typical weight matrix and b is the corresponding bias vector. For all activations, we choose the rectified linear unit activation defined as

$$\begin{equation} \alpha\left(x\right) = \max\left(0, x\right) : = \begin{cases} x\ \text{if}\ x > 0\\ 0 \ \text{otherwise.} \end{cases} \end{equation} \tag{ 52 }$$

Let $D \subset \chi$ denote the set of training data. Upon the introduction of each new item of training data, $y \in D$, the parameters of the autoencoder are updated ¹⁵ to minimize a given loss function $L(y \mid \theta,\phi)$. In the following numerical examples, we consider the typical mean squared error (MSE) loss defined by

$$\begin{equation} L\left(y \mid \theta, \phi\right) = \frac{1}{n} \sum_{j = 1}^{n} \left(y_j - D_{\phi}E_{\theta}\left(y\right)_j\right)^2 . \end{equation} \tag{ 53 }$$

In the course of this note, we have stressed that a crucial question in any data science context is how we can arrive at an effective model for a given system using only the most relevant set of parameters. Indeed, it is typical for models to possess parameters that are comparatively less sensitive than others, and we have argued that these can be identified systematically via the Fisher matrix. As we have stressed in table 1 a neural network, like our autoencoder, can be understood as a family of statistical models quantified in terms of the various parameters used to initialize its architecture. In our case, each set of parameters $(\theta,\phi)$ corresponds to a different autoencoder, with our best estimate of the 'true' or optimal autoencoder given by the value of these parameters which is realized after all of the training data have been utilized. We denote these optimal parameters by $(\theta^*,\phi^*)$. Appealing again to table 1, we can give a more decidedly probabilistic viewpoint on the family of autoencoders by regarding each autoencoder $(\theta,\phi)$ as initializing a probability distribution over data of the form

$$\begin{equation} p_{Y \mid \Theta,\Phi}\left(y \mid \theta,\phi\right) \propto \text{exp}\left(-L\left(y \mid \theta,\phi\right)\right). \end{equation} \tag{ 54 }$$

Then, the minimization of the loss can equivalently be regarded as a maximization of the log-likelihood specified in (54).

Regarding the set of all possible autoencoders as a family of statistical models, we can employ the machinery developed in section 3.1 to endow this space with an information geometry. In particular, the Fisher matrix of the encoder layer may be calculated explicitly from the trained model using the approach outlined in [75]. During this analysis the trained parameters of the decoder are left untouched—this allows comparison between pruning methods while eliminating inter-layer interactions. We denote by $\mathcal{I}_{ij}$ the components of the resulting Fisher matrix. Because it is computed after training one should regard this as the Fisher metric evaluated at the point $(\theta^*,\phi^*)$.

As we have discussed, the eigenvalues of the Fisher metric serve as a proxy for the relevance of the various parameters which are included in the model class. From a physical perspective, one may think of each parameter θ_i as a mode of a particular theory, and of its associated diagonal element in the Fisher metric $\mathcal{I}_{ii}$ as corresponding to its characteristic 'length scale'. Parameters with small Fisher diagonals vary weakly under the acquisition of new data and therefore require either extremely fine measurements or very large amounts of data in order to substantiate even an incremental change in their trained value. For this reason, we can think of these parameters as changing only on very fine length scales in the space of models. Conversely, parameters with larger Fisher values vary strongly with the model and therefore possess features that can be distinguished more broadly with less or less fine-tuned data.

In light of these observations, we now propose the most simple-minded form of Bayesian renormalization. We introduce a sliding parameter, Λ, and divide the set of parameters into two subsets:

$$\begin{equation} \Theta^{>}_{\Lambda} \equiv \left\{\theta_i \; | \; \mathcal{I}_{ii} > \Lambda \right\}, \qquad \Theta^{<}_{\Lambda} \equiv \left\{\theta_i \; | \; \mathcal{I}_{ii} \unicode{x2A7D} \Lambda\right\}. \end{equation} \tag{ 55 }$$

The set $\Theta^{\gt}_{\Lambda}$ consists of those parameters that covary with the model output on 'length' scales greater than Λ. We refer to these parameters equivalently as relevant/stiff parameters relative to the scale Λ for obvious reasons. Conversely, the set $\Theta^{\lt}_{\Lambda}$ consists of those parameters that covary with the model output on 'length' scales below Λ. We refer to these parameters equivalently as irrelevant/sloppy relative to the scale Λ. The renormalization procedure we shall consider is simply to remove the parameters in $\Theta^{\lt}_{\Lambda}$. This realizes a reduced/renormalized model that depends only on the parameters $\Theta^{\gt}_{\Lambda}$. One should think of this as akin to a hard cutoff regularization scheme in a quantum field theory (QFT) in which modes that depend on physics at length scales smaller than a given cutoff Λ are systematically removed.

For our experiment, we begin by training the autoencoder described in section 4.1 on the MNIST dataset. Recall that MNIST is a series of approximately 50 thousand training samples and ten thousand testing samples consisting of $28 \times 28$ pixel (784 dimensional) images depicting handwritten digits 0 through 9. The network and optimizer ¹⁶ settings are outlined in table 2.

Table 2. Values of configurable properties of the autoencoder network.

Model property	Value
Input (output) dimension	784
Latent space dimension	20
Number of encoder parameters (pre-prune)	15680
Mean diagonal Fisher value	0.7148
Optimizer	ADAM
ADAM $(\alpha, \beta_1, \beta_2, \epsilon)$	$(0.001, 0.9, 0.999, 10^{-8})$
Cutoff Λ	$(0, 0.125, 0.25, \ldots, 2.375)$

After having trained the model, we perform the Bayesian renormalization scheme outlined above. The result of this procedure at various values of the cutoff parameter Λ can be seen in figure 3. To illustrate the sense in which the Bayesian RG scheme more adequately coarse grains the model with respect to the information contained in its various parameters we have also provided two alternative schemes. In the first scheme, demonstrated in figure 4, the same number of parameters are removed in each iteration as in the Fisher-inspired scheme, but using a uniform distribution over the parameters of the model. The second alternative scheme is a popular form of pruning for neural networks which we refer to as 'magnitude inspired pruning'. In the magnitude inspired approach a fixed number of parameters are removed in each iteration corresponding to the remaining parameters with the smallest absolute value, hence the name.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

As one can see in figure 3, the autoencoder remains remarkably effective even once more than half of its parameters have been removed, provided these parameters are removed using the Bayesian scheme. To understand this result it is instructive to look at the distribution of (diagonal) Fisher elements associated with the encoder—see figure 5. Here one can see that there is a large gap in the spectrum at $\Lambda \sim 0.1$—we regard this gap as identifying the genuinely 'sloppy' parameters in the model. In figure 2 we explore the loss landscape as a function of the number of removed parameters, comparing the Fisher-motivated scheme with random parameter removal and magnitude inspired pruning. In regards to this comparison there are two important points to be made. Firstly, the Fisher-motivated scheme produces minimal losses up to the aforementioned cutoff at $\Lambda \sim 0.1$. Shortly thereafter, the losses of the Fisher motivated pruning scheme exceed those of the magnitude scheme. This is expected; for $\Lambda \gt 0.1$ the Fisher scheme begins to remove parameters that are relevant to the performance of the model. The loss responds strongly to the removal of these parameters. In other words, figure 2 demonstrates that the Fisher scheme effectively distinguishes both irrelevant and relevant parameters.

Zoom In Zoom Out Reset image size

A different way of understanding the preceding discussion serves to contextualize the relationship between RG and statistical learning. One can imaging performing the pruning procedure in reverse, whereby one incorporates new parameters one at a time in order to improve the model's performance. From this perspective, figure 2 demonstrates that the Fisher scheme most rapidly reaches minimal losses among the three approaches here considered.

One can summarize the preceding discussion by observing the distinctive shape of the loss curve produced by the Fisher pruning scheme. Up to the aforementioned 'scale' of $\Lambda \sim 0.1$ the loss is essentially stable to the pruning of parameters. Thereafter, the removal of subsequent parameters results in noticeable decreases in the performance of the model. The result is a 'hockey-stick' shaped curve: for $\Lambda \lesssim 0.1$ the pruning removes irrelevant parameters that covary weakly with the output of the model, and thus only marginally affect the loss; likewise for $\Lambda \gt 0.1$ the pruning begins to remove 'relevant' parameters that covary heavily with the output of the model. In other words, the Bayesian RG scheme has identified a characteristic 'scale' in the model and has dictated that the inclusion of parameters which are irrelevant with respect to that scale may be excluded from the model without any change in performance—this is precisely the goal of an RG flow. It is instructive to recognize that the notion of scale which is relevant in this instantiation of RG has a direct relationship with the number of parameters included in the model.