Event-based Shape from Polarization with Spiking Neural Networks

Shurcliff proposed the method of shape recovery by polarization information in 1962 [37]. Essentially, when unpolarized light reflects off a surface point, it becomes partially polarized. And the observed scene radiance varies with changing the polarizer angle, which encodes some relationship with surface normals. Therefore, by analyzing such relationship at each surface point through Fresnel equations [38], shape from polarization methods can measure the azimuthal and zenithal angles at each pixel and recover the per-pixel surface normal with high resolution. Generally, two schemes are utilized to collect polarization images. One is Division of Time (DoT) [13, 19, 20] that provides full-resolution polarization images but increases the acquisition time significantly, while the other one is Division of Focal Plane (DoFP) [16, 17, 21] that trade-offs spatial resolution for low latency. After collecting the polarization images, various physical-based or learning-based methods [39] can be utilized to estimate the surface normals. However, since a linear polarizer cannot distinguish between polarized light that is rotated by $\pi$ radians, this results in two confounding estimates for azimuth angle at each pixel [16, 40]. To solve such the ambiguity, we have to carefully design the estimation methods by exploring additional constraints from various aspects, such as geometric cues [41, 42, 43], spectral cues [14, 44, 45], photometric cues [15, 46, 47], or priors learned from deep learning techniques [16, 17].

Recently, with the prevalence of bio-inspired neuromorphic engineering, researchers begin to shift their focus to high-speed energy-efficient event cameras and propose solutions that combine polarization information with event cameras. Specifically, inspired by the polarization vision in the mantis shrimp eye [48], [49] proposed the PDAVIS polarization event camera. The researchers employed the DoFP scheme to design such the camera. This involved fabricating an array of pixelated polarization filters and strategically positioning them atop the sensor of an event camera. While this camera is adept at capturing high dynamic range polarization scenes with high speeds, it still faces challenges with low spatial resolution, a common issue inherent in the DoFP methods. To bridge the high resolution of DoT with the low latency of DoFP, [22] adopted the DoT scheme and collected polarization events by placing a rotating polarizing filter in front of an event camera. Due to the high resolution of DoT and the low latency of event cameras, this method facilitates shape from polarization at both high speeds and with high spatial resolution. Typically, the captured polarization events are transformed into frame-like event representations [50], which are then processed using ANN models [22] to estimate surface normals. While these learning-based methods demonstrate superior performance over traditional physics-based methods, they significantly increase the energy consumption of the overall system, primarily due to the lower energy efficiency of ANNs. Through processing event polarization data collected by the promising DoT scheme, this paper aims to address this challenge by conducting event-based shape from polarization using SNNs, presenting a more energy-efficient alternative in this domain.

With the development of ANNs, artificial intelligene models today have demonstrated extraordinary abilities in many tasks, such as computer vision, natural language processing, and robotics. Nevertheless, ANNs only mimic the brain’s architecture in a few aspects, including vast connectivity and structural and functional organizational hierarchy [24]. The brain has more information processing mechanisms like the neuronal and synaptic functionality [51, 52]. Moreover, ANNs are much more energy-consuming compared to human brains. To integrate more brain-like characteristics and make artificial intelligence models more energy-efficient, researchers propose SNNs, which can be executed on power-efficient neuromorphic processors like TrueNorth [53] and Loihi [54]. Like ANNs, SNNs are capable of implementing common network architectures, such as convolutional and fully-connected layers, yet they distinguish themselves by utilizing spiking neuron models [30], such as the Leaky Integrate-and-Fire (LIF) model [29] and the Spike Response Model (SRM) [28]. Due to the non-differentiability of these spiking neuron models, training SNNs can be challenging. However, progress has been made through innovative approaches such as converting pre-trained ANNs to SNNs [55, 56] and developing methods to approximate the derivative of the spike function [57, 58]. Thanks to the developement of these optimization techniques, several models have been proposed recently to tackle the complex tasks in 3D scenes. Notably, StereoSpike [59] and MSS-DepthNet [60] have pioneered the development of deep SNNs for depth estimation, achieving performance on par with the state-of-the-art ANN models. Additionally, SpikingNeRF [61] has successfully adapted SNNs for radiance field reconstruction, yielding synthesis quality comparable to ANN baselines while maintaining high energy efficiency. In this paper, our emphasis is on employing SNNs to tackle event-based shape from polarization, aiming to establish a method that is not only effective but also more efficient for event-based surface normal estimation.

To ensure a fair comparison between our proposed methods and those utilizing ANNs for event-based shape from polarization, we transform the sparse event stream into frame-like event representations, which serve as the input for our methods. Specifically, similar to [22], we take the CVGR-I representation due to its superior performance. The CVGR-I representation combines the Cumulative Voxel Grid Representation (CVGR) with a single polarization image (I) taken at a polarizer angle of 0 degrees. The CVGR is a variation of the voxel grid [50]. Similar to previous works on learning with events [62, 63], the CVGR first encodes the events in a spatio-temporal voxel grid $V$. Specifically, the time domain of the event stream is equally discretized into $B$ temporal bins indexed by integers in the range of $[0,B-1]$. Each event $e_{i}=(x_{i},y_{i},t_{i},p_{i})$ distributes its sign value $p_{i}$ to the two closest spatio-temporal voxels as follows:

$$V(x,y,t)=\sum_{x_{i}=x,y_{i}=y}p_{i}\max(0,1-|t-t_{i}^{*}|),\hskip 8.53581ptt_{i}^{*}=\frac{B-1}{\Delta T}(t_{i}-t_{0}),$$

(2)

where $(x,y,t)$ is a specific location of the spatio-temporal voxel grid $V$, $\Delta T$ is the time domain of the event stream, and $t_{0}$ is the timestep of the initial event in the event stream. Then, the CVGR calculates the cumulative sum across the bins and multiplies this total by the contrast threshold:

$$E(x,y,b)=C\sum_{i=0}^{b}V(x,y,i),\hskip 8.53581ptb=\{0,1,2,3,...,B-1\},$$

(3)

Finally, to enhance surface normal estimation in areas with insufficient event information, a single polarization image of 0 polarizer degree is incorporated, resulting in $E=I[0]+E$, thereby providing additional context. This resulting event representation $E$ will serve as the input of our models. Its dimensions are $B\times H\times W$, where $H$ and $W$ represent the height and width of the event camera, respectively. We present a concrete input example of “cup” in Fig. 1.

Spiking neuron models are mathematical descriptions of specific cells in the nervous system. They are the basic building blocks of SNNs. In this paper, we primarily concentrate on using the Integrate-and-Fire (IF) model [29] to develop our proposed SNNs. The IF model is one of the earliest and simplest spiking neuron models. The dynamics of the IF neuron $i$ is defined as:

$$u_{i}(t)=u_{i}(t-1)+\sum_{j}w_{ij}x_{j}(t),$$

(4)

where $u_{i}(t)$ represents the internal membrane potential of the neuron $i$ at time $t$, $u_{i}(t-1)$ is the membrane potential of the neuron $i$ at the previous timestep $t-1$, and $\sum_{j}w_{ij}x_{j}(t)$ is the weighted summation of the inputs from pre-neurons at the current time step $t$. When $u_{i}(t)$ exceeds a certain threshold $u_{th}$, the neuron emits a spike, resets its membrane potential to $u_{reset}$, and then accumulates $u_{i}(t)$ again in subsequent time steps.

In addition to the IF model, we also build our proposed models with the Leaky Integrate-and-Fire (LIF) model [29]. Compared to the IF model, LIF model contains a leaky term to mimic the diffusion of ions through the membrane. The dynamics of the LIF neuron $i$ can be expressed as:

$$u_{i}(t)=\alpha u_{i}(t-1)+\sum_{j}w_{ij}x_{j}(t),$$

(5)

where $\alpha$ is a leaky factor that decays the membrane potential over time. Drawing inspiration from previous work [64], we also construct models using the Parametric Leaky Integrate-and-Fire (PLIF) model, which enables automatic learning of the leaky factor. In our experiments, we demonstrate that the IF model can offer better performance as it retains more information by not incorporating the leaky factor, thus striking a balance between high performance and biological plausibility.

In this section, we propose two SNNs that take the CVGR-I event representation as the input and estimate the surface normals $\mathbf{N}$. Both of them can process the information through the spiking neuron models mentioned above. Due to the potential of IF neurons in event-based shape from polarization, we will present the proposed models based on the dynamics of IF neurons.

We normalize outputs from spiking neurons into unit-length surface normal vectors $\mathbf{\hat{N}}$ and then apply the cosine similarity loss function:

$$\mathcal{L}=\frac{1}{H\times W}\sum_{i}^{H}\sum_{j}^{W}(1-\left<\mathbf{\hat{N}}_{i,j},\mathbf{N}_{i,j}\right>),$$

(10)

where $\left<\cdot,\cdot\right>$ indicates the dot product, $\mathbf{\hat{N}}_{i,j}$ refers to the estimated surface normal at the pixel location $(i,j)$, while $\mathbf{N}_{i,j}$ denotes the ground truth surface normal at the same location. The objective is to minimize this loss, which is achieved when the orientations of $\mathbf{\hat{N}}_{i,j}$ and $\mathbf{N}_{i,j}$ align perfectly.

To optimize the Single-Timestep Spiking UNet, we utilize the backpropagation method [68] to calculate the weight updates:

$$\Delta w^{l}=\frac{\partial\mathcal{L}}{\partial w^{l}}=\frac{\partial\mathcal{L}}{\partial o^{l}}\frac{\partial o^{l}}{\partial u^{l}}\frac{\partial u^{l}}{\partial w^{l}},$$

(11)

where $w^{l}$ is the weight for layer $l$, $o^{l}$ is the output of spiking neurons in layer $l$, and $u^{l}$ is the membrane potential of spiking neurons in layer $l$. Similarly, to optimize the Multi-Timestep Spiking UNet, we utilize the BackPropagation Through Time (BPTT) [69] to calculate the weight updates. In BPTT, the model is unrolled for all discrete time steps, and the weight update is computed as the sum of gradients from each time step as follows:

$$\Delta w^{l}=\sum_{t=0}^{B-1}\frac{\partial\mathcal{L}}{\partial o_{t}^{l}}\frac{\partial o_{t}^{l}}{\partial u_{t}^{l}}\frac{\partial u_{t}^{l}}{\partial w^{l}},$$

(12)

where $w^{l}$ is the weight for layer $l$, $o_{t}^{l}$ is the output of spiking neurons in layer $l$ at the time step $t$, and $u_{t}^{l}$ is the membrane potential of spiking neurons in layer $l$ at the time step $t$. Based on the Heaviside step functions in Eq. 6 and Eq. 8, we can see that both $\frac{\partial o^{l}}{\partial u^{l}}$ and $\frac{\partial o_{t}^{l}}{\partial u_{t}^{l}}$ cannot be differentiable in spiking convolutional layers. To overcome the non-differentiability, we use the differentiable ArcTan function $g(x)=\frac{1}{\pi}arctan(\pi x)+\frac{1}{2}$ as the surrogate funciton of the Heaviside step function [70]. For the final prediction layer with potential-assisted spiking neurons, since they output membrane potential instead of spikes, we have $\frac{\partial o^{l}}{\partial u^{l}}=1$ and $\frac{\partial o_{t}^{l}}{\partial u_{t}^{l}}=1$ for these layers’ weight updates.

Our models are implemented with SpikingJelly [71], an open-source deep learning framework for SNNs based on PyTorch [72]. To fairly compare with the counterpart ANN models, we ensure our models have the similar settings like the ANN models in [22]. Specifically, we set $B=8$ for the input event representation. In addition, our models have $N_{e}=4$ encoder blocks and $N_{d}=4$ decoder blocks. And the event encoding module outputs the binary spiking representation with the channel size of $N_{c}=64$. For the spiking-related settings, all the spiking neurons in the spiking convolutional layers are set with a reset value ($u_{reset}$) of 0 and a threshold value ($u_{th}$) of $1$. Following [64, 73], normalization techniques are applied after each convolution (Conv) operation for faster convergence. We train our models for $1000$ epochs with a batch size of 2 on Quadro RTX 8000. We use the Adam [74] with a learning rate of $1e-4$ to optimize our models.

We evaluate our proposed models on two latest large-scale datasets for event-based shape from polarization, including the ESfP-Synthetic Dataset and ESfP-Real Dataset.

The ESfP-Synthetic Dataset was generated using the Mitsuba renderer [75], which created scenes with textured meshes illuminated by a point light source. For each scene, a polarizer lens, positioned in front of the camera, was rotated through angles ranging from 0 to 180 degrees with 15 degrees intervals, producing a total of 12 polarization images. With these images, events were simulated using ESIM [76] with a 5% contrast threshold. Therefore, each scene in the dataset is accompanied by rendered polarization images, simulated events, and groundtruth surface normals provided by the renderer.

The ESfP-Real Dataset is the first large-scale real-world dataset for event-based shape from polarization. It contains various scenes with different objects, textures, shapes, illuminations, and scene depths. The dataset was collected using a Prophesee Gen 4 event camera [77], a Breakthrough Photography X4 CPL linear polarizer [78], a Lucid Polarisens camera [21], and a laser point projector. Specificially, the polarizer rotated in front of the event camera that captured the events for each scene in the dataset. The Lucid Polarisens camera was used to collect polarization images of the same scene at 4 polarization angles {0, 45, 90, 135}. And the groundtruth surface normals were generated using Event-based Structured Light [79], a technique that involves integrating the laser point projector with the event camera.

We evaluate our models against the state-of-the-art physics-based and learning-based methods in the field of shape from polarization. Smith et al. [47] combine the physics-based shape from polarization with the photometric image formation model. The method directly estimates lighting information and calculates the surface height using a single polarization image under unknown illumination. Mahmoud et al. [80] present a physics-based method to conduct shape recovery using both polarization and shading information. Recently, Muglikar et al. [22] are pioneers in addressing event-based shape from polarization, employing both physics-based and learning-based approaches. Their models are notable for directly using event data as inputs. In this paper, our focus is on comparing our proposed models with the learning-based model developed by Muglikar et al. We aim to demonstrate that our SNN-based models can match their performance while offering greater energy efficiency.

To evaluate the accuracy of the predicted surface normals, we employ four metrics: Mean Angular Error (MAE), % Angular Error under 11.25 degrees (AE$<$11.25), % Angular Error under 22.5 degrees (AE$<$22.5), and % Angular Error under 30 degrees (AE$<$30). MAE is a commonly used metric that quantifies the angular error of the predicted surface normal, where a lower value indicates better performance [16, 17]. The latter three metrics, collectively referred to as angular accuracy, assess the proportion of pixels with angular errors less than 11.25, 22.5, and 30 degrees, respectively, with higher percentages indicating better accuracy [22].

We thoroughly evaluate our proposed models on the ESfP-Synthetic Dataset, using both quantitative metrics and qualitative analysis. Specifically, Table 1 presents the performance of both baselines and our methods in surface normal estimation on the ESfP-Synthetic Dataset. In addition, Figure 4 showcases the qualitative results of our models and the ANN counterpart on the ESfP-Synthetic Dataset.

From Table 1, we can see that our proposed models significantly outperform the physics-based methods. The reason why our model can achieve the better performance is that our models benefit from the large-scale dataset and utilize the spiking neurons to extract useful information for event-based shape from polarization. Despite this success, our models do not quite match the overall performance of their ANN counterpart on this dataset, likely due to the limited representation capacity of spiking neurons. However, as Fig. 4 illustrates, our Multi-Timestep Spiking UNets still manage to achieve comparable, and in some cases superior, results in shape recovery across various objects in the test set, compared to the ANN models.

Table 1 clearly demonstrates that the temporal dynamics inherent in spiking neurons enable the Multi-Timestep Spiking UNets to surpass the Single-Timestep versions in surface normal estimation. Additionally, nearest neighbor sampling, as compared to bilinear upsampling, shows comparable performance while preserving the binary nature and compatibility with SNNs.

Recognizing the effectiveness of Multi-Timestep Spiking UNets, we undertook an ablation study aimed at identifying the ideal spiking neurons to fully leverage their temporal dynamic capabilities. The results, detailed in Table 2, indicate that IF neurons offer superior performance. This is largely due to their ability to retain more extensive temporal information, as they operate without the influence of a leaky factor.

We also compare these methods on the ESfP-Real Dataset. Specifically, we show the quantitative performance in Table 3 and illustrate the qualitative results in Fig. 5. Similar to the results on the ESfP-Synthetic Dataset, our models demonstrate superior performance compared to physics-based methods on the real-world dataset. Moreover, as indicated by Table 3 and Fig. 5, our models not only match the overall performance of the ANN counterpart but also excel in qualitative results across diverse scenes in the test dataset. This enhanced performance on the ESfP-Real Dataset can be attributed to the sparser nature of this real-world dataset [22]. In addition, compared to the ANN counterpart, our model is more compatible with the sparse events and better maintains the sparsity to prevent overfitting on this dataset.

Mirroring the outcomes observed on the ESfP-Synthetic Dataset, results from Table 3 and Fig. 5 also show that the Multi-Timestep Spiking UNet slightly outperforms the Single-Timestep Spiking UNet. Additionally, nearest neighbor upsampling is on par with bilinear upsampling in terms of surface normal estimation performance.

In earlier sections, we demonstrated that our models, employing nearest neighbor upsampling, can achieve performance comparable to those using bilinear upsampling in event-based shape from polarization. To delve deeper into the advantages of these fully spiking models, we will now estimate the computational cost savings they offer compared to their fully ANN counterpart [22] on the ESfP-Real Dataset. Commonly, the number of synaptic operations serves as a benchmark for assessing the computational energy of SNN models, as referenced in studies like [81] and [82]. Moreover, we can approximate the total energy consumption of a model using principles based on CMOS technology, as outlined in [83].

Unlike ANNs, which consistently perform real-valued matrix-vector multiplication operations regardless of input sparsity, SNNs execute computations based on events, triggered only upon receiving input spikes. Therefore, we initially assess the mean spiking rate of layer $l$ in our proposed model. In particular, the mean spiking rate for layer $l$ in an SNN is calculated as follows:

$$F^{(l)}=\frac{1}{T}\sum_{t\in T}\frac{S^{(l)}_{t}}{K^{(l)}}$$

(13)

where $T$ is the total time length, $S^{(l)}_{t}$ is the number of spikes of layer $l$ at time $t$, and $K^{(l)}$ is the number of neurons of layer $l$. Table 4 shows the mean spiking rates for all layers in our fully spiking models, including the Single-Timestep Spiking UNet and Multi-Timestep Spiking UNet. Notice that we do not consider the components without trainable weights, such as max pooling and nearest neighbor upsampling layers. From the table, we can see that the Multi-Timestep Spiking UNet exhibits a higher average spiking rate across all its layers compared to the Single-Timestep Spiking UNet. This increased spiking rate aids in preserving more information, thereby enhancing the accuracy of surface normal estimation.

With the mean spiking rates, we can estimate the number of synaptic operations in the SNNs. Given $M$ is the number of neurons, $C$ is the number of synaptic connections per neuron, and $F$ indicates the mean spiking rate, the number of synaptic operations at each time in layer $l$ is calculated as $M^{(l)}\times C^{(l)}\times F^{(l)}$. Thus, the total number of synaptic operations in an SNN is calculated by:

$$\#OP=\sum_{l}M^{(l)}\times C^{(l)}\times F^{(l)}\times T.$$

(14)

In contrast, the total number of synaptic operations in the ANNs is $\sum_{l}M^{(l)}\times C^{(l)}$. Due to the binary nature of spikes, SNNs perform only accumulation (AC) per synaptic operation, while ANNs perform the multiply-accumulate (MAC) computations since the operations are real-valued. Based on these, we estimate the number of synaptic operations in the our proposed models and their ANN counterpart. Table 5 illustrates that, in comparison to ANNs, our models primarily perform AC operations with significantly fewer MAC operations that transform real-valued event inputs into binary spiking representations. Furthermore, the Multi-Timestep Spiking UNet executes more AC operations than the Single-Timestep Spiking UNet due to its higher average spiking rate and the utilization of temporal dynamics across multiple timesteps.

In general, AC operation is considered to be significantly more energy-efficient than MAC. For example, an AC is reported to be $\mathbf{5.1\times}$ more energy-efficient than a MAC in the case of 32-bit floating-point numbers (0.9pJ vs. 4.6pJ, 45nm CMOS process) [83]. Based on this principle, we obtain the computational energy benefits of SNNs over ANNs in Table 5. From the table, we can see that the SNN models are 3.14$\times$ to 28.80$\times$ more energy-efficient than ANNs on the ESfP-Real Dataset.

These results are consistent with the fact that the sparse spike communication and event-driven computation underlie the efficiency advantage of SNNs and demonstrate the potential of our models on neuromorphic hardware and energy-constrained devices.