Removing cloud shadows from ground-based solar imagery

This work was funded by ANR grant N° ANR-20-CE23-0014-01. One of the authors (JA) thanks CNES (Centre National d’Etudes Spatiale, the French space agency) for its support in this project.

This paper addresses the problem of cleaning ground-based solar imagery from cloud shadow contaminants, as illustrated in Fig. 1, in support to forthcoming analyses with solar physics and space weather applications. Monitoring the activity of the Sun is a task of growing importance in the frame of space weather research. The main sources of severe space weather disturbances at Earth are energetic outbursts from the Sun, e.g. solar flares and coronal mass ejections. They are strongly associated with solar structures of the chromosphere (e.g. active regions (AR) and filaments, as illustrated in Fig. 5), visible from the Earth ground in the light emitted by ionised calcium (Ca-II, at a wavelength of 3933.7Å) and the H-$\alpha$ line of neutral hydrogen (6562.8Å). Despite their affordability, images taken from the ground present a particular difficulty: they are prone to be tainted by cloud shadows when the sky is not clear. These shadows are rarely uniform and may be mistaken for solar structures (e.g. filaments). Thus, it is substantial to remove them before automatic solar structure detection and monitoring.

Very few previous works addressed this problem. [1, 2] designed denoising methods for H-$\alpha$ images. Feng et al. [1] proposed a metric for assessing the level of cloud coverage in an image by considering the homogeneity of image intensity between different quadrants of the solar disk. Additionally, they proposed a denoising method that first estimates the pixel-wise transmittance of clouds $\hat{I}_{t}$ from a cloud-free temporal neighbour using a low-pass morphological filter to smooth out solar structures (i.e. filaments). Second, it uses it to normalise the original image $I_{\text{ini}}$:

In [2], Fuller and Aboudarham perform two stages of median filtering to identify large scale intensity variations that correspond to clouds, while thresholding out the solar structures. In both methods, the use of low-pass filtering to distinguish clouds from solar structures is a possible limitation when their spatial scales are similar in at least one direction.

In other application domains, some works removed large contaminants, such as clouds and shadows, from remote sensing or natural images. Due to clouds transparency, the task of removing clouds in Sun observations resembles to a degree to these tasks. While there is a large literature on denoising methods e.g. [3], we focus on the removal of large and semi-transparent contaminants, as this is the closest to our task. Methods that perform inpainting over opaque contaminants, such as [4] for thick clouds, are not considered to avoid introducing artefacts in the study of solar structures (images with thick clouds are therefore not usable). We also discard methods that exploit the physics of image formation for natural images, e.g. [5, 6, 7, 8], as solar imaging follows a different physics. Thin cloud and shadow removal methods are good candidates for our task of removing semi-transparent clouds while preserving the background solar structures.

Some shadow suppression methods e.g. [9] rely upon the sparse approximation theory. A local dictionary of atoms is learnt from image patches as a basis for sparse coding. This allows decomposing the image into reflectance (invariant to shading), geometry (fine structural details), and shadow components, the latter being excluded during image reconstruction. The use of local dictionaries, rather than generic ones (e.g. wavelets, curvelets), helps in learning atoms that adapt to each single image. However, this approach cannot gather further insights from multiple examples.

This limitation is addressed by (deep) learning-based methods. [10, 11] use encoder-decoder deep neural networks (DNN) to remove shadows by learning a mapping between the initial images $I_{ini}$ and their binary mask $\hat{M}$ of shadow ratio, which is reminiscent of the transmittance image of [1]:

[12] and pix2pix [13] in [4] rely on skip connections (U-Net-style) to preserve spatial resolution, and a conditional generative adversarial network (C-GAN) for more efficient training. They obtain state-of-the-art (SoTA) results in shadow and thin cloud removal. In [12], an additional C-GAN outputs the denoised image from predicted shadow mask, while pix2pix of [4] outputs the cleaned image directly.

We explore the applicability of these denoising methods for the new application of removing cloud shadows from solar imagery. We evaluate sparse representation-based and deep learning-based approaches, and compare against [1, 2] (Section 4). Inspired by [12, 4], we use a combination of U-Net and C-GAN (Section 3) to obtain the new SoTA for this task.

The methods are comparatively assessed on two imaging modalities. To support these experiments, we introduce two new datasets of Sun images with artificial and real clouds.

Our images are from Paris-Meudon (PM) observatory¹¹1The Paris-Meudon observatory is part of Observatoire de Paris-PSL: https://www.observatoiredeparis.psl.eu/-observatoire-de-paris-.html?lang=en. They are $1024\times 1024$ px full-disk spectroheliograms. Care has been taken in their collection to ensure a good representativity. We used images of two modalities: Ca-II (for studying ARs), and H-$\alpha$ (for filaments), as highlighted in Fig. 5. These are major modalities in ground-based solar imaging. Observations span different periods of the solar cycle in order to reflect variations in the Sun’s activity and number / size of the solar structures that may be observed at the surface of the Sun. 96 Ca-II and 111 H-$\alpha$ images are sampled between May 2002 and June 2017. Sampling uses the metric of cloud coverage level proposed by [1] to select the images that are tainted with cloud shadows.

PM images are taken in principle during clear sky days. However, even during those days, small thin clouds emerging in the horizon cannot be prevented and result in light cloud stains. Cloud shadows can come with different textures. In PM observatory, the Sun disk is scanned by a thin moving slit over 20 to 100 sec. [14]. In case of fast enough cloud motion, due to the slit movement alongside the motion of clouds, cloud spots may be shifted along the scanning process, creating straight lines on the image (denoted as “streaked” texture). Slower cloud motions result in the “fluffy” texture that is customary of many terrestrial clouds. It is also a common texture in solar observations from observatories that use 2D cameras rather than a spectrograph. The “streaked” texture may provide more challenges to denoising methods, due to having different spatial scales in different directions. Fig. 5 illustrates samples of streaked and fluffy textures with different cloud cover thicknesses and streak orientations.

The cloudy images don’t have cloud-free counterparts for training DNNs. Thus, we introduce a dataset of real (initially cloud-free) images with added synthetic cloud shadows. Using the cloud coverage level metric of [1], we selected reasonably cloud-free images in the same time period as above. A basic cloud texture was obtained from [15]. For each cloud-free image, a new cloud shadow was composed by summing a random number (in range [2, 3]) of duplicates of the basic texture with horizontal and vertical random resizing between 50-100% of the original size, random flips (around one or two axes), and random transparencies (in range [0.1, 0.4]). It was then overlapped to the cloud-free image. This procedure provides realistic images of fluffy and semi-transparent cloud shadow contamination. They are diverse and representative of possible cloud shapes and opacity levels. The Ca-II and H-$\alpha$ synthetic datasets comprise respectively 319 and 367 pairs of shadow/shadow-free images, split into 179/44/96 and 205/51/111 training/validation/testing pairs. The combination of our real and artificial cloud datasets provides a variety of streaked and fluffy cloud textures. They are publicly available²²2https://doi.org/10.5281/zenodo.7684200, along with the pretrained cloud-removal DNNs³³3The source code and model weights are available via https://gitlab.lis-lab.fr/presage/cloud-removal-from-solar-imagery..

Images have their intensity normalised in range $[0,1]$, centred solar disks with normalised radii, and background pixels are set to zero outside of the solar disk to remove any distracting features from background stars and imaging artefacts.

We adopt the deep learning approach due to its SoTA results in large contaminants removal. As in [12, 13], we use skip connections to preserve spatial resolution and details⁴⁴4We use the U-Net implementation from https://github.com/mateuszbuda/brain-segmentation-pytorch.. Inspired by the results of previous methods, we compare the classical fully supervised and the C-GAN training strategies.

We experiment with our DNNs predicting either the cleaned image directly, or a cloud shadow mask, to be used as in [1, 10, 11]. To avoid division by zero and improve stability, we compute the clean image as:

We empirically verified that the exact value of $\epsilon$ has little impact on the results, and we set it to $1e^{-5}$ in our experiments. Regressing a shadow mask instead of the cleaned image directly is motivated by the physics of solar imaging, as in [1]. In addition, estimating a transmittance or shadow ratio, rather than the pixel’s absolute intensity, reduces the risk to create artefacts that would hinder the forthcoming study of solar structures. Finally, estimating a shadow mask should be a less complex task for the DNN, which may be learnt more efficiently. Indeed, reproducing the exact patterns of the more complex and diverse solar structures in the output image would be an unnecessary additional task for the DNN.

With Eq. 3, the loss gradient varies as $I_{\text{ini}}/\hat{M}^{2}$, meaning that the darkest pixels of the images, where cloud thickness is high, see higher loss gradients and require a finer adjustment of their mask value. In an attempt to improve stability during training through more consistent loss gradient magnitudes, we also experiment with an alternative implementation, i.e. adding in the intensity that was removed by clouds. This is similar to predicting a residual mask:

We compare two training strategies: a fully supervised DNN (denoted as FS) and a C-GAN following the pix2pix setup. For ease of comparison, the FS DNN and C-GAN generator have the same U-Net-style architecture${}^{\ref{foot:unet}}$: down-sampling and up-sampling blocks along with skip connections, each block as defined in [16]. We experiment with a varying number of blocks to determine the optimal complexity and receptive field needed to account for both large scale contaminants and smaller scale solar structures. Due to memory constraints, we go up to 6 down/up-sampling blocks, and find it to be the best performing architecture (see supplementary materials). In the future, and on a different hardware, more blocks could be tried.

We use a linear activation in the last block when predicting $\hat{I}_{\text{clean}}$ directly or $\hat{M}$ as in Eq. 4. For Eq. 3, we find that imposing the output to be in range $[0,1]$ is necessary to avoid instability issues, so we use $0.5\times\tanh+0.5$ as activation.

The discriminator in the C-GAN is the same as in pix2pix [13]: a PatchGAN network that considers image patches (small receptive fields) to capture high-frequencies. We experimentally verified that the recommended receptive field of $70\times 70$ is also optimal in our application. The discriminator is applied to the cleaned image rather than the predicted shadow mask to assess the preservation of solar structures.

For the C-GAN, we use the loss function suggested in [13]. The loss function of the FS network and of the C-GAN generator uses the pixel-wise $L_{1}$-norm since it promotes sparsity and encourages less blurring as noted in [13]. The pixel-level loss is computed between the target $I_{t}$ and cleaned image $\hat{I}_{\text{clean}}$:

For both DNNs, we use the default U-Net${}^{\ref{foot:unet}}$ training scheme and batch size of 3. We train for 100 epochs and test on the epoch with the best validation loss (FS DNN) or generator’s training loss (C-GAN, following the usual practice for GANs). During testing of the C-GAN, the generator is run with dropout, and batch normalisation considers the statistics of the batch, as in [13].

We investigated deep learning methods for removing large cloud shadow contaminants from ground-based solar imagery. We adopted the U-Net-style architecture for more detailed outputs. We compared three possible inference methods (i.e. direct and through a transmittance or residual mask), and two training setups: fully-supervised and C-GAN. The DNNs obtained better results than the domain’s SoTA and a non-leaning method based on sparse approximation, with better restoration of the underlying solar structures, and are the new SoTA. The two training setups have different strengths and weaknesses: full supervision produces more homogeneous solar disks that are more physically plausible, and C-GAN better handles strong clouds and thick filament restoration. Thus, there is no obvious “best setup”, and the choice of training setup depends on the intended usage.

The DNNs have been trained on fluffy cloud textures only. While they demonstrate a good generalisation to streaked cloud textures, further improvement might be obtained from training on streaked clouds in the future. Future work may also further focus on filament restoration, and evaluate the effect of cloud removal on a subsequent image analysis e.g. the automatic detection of solar structures.

Supplementary information

Supplementary materials are provided with additional comparative results.

Acknowledgments

This work was funded by ANR grant N° ANR-20-CE23-0014-01. One of the authors (JA) thanks CNES (Centre National d’Etudes Spatiale, the French space agency) for its support in this project.

The data used in this study is available at https://doi.org/10.5281/zenodo.7684200. The code is available at https://gitlab.lis-lab.fr/presage/cloud-removal-from-solar-imagery.

Adeline Paiement and Jean Aboudarham contributed to the study conception and design. Data preparation was performed by Jay Paul Morgan, experiments were performed by Amal Chaoui and Jay Paul Morgan, analysis was performed by Amal Chaoui, Jay Paul Morgan and Adeline Paiement. All authors contributed to the manuscript.