Transfer learning application of self-supervised learning in ARPES

Angle-resolved photoemission spectroscopy (ARPES) is a powerful tool to visualize the electronic band structure, which has been used mostly for condensed matter physics. Complexities in ARPES measurements and data analysis start to manifest themselves as measurements can now be performed in multiple dimensions almost simultaneously, demanding the development of automation and assistive tools to help the experiment. In short, ARPES evolved stepwise from one-dimension (1D) (photoelectron intensity (I) vs. energy) available in x-ray photoelectron spectroscopy [1], to 2D where small 1D detectors are placed in an array creating a 2D analyzer slit for angle-resolved measurements (I vs. angle ($\phi$), energy axes) (figure 1(c)) [2-6] with a typical resolution of $\sim\!{0.1}^\circ$, to 3D (I vs. angle 1 ($\theta$), angle 2 ($\phi$), energy axes) (figure 1(d)) by either moving the 2D analyzer slit along the perpendicular angle or by rotating the sample itself [7-10]. This allows us to explore momentum space in two dimensions, i.e. $\left( {{k_x},{k_y},E} \right)$ axis while the remaining perpendicular momentum ${k_z}$ can be explored by varying the incoming photon energy ($h\nu$), thus completing the picture of the band structure in 3D solids [11, 12] where Fermi surface tomography can also be performed [13]. The other dimension like time ($t$) can be investigated [14, 15] depending upon the stimulus (e.g. sample cleave or laser pulse) applied at t = 0 to start the experiment while the spin information $\left( {{s_x},{s_y},{s_z}} \right)$ carried by the photo-emitted electrons can be probed by a spin detector, e.g., VLEED [16] or Mott detector [17], establishing both in-plane and out-plane components.

Zoom In Zoom Out Reset image size

Meanwhile, the introduction of a micrometer size (and smaller) beam spot with conventional laser, synchrotron, or free electron laser photon sources, expands the technique into a microscopy, where a multidimensional ARPES data set is associated with each spot position on the sample [18-20]. This means that we can spatially resolve different band structures (I vs. angle ($\phi$), energy axes), forming potentially 10D data sets of (I vs. $x$, $y,$ angle 1 ($\phi$), angle 2 ($\theta$), perpendicular momentum (${k_z})$, spin (3 directions), energy axes, and time after, e.g., the start of the experiment or application of an optical stimulus). Although we are yet to arrive at this complexity (10D dataset) [18-20], the current micro $\left( \mu \right)$ or nano (n)—ARPES measurements (4D dataset, I vs. $x$, $y,$ angle 1 ($\phi$), energy) already pose difficulties in the following ways. First, the spatial data points grow as ${N^2}$, where $N$ is the spatial scan size, increasing the analysis time quadratically if performed without any automation. Following that, the original interest in mapping out different band structures in real space is usually circumvented during the experiment and data analysis by directly plotting the integrated intensity I of the 2D analyzer as a function of position (x, y), reducing the data dimension to two (figure 1(e)) with effectively no analysis time needed. Especially during the experiment, we can also bin the pixels from the 2D analyzer reducing the file size (easily reaching ∼20 GB worth of 4D data), and also shorten the analyzer acquisition time to reduce the experiment time, which will unfortunately reduce the signal to noise ratio. For example, the work in [21] shows the plot of the integrated intensity over the dispersion cut (I vs. angle ($\phi$), energy axes) onto the real space, visualizing domains where different numbers of layers of graphene lie. Further visualization of the band structure can be done by specifically revisiting the points of interest and performing a high statistic ARPES measurement. Meanwhile, a synchrotron-based measurement can also trace core levels or any other bands lying deep below the Fermi surface to distinguish different chemical environments experienced by the element of interest, e.g., areas with different termination [22]. In any case, the spatial domain assignment task and the visualization of the band structure are performed separately, e.g., one file for spatial domains determined from core levels and several files for band structures at different positions. However, there are cases where the core level is not available, e.g., for laser-based ARPES or simply the spatial inhomogeneities are electronic and not chemical and therefore only visible in the band structure, as for example in [23, 24]. Therefore, it is still desirable to perform a proper 4D-data measurement where for each spatial position we still have ARPES dispersion with good statistics from which we map the differences spatially (creating a spatial domain), trace the differences in a fine step (especially through the domain), and ultimately perform the analysis automatically in a memory saving way without any human intervention.

Labeling each band structure observed (I vs. angle ($\phi$), energy axes) at different positions ($x$, $y$) is a tedious yet important task. This task can be even more challenging, especially if we are running against the decay time of a clean sample surface or simply because the statistics are limited by the time available for the experiment. Plotting the integrated intensity (including intensity normalization relative to some standard feature visible in each band structure slice) can be a preliminary step to draw and usually is the first-to-go technique. Subsequently, one may find a unique region of interest that can define a representative band structure distinct from others. Yet, this depends on the case-specific judgment and can take precious time during the measurement. Therefore, it is ultimately favorable to have an automated procedure to help the labeling/clustering of different band structures so we can focus on the physics problem in an efficient, unbiased fashion.

Meanwhile, there have been attempts to implement techniques available in the machine learning field to scientific experiments like scanning tunneling [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib25" id="fnref-mlstaced7dbib25"="">25</a>–<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib28" id="fnref-mlstaced7dbib28"="">28</a>] and atomic force microscopies [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib29" id="fnref-mlstaced7dbib29"="">29</a>, <a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib30" id="fnref-mlstaced7dbib30"="">30</a>], where the majority entails supervised models that need manual labeling, to begin with. There is also similar work for various ARPES applications. For example, a deep-layer convolutional neural network (ConvNet) has been trained to denoise ARPES data [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib31" id="fnref-mlstaced7dbib31"="">31</a>]. Afterward, there have been efforts to determine how the band structure calculation is related to the ARPES data [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib32" id="fnref-mlstaced7dbib32"="">32</a>, <a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib33" id="fnref-mlstaced7dbib33"="">33</a>], where the [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib33" id="fnref-mlstaced7dbib33"="">33</a>] provides an additional feature for obtaining the result even through noisy data (simulated noise). There has been also work in automation of spatial domain assignment with a smaller subset of data over a predetermined area, where subsequent position measurement is calculated with the Gaussian process to give the possible highest amount of information [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib34" id="fnref-mlstaced7dbib34"="">34</a>]. Our work here is in the same direction as [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib34" id="fnref-mlstaced7dbib34"="">34</a>], which is to automate domain assignment. However, our approach has a stark difference: we use representation learning from self-supervised models to represent our ARPES images.

In the context of computer vision, we need to distinguish the term 'to classify' and 'to cluster'. The term classify usually refers to supervised labeling where there is a definite set of labels for these band structures. For example, we can have a clear pre-defined label of 'with-gap' and 'no-gap' band structure which each ARPES cut can be categorized as. Meanwhile, clustering refers to the same-ness, like-ness, or affinity to a set of standard images to which band structures are compared; different image references will create their different clusters. From the ARPES perspective, the final product of 'to cluster' and 'to classify' can be identical, which is a list of numbers indicating the group this band structure belongs to, yet the methods to produce the list are technically different.

Naturally, one may resort to the automated labeling technique called supervised learning, e.g. convolutional neural network (ConvNet) [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib35" id="fnref-mlstaced7dbib35"="">35</a>] in this case and come up with a trained neural network model that can classify the given band structure with a well-defined label. The ConvNet is a well-known and robust machine learning model to learn image representation by learning from examples. Despite this, the ConvNet model may lack generality in the low-resource setting when the model is trained with very limited data and we need to train the model from scratch. Meanwhile, the amount of data from each measurement may not be sufficient for a neural model to be trained properly, despite the many available data enhancement procedures [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib33" id="fnref-mlstaced7dbib33"="">33</a>].

We may argue that any well-known pre-trained deep neural network like ResNet50, which is pre-trained with ImageNet dataset, can be used in the model initialization, where subsequent training with our training dataset will be done; the processes are described in figure 2, and the details about the dataset used are in the supporting information section B (SI.B). In short, we run the evaluation using k-fold cross-validation, and the performance metrics are calculated by taking the average performance on all folds. In this approach, we set the result in a probabilistic manner for each label (total of four labels) on one image. The performance of this fine-tuned ResNet50 is evaluated by the metrics, such as accuracy, F1, precision, and recall, as shown in table 1; refer to appendix for the definition of each metric element. The approach clearly demonstrates some usefulness as the accuracy, and F1 is relatively high, given the sparsity of the training data. Thus, this positive result suggests leveraging pre-trained models for ARPES data in supervised learning scenarios. In figure 2, we display how all the data are subsequently predicted by fine tuning the ResNet50 model, where the group probability of each real space position is shown. We can see that the result of the ResNet50 prediction can be used as a guide to investigating different domains in the experimental context. It also shows that the ResNet50 trained with generic images can handle experimental data images with small fine-tuning. However, it should be restated that this approach lacks generality as the fine-tuned ResNet50 model can only be used for the Fe₃Sn₂ data; a new sample needs a new re-training which implies the need for a new set of labeled data, which is precisely what we aim to avoid. Following this, the retraining can be expensive computationally (involving GPU) and not something favorable to be performed during the ARPES measurement itself as it also costs valuable time.

Zoom In Zoom Out Reset image size

Another trivial attempt in the automation can be made by an unsupervised clustering method, such as <em="">k</em>-means [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib36" id="fnref-mlstaced7dbib36"="">36</a>], where the data can be clustered according to their affinity to each other; usually, the Euclidean distance of the feature dimension is used to define the affinity. The typical data processing done before applying this procedure is to flatten the picture, i.e. convert the 2D array of the image into a 1D array and perform the <em="">k</em>-means clustering in this 'hyper-dimensional' array. However, the 'curse of dimensionality', as coined by Bellman [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib37" id="fnref-mlstaced7dbib37"="">37</a>], might play a role where this many-dimension vector effectively hides the feature that needs to be captured. For this issue, one may do the common dimension reduction technique such as principal component analysis [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib38" id="fnref-mlstaced7dbib38"="">38</a>] or a <em="">t</em>-distributed stochastic neighbor embedding (<em="">t</em>-SNE) [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib39" id="fnref-mlstaced7dbib39"="">39</a>] to reduce the number of dimensions and subsequently perform the <em="">k</em>-means clustering on this reduced dimension space. This approach may help solve the curse of dimension problem, but table <a xmlns:xlink="http://www.w3.org/1999/xlink" class="tabref" href="#mlstaced7dt2"="">2</a> shows that the performance metric is still comparable when the image data is simply flattened and clustered (method = 'none' in table <a xmlns:xlink="http://www.w3.org/1999/xlink" class="tabref" href="#mlstaced7dt2"="">2</a>). In any case, some kind of 'flattening' is still done on the picture where the object of interest is locked into some position in the array. This will greatly affect the result as the feature of interest needs to be on the same array pointer, and some data preprocessing needs to be done to keep the features in place before flattening.

Recently, pre-trained self-supervised models are utilized to generate high-dimensional representation of data, where instead of comparing the input data directly, the model will map the input data onto a representational space on which further training can be done, or simply use them for clustering [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib40" id="fnref-mlstaced7dbib40"="">40</a>]; the input data can be speech, text, or images (refer to supporting information for details). In short, these models learn the feature from the input data in a self-supervised way, which means the training objective is generated without using annotated human labels but using pretext tasks. In the context of natural science experiments, there are already attempts to apply some form of self-supervised learning, for example in the training of self-supervised model built for single cell image analysis [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib41" id="fnref-mlstaced7dbib41"="">41</a>], training of self-supervised models to denoise tomography data (Noise2inverse) [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib42" id="fnref-mlstaced7dbib42"="">42</a>, <a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib43" id="fnref-mlstaced7dbib43"="">43</a>], analysis of electrocardiography database with a general purpose self-supervised model [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib44" id="fnref-mlstaced7dbib44"="">44</a>], quantifying hidden features in single-molecule charge transport data with transfer learning [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib45" id="fnref-mlstaced7dbib45"="">45</a>], etc. Meanwhile, the field of computer science has already introduced several general-purpose self-supervised models for computer vision, such as MoCo [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib46" id="fnref-mlstaced7dbib46"="">46</a>], SimClr [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib47" id="fnref-mlstaced7dbib47"="">47</a>], BYOL [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib48" id="fnref-mlstaced7dbib48"="">48</a>], SwAV [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib49" id="fnref-mlstaced7dbib49"="">49</a>], and DINO [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib50" id="fnref-mlstaced7dbib50"="">50</a>], which ARPES can take advantage of through <strong=""> <em="">transfer learning</em> </strong>, i.e. utilize a pre-trained model that was trained using self-supervision fashion on large image datasets (e.g., ImageNet [<a xmlns:xlink="http://www.w3.org/1999/xlink" class="cite" href="#mlstaced7dbib51" id="fnref-mlstaced7dbib51"="">51</a>]). The summary of the procedure is described in the supporting information and the clustering pipeline is shown in figure <a xmlns:xlink="http://www.w3.org/1999/xlink" href="#mlstaced7df3"="">3</a>. Table <a xmlns:xlink="http://www.w3.org/1999/xlink" class="tabref" href="#mlstaced7dt2"="">2</a> shows the result of <em="">k</em>-means clustering performance metrics done on the representation space of DINO, SwAV, MoCo, BYOL, with and without <em="">t</em>-SNE dimension reduction. We can see that the performance is worse than the ResNet50 supervised performance (table <a xmlns:xlink="http://www.w3.org/1999/xlink" class="tabref" href="#mlstaced7dt1"="">1</a>) and unfortunately it is even comparable to not using a pre-trained self-supervised model (method = 'none') on the image; dimension reduction also offers no significant help in this approach. This result may imply that the features of ARPES images are not captured well in the knowledge extracted from the ImageNet database; this invites the need for a collective ARPES database for training purposes and also a novel objective task of self-supervised models for ARPES specifically or for any scientific method which relies heavily on images. It can also mean that ARPES images from the Fe<sub="">3</sub>Sn<sub="">2</sub> dataset are not well separated in the representation space; the representation learning space on images may be well spread, yet they do not make well-separated islands on which <em="">k</em>-means clustering can be of any use.

Zoom In Zoom Out Reset image size

As a current solution, we introduce a k-nearest neighbor (kNN) procedure (the pipeline is shown in figure 4), where instead of direct clustering on the representation space, we take k-numbers of images as references to which the rest of the images can be compared. The references here are the images with a known label set by the expert. The input images and the references are first converted into the representation space by passing to a pre-trained self-supervised model accordingly and the Euclidean distances of the converted images are measured with respect to the converted references. The label is then determined by the closest reference. In our case of Fe₃Sn₂, we have four labels and thus k-numbers of images on each label. In the case of $k = 1$, we can see that this approach is equivalent to generating k-means clustering prediction but with a pre-defined centroid (four pre-defined centroids for the case of Fe₃Sn₂). This problem can be a quick solution in the case when the images in the representation learning space are not well separated yet they are well spread. As we increase k, we expect the model to be more able to discriminate the given input. The pipeline of this process is drawn in figure 4.

Zoom In Zoom Out Reset image size

Table 3 is the summary of the kNN performances. We can see that as we set $k = 20$ their performances are roughly similar to each other with ranking as follows: DINO, none, BYOL, MoCo, SwAV. This result is expected because as more known labels are given, we are approaching more towards a supervised method like ConvNet above. This result also tells us that different representational techniques capture different features of ARPES data differently as their performances vary between techniques. Importantly, we can see that the DINO technique captures ARPES features better than the other techniques. Reducing the k down to the single reference case $k = 1$ also reveals the strength of the DINO model [50] (we average 50 different single references for each technique) as the performance still compares favorably to the other techniques. We display the attention map of DINO in the SI for further visualization on how it works.

From the kNN results, we propose the following pipeline to finally solve our problem of automated domain labeling in pseudocode 1. As the spatial scan begins, we take the first image as our reference ${\text{ }}k = 1$ case. Afterward, the subsequent measured images' affinity to the reference is calculated (in representation learning space). For each measured image, the affinity is considered if it has exceeded a certain threshold. The threshold itself is an arbitrary value that will be determined as the experiment runs (the expert may intervene during the experiment to decide the threshold value). At some point, the Euclidean distance of an image might be relatively further than the others. In this situation, we (as the expert during the measurement) may decide if the new image is regarded as a new label or not, and thus we may determine if a new label is needed. Afterward, the experiment again proceeds with the updated number of labels. Finally, the experiment loop will end with a complete set of ARPES data set and their labels as measured with respect to the decided references. The method described here is still not fully automatic. Nonetheless, it is a semi-supervised way of automation and is already a significant improvement with the help of the learned representation from self-supervised models.

====== pseudocode 1. kNN pipeline proposal ======

%%% pseudocode for kNN pipeline %%%

Require: $M$ as the list of ARPES matrices

Require: $L$ as the number of labels

Require: $r$ as the list of APRES reference images, where ${r_i}$ is the image for label $i$

Require: $\Theta$ as the pretrained model for all $m$ in $M$:

$x = \Theta \left( m \right)$ #get the representation of an image $m$

${\text{re}}{{\text{f}}_x} = \Theta \left( r \right)$ closest_label = arg_min ${i_1}$ to $L$ (euclidean_distance($x,{\text{re}}{{\text{f}}_{{x_i}}}$)) if closest_label &gt; threshold: prompt_user

End for

====== pseudocode 1. kNN pipeline proposal ======

We demonstrate that ARPES, which relies heavily on pictorial data analysis, can take advantage of the recent development of representational learning in the computer vision field for help in automation. In this work, we demonstrate the transfer learning application of pre-trained self-supervised models that are applied to ARPES images in supervised and unsupervised manners. We apply the KNN method, where the affinities of the ARPES images are measured with respect to reference images, and we show that using only a single reference image can achieve acceptable performance. The kNN proposal presented here is not limited to ARPES image analysis and can be used for other experimental techniques that yield images and sparsely labeled data. Our work urgently invites the creation of an expanded ARPES image database on which machine-learning models can be trained. This will entail further interdisciplinary research collaborations between ARPES and computer vision to find more suitable representational learning methods for ARPES. We anticipate that similar paradigms will advance the analysis of other measurements producing high dimensional data sets.

We thank Benjamin Bejar Haro from Laboratory for Simulation and Modelling, Paul Scherrer Institute, for the insightful discussion on the k-shot model.

We thank Yao Mengyu for his participation in the ARPES measurement.

We thank Yihao Wang and Yimin Xiong for providing us with the sample.

We thank Felix Baumberger for providing access to the micro-focused laser ARPES setup.

We thank Matthew Watson from i05 Diamond Light Source for the valuable discussion in spatially resolved ARPES.

We thank Vincent Tatan for the early discussion on the ConvNet technique.

S A E and G A acknowledge the support from the European Research Council HERO Synergy Grant SYG-18 810451, NCCR-MARVEL funded by the Swiss National Science Foundation, the European Union's Horizon 2020 research and innovation program under the Marie Skłodowska-Curie Grant Agreement No. 701647.

The data that support the findings of this study are available upon reasonable request from the authors.

The study does not include human subjects, human data or tissue, or animals.

The authors declare no competing interests.

Accuracy (ACC)

$$\begin{equation*}{\text{ACC}} = \frac{{{\text{TP}} + {\text{TN}}}}{{{\text{P}} + {\text{N}}}}.\end{equation*}$$

F1

$$\begin{equation*}{\text{F}}1 = \frac{{2{\text{TP}}}}{{2{\text{TP}} + {\text{FP}} + {\text{FN}}}} = \frac{{2 \times {\text{PRE}}\, \times {\text{REC}}}}{{{\text{PRE}} + {\text{REC}}}}.\end{equation*}$$

Recall (REC) or true positive rate (TPR) or sensitivity (SEN)

$$\begin{equation*}{\text{REC}} = \frac{{{\text{TP}}}}{{\text{P}}} = \frac{{{\text{TP}}}}{{{\text{TP}} + {\text{FN}}}}.\end{equation*}$$

Precision (PRE) or positive predictive value (PPV)

$$\begin{equation*}{\text{PRE}} = \frac{{{\text{TP}}}}{{{\text{PP}}}} = \frac{{{\text{TP}}}}{{{\text{TP}} + {\text{FP}}}}.\end{equation*}$$

From the definition above, imbalanced data may create stark differences between REC and PRE as in our Fe<sub="">3</sub>Sn<sub="">2</sub> example, where this value is reflected in F1. In this case, F1 is a good indicator for the goodness of the model as it 'averages' REC and PRE.