Improved Tomographic Binning of 3 × 2 pt Lens Samples: Neural Network Classifiers and Optimal Bin Assignments

The CosmoDC2 simulation covers 440 deg² on the sky and is complete to a magnitude r = 28 and contains galaxies up to a redshift of 3. The CosmoDC2 catalog was created using dark matter particles from the Outer Rim simulations (Heitmann et al. 2019). Galaxies with a limited set of specified properties were then assigned to dark matter halos using the UniverseMachine simulations (Behroozi et al. 2019) and the GalSampler technique (Hearin et al. 2020). These objects were matched to the outputs of the Galacticus model (Benson 2012), which generated complete galaxy properties.

The Buzzard catalog was developed to simulate DES Year 1 data and contains galaxies up to a redshift of 2.3 over an 1120 deg² area and complete to r ∼ 26. The Buzzard catalog is based on dark matter simulations from L-GADGET2 (Springel 2005). Galaxies were added to dark matter halos based on abundance matching using AddGals (Wechsler et al. 2022). Galaxies are drawn from a distribution of luminosities and overdensities and matched to halos of the same overdensity. Spectral energy distributions (SEDs) are assigned to match the SED−luminosity−density relationship measured from Sloan Digital Sky Survey (SDSS) data.

The LSST Dark Energy Science Collaboration (DESC) Tomographic Challenge (Zuntz et al. 2021) was designed as a first step toward optimizing the tomographic binning for the source sample of galaxies in a 3 × 2 analysis. Participants in the challenge were invited to use any method they liked for determining the binning but were limited to using (g)riz photometry. This choice was made to reflect the fact that the DESC plans to compute shears using the metacalibration technique (Sheldon & Huff 2017; Sheldon et al. 2020), which requires selections to be performed only in bands in which the point-spread function (PSF) is well measured.

For use in the Tomographic Challenge, the CosmoDC2 and Buzzard catalogs had noise added to simulate real LSST Year 1 observations using the DESC TXPipe ⁶ framework and following the methodology of Ivezic et al. (2019). The catalogs were further cut based on signal-to-noise ratio (S/N) and galaxy size to simulate the selection criteria used for real lensing catalogs. Galaxies in the full simulated catalogs were kept for the final sample of galaxies if the combined riz S/N satisfied S/N > 10 and the size satisfied T/T_psf > 0.5. Here T measures the squared, deconvolved radius of the galaxy and is the trace I_xx + I_yy of the moments matrix, and T_psf is derived from the assumed FWHM of the Rubin PSF, T_fwhm = 075. After these cuts, the CosmoDC sample contained 36 million objects and the Buzzard sample contained 20 million objects. From these samples, 25% were randomly assigned to the training sample, 50% to the validation sample, and the remaining 25% to the application sample, making the training sample fully representative. For this work, we have made use of the Tomographic Challenge training samples, which we have split into training and validation samples, and the Tomographic Challenge validation samples as our application sample.

While the Tomographic Challenge methods were built and run using a training sample that was fully representative of the application sample, this will not be the case with LSST data. Previous attempts to account for the effects of a nonrepresentative training sample can be found in Beck et al. (2017) and Stylianou et al. (2022).

To divide the CosmoDC2 and Buzzard data sets into training and application samples that are nonrepresentative, we make use of the second data release (HSC DR2) of the Hyper Suprime-Cam Subaru Strategic Program (HSC SSP; Aihara et al. 2018). From the HSC Wide (300 deg² at i < 26.2; Aihara et al. 2019) catalog, we selected objects with i ≤ 25 and either g ≤ 27.5, r ≤ 27.7, or z ≤26.2, which mimics early LSST observations, as noted in Broussard & Gawiser (2021). After this photometric cut, there are 3,469,800 remaining objects; a color–magnitude diagram of a randomly selected subset of these objects is shown in the middle panel of Figure 1 of Broussard & Gawiser (2021). This sample is named the HSC phot sample.

Zoom In Zoom Out Reset image size

In addition, some galaxies in the HSC Wide field have been matched to spectroscopic catalogs from zCOSMOS (Lilly et al. 2009), WiggleZ DR1 (Drinkwater et al. 2010), DEEP3 (Cooper et al. 2011), PRIMUS DR1 (Coil et al. 2011; Cool et al. 2013), DEEP2 DR4 (Newman et al. 2013), UDSz (Bradshaw et al. 2013; McLure et al. 2013), VVDS (Le Fèvre et al. 2013), 3D-HST (Skelton et al. 2014; Momcheva et al. 2016), VIPERS PDR1 (Garilli et al. 2014), FMOS-COSMOS (Silverman et al. 2015; Kashino et al. 2019), GAMA DR2 (Liske et al. 2015), SDSS DR12 (Alam et al. 2015), and the SDSS IV QSO catalog (Pâris et al. 2018). There are 307,193 objects meeting the same photometric requirements as above that also have matched spectroscopic redshifts. A color–magnitude diagram of this sample, named HSC spec, is shown in Broussard & Gawiser (2021) in the left panel of their Figure 1.

To build the nonrepresentative training sample, we divide the space of i-band magnitude and (g-z) color into 100 × 100 bins. In each bin, we compute the ratio between HSC spec objects with spectroscopic redshifts and HSC phot objects. Even at fixed brightness and color, galaxy spectral features can vary significantly, and spectroscopic confirmation rates are often significantly lower for higher-redshift galaxies in a given color–magnitude bin. To model this, we find the 99th percentile in spectroscopic redshift within each bin and assign that as ${z}_{\max }$ for that color–magnitude bin.

Objects in the CosmoDC2 and Buzzard catalogs are then sorted into the same bins in color–magnitude space. In each bin, objects with photometric redshifts greater than ${z}_{\max }$ are automatically assigned to the application sample. The remaining objects are then randomly assigned to the training or application sample based on the ratio of HSC spec to HSC phot objects. If there were no HSC spec objects in a given color–magnitude bin, all of the objects were assigned to the application sample. This method of producing a nonrepresentative training sample has been incorporated into the DESC RAIL ⁷ framework as the GridSelection degrader. Inevitably, this is still an approximation of true nonrepresentativeness, which can get worse than what is presented here. As an example, a spectroscopic survey might use B-band selection, which is not directly replicable with the ugrizy photometry available for LSST. The nonrepresentative sample presented here may still prove to be somewhat optimistic, unless efforts are made to obtain spectroscopic samples with a greater coverage of color–magnitude–redshift space.

Figure 1 shows the partitioning of the CosmoDC2 sample into training and application samples. Critically, the training sample is brighter (the training sample has median i-band magnitude of 21.1) than the application sample (median i-band magnitude of 23.9), as is typical for spectroscopic follow-up of deep photometric surveys. The mean color of the training sample is also redder, as is expected. The median (g − z) color of the training sample is 2.20, while the median color of the application sample is 1.55. The training sample shown in Figure 1 is used for training TPZ and the NNCs and represents a spectroscopic sample, while the application sample is used for estimating photo-z's, NNC confidences, and the binning results and represents LSST data.

A further illustration of the nonrepresentative nature of the training sample we have built can be seen in Figure 2, which shows the normalized true redshift distribution of the selected application sample compared to that of the selected training sample. The redshift distribution of the training sample falls off above a redshift of 1.0 with a tail to z = 1.5, while the application sample falls off more gradually and extends out to a redshift of 3. In the application sample 34% of the objects are at z > 1.0, while in the training sample only 0.085% of objects are at z > 1.0. Additionally, the peak of the redshift distribution of the training sample occurs at lower redshifts than for the application sample. Note that the heights of the peaks in Figure 2 are not directly comparable; the training sample is much smaller overall.

Zoom In Zoom Out Reset image size

3.1.1. Training Sample Size

TPZ is a machine-learning method of photo-z estimation that uses random forests, and we used the regression mode for this work. We used 100 trees for the random forest, with a minimum leaf size of 30. TPZ also allows us to choose m_*, the number of features that are randomly selected for splitting a given tree node; we provide TPZ with 11 features for training and choose m_* = 6.

We provide the following 11 features as inputs to TPZ for training: the apparent magnitudes in the ugrizy filters and the colors (u–g), (g–r), (r–i), (i–z), and (z–y). Galaxies with a nondetection in a given band are assigned a magnitude of 30 in that band. We additionally tried adding the band triples, defined as the difference between neighboring colors in Broussard & Gawiser (2021). In analogy to the manner in which colors act as a derivative of the magnitudes, band triples act as a second derivative. From the set of features we provide, the available band triples are [(u–g) – (g–r)], [(g–r) – (r–i)], [(r–i) – (i–z)], and [(i–z) – (z–y)]. We find here that including the triples in the training does improve the photo-z estimation by a marginal amount, but the NNCs do just as well without them, and it is too computationally expensive to be worth including them.

TPZ provides three outputs for each galaxy in the application sample: z_phot, the mean of the probability distribution function (pdf); σ_TPZ, the associated Gaussian uncertainty in the z_phot estimate; and z_conf, the integrated probability between ${z}_{\mathrm{phot}}\pm {\sigma }_{\mathrm{TPZ}}\left(1+{z}_{\mathrm{phot}}\right)$.

Tomographic Challenge methods were evaluated using two metrics: the S/N of the angular power spectra derived from the n_i (z) distribution of each bin i, and the FOM. The FOM comes in two flavors: the S₈ − Ω_m FOM, and the w₀ − w_a FOM, which constrains the dark energy equation-of-state parameters (Albrecht et al. 2006). We choose to focus on the w₀ − w_a FOM (hereafter simply FOM) as our metric for optimizing the binning choices since this is specific to the DESC goal of measuring the dark energy equation-of-state parameters. The binning optimization procedure we outline in the rest of the paper can be used for other science goals, but a different metric may be preferable in those cases.

Each metric has contributions from galaxy clustering and weak-lensing components, which are combined to form the 3 × 2pt FOM. Since assumptions made about galaxy bias in the Tomographic Challenge lead to the total 3 × 2pt FOM being dominated by the clustering signal, we continue to use the 3 × 2pt FOM despite focusing on the choice of lens sample. In doing so, we have assumed a source sample that is equivalent to the lens sample, even though the source sample cannot be selected using all of the bands in this work. This makes the exact FOM values reported here somewhat optimistic, but it is the relative changes in the FOM between binning choices that are most important. In order to directly compare the tomographic binning method described here to the results of the Tomographic Challenge, we use the same method for calculating the FOM.

The FOM is calculated using the inverse area of a Fisher matrix ellipse representing the area of the posterior pdf of the two parameters we want to measure. The FOM is then related to the constraining power of the chosen binning scheme: a higher FOM corresponds to a smaller area, meaning tighter constraints on w₀ and w_a . The Fisher matrix is given by

$$\begin{eqnarray}&&F={\left(\displaystyle \frac{\partial \mu }{\partial \theta }\right)}^{T}{C}^{-1}\left(\displaystyle \frac{\partial \mu }{\partial \theta }\right),\end{eqnarray} \tag{ 1 }$$

where μ is the vector of theoretical predictions for the C_ℓ spectra for each pair of tomographic bins, θ are the list of cosmological parameters included in the theory, and C is a Gaussian estimate of the covariance between them following Takada & Jain (2004). To compute the C_ℓ , we use the true redshift distribution, n_i (z), which is the distribution of the actual redshifts of the galaxies in each bin, rather than the distribution of photometric redshifts. The C_ℓ are calculated for 100 values of ℓ between 100 and 2000. More details can be found in the Appendix of Zuntz et al. (2021).

The FOM is then calculated as

$$\begin{eqnarray}&&\mathrm{FOM}=\frac{1}{2\pi \sqrt{\det \left({\left[{F}^{-1}\right]}_{{p}_{1},{p}_{2}}\right)}},\end{eqnarray} \tag{ 2 }$$

where p₁ = w₀ and p₂ = w_a extracts the 2 × 2 submatrix of F⁻¹ corresponding to the dark energy equation-of-state parameters. The list of parameters used for calculating the Fisher matrix is the native combination in the DESC core cosmology library (Chisari et al. 2019): Ω_c , Ω_b , H₀, σ₈, n_s , w₀, and w_a . The Tomographic Challenge FOM calculation does not include any galaxy bias values as nuisance parameters, which leads to the overall 3 × 2 pt FOM being dominated by the galaxy clustering signal.

Our analysis required some changes to the FOM calculation from the Tomographic Challenge. The Challenge used a hard-coded density of galaxies on the sky, which we updated to reflect the actual galaxy density of the CosmoDC2 and Buzzard simulations. The CosmoDC2 simulation covers 440 deg² on the sky, and the final CosmoDC2 sample contained 36M objects, so this density was used as the starting density. Although we did not use the entire Challenge catalog, we used the appropriate sky density for the sample to obtain comparable FOM values. Instead of hard coding a sky density, this ensures that sample cuts using the NNCs will become susceptible to shot noise in bins with few galaxies, thereby reducing the FOM as appropriate. We also updated the hard-coded fraction of the sky covered by the survey. We implemented two options: scaling the survey area to match the fraction of the catalog we used to maintain a comparable survey density, or assuming the sky density for a given simulation but scaled to the LSST year 10 survey area. The rest of this paper reports results for the assumed year 10 area.

Once photo-z estimates are made, galaxies can be binned. Based on the optimization of galaxy clustering S/N explored in Broussard & Gawiser (2021), we utilize 12 tomographic bins covering 0 < z < 3. We start with three possible binning options: bin edges equally spaced in redshift (equal Δz binning), bin edges equally spaced in comoving distance (equal Δχ binning), and bins with equal numbers of galaxies in each bin (equal number binning). Examples of the three binning methods, along with the computed FOM values, are shown for the CosmoDC2 sample in Figure 3. Note that for binning choices related to distance, like equal Δz and equal Δχ, this results in a couple of sparsely populated bins at the high-redshift tail of the sample. In these cases, there are ∼10 effective bins up to z ∼ 1.2 that contain most of the information. This is not the case for equal number binning.

Zoom In Zoom Out Reset image size

Figure 3 illustrates that the equal Δχ binning case produces the highest FOM value, while equal number binning performs the worst out of our three base binning choices. Equal Δχ binning has not been explored much in the literature, but the Tomographic Challenge method that achieved the highest FOM also utilized equal Δχ bins (Zuntz et al. 2021). Although they did not include equal Δχ bins in their analysis, Pocino et al. (2021) also found that equally spaced redshift bins performed better than equal number bins for galaxy clustering and galaxy–galaxy lensing when using 13 bins, and the two performed about evenly with 10 bins, which is consistent with our findings. This is in contrast with Sipp et al. (2021), which found that equal number binning outperformed equal Δz binning for a Euclid-like survey, although we find agreement with these results when the S/N is used as the evaluation metric. Taylor et al. (2018) and Kitching et al. (2019) found that bins equally spaced in redshift performed better than bins with equal numbers of galaxies for cosmic shear analysis, although neither were for the same number of bins used here. Taylor et al. (2018) concluded that while having a large number of bins equally spaced in redshift provides the best constraints on cosmological parameters, bins with equal numbers of galaxies were better for N_bin < 20. Meanwhile, Kitching et al. (2019) only tested on three, four, and five tomographic bins.

In addition to the three common binning choices above, there are many more ways that the bin edges could be chosen, so we introduce a new parameter, ${ \mathcal M }$, that allows us to investigate an extended family of possible bin definitions. We define ${ \mathcal M }$ as

$$\begin{eqnarray}&&{ \mathcal M }={\int }_{0}^{{z}_{\max }}{\left(\displaystyle \frac{{dN}}{{dz}}\right)}^{\alpha }{\left(\displaystyle \frac{d\chi }{{dz}}\right)}^{\beta }{dz},\end{eqnarray} \tag{ 3 }$$

where α and β will be chosen to maximize the final S/N of the resulting bins. Once ${ \mathcal M }$ is calculated for a given α and β, ${ \mathcal M }$ is divided into equal intervals based on the number of bins desired, in this case 12, and then those values of ${ \mathcal M }$ are converted back into redshift values for the bin edges by interpolation.

There are three sets of α and β values that can recover the previous binning choices. When α = β = 0, ${ \mathcal M }={z}_{\max }$, and we recover equal Δz binning. When α = 0 and β = 1, ${ \mathcal M }={\chi }_{\max }$, and we recover equal Δχ binning. When α = 1 and β = 0, ${ \mathcal M }=N$, the total number of galaxies in the sample, and we recover equal number binning. Other values for α and β represent variations on these approaches.

Since LSST is expected to observe enough galaxies that it will not be shot noise limited, it should be possible to improve the FOM of the bins shown in Figure 3, or determined by Equation (3), by removing galaxies with poor photo-z estimates. After training and applying TPZ, we implement the NNC as described in Broussard & Gawiser (2021), which we refer to here as the "Outlier" NNC.

The NNC takes as its inputs the three outputs from TPZ, z_phot, σ_TPZ, and z_conf, along with the size of the galaxies, the i-band magnitude, and the same colors that were used for training TPZ. The NNC consists of four fully connected hidden layers, with [100, 200, 100, 50] neurons using a Scaled Exponential Linear Unit (SELU) activation function. The output neuron produces a value between 0 and 1, indicating redshift accuracy confidence, by using a sigmoid function with a binary cross-entropy loss function. An NNC confidence output close to 1 indicates high confidence in the accuracy of the photo-z estimate, while a confidence output near 0 indicates a probable outlier. The output NNC confidence values are then used to make a sample cut.

To define what counts as an accurate photo-z estimate when training the NNC, we must choose a maximum acceptable value for the "accuracy parameter" $\left|{\rm{\Delta }}z\right|/\left(1+z\right)$. Broussard & Gawiser (2021) tested accuracy parameters between 0.04 and 0.15. They found that while the value of the accuracy parameter has an impact on the fit, if the goal is to cut a certain fraction of the sample, e.g., the worst 10%, the value of the accuracy parameter does not change which galaxies are in that worst 10%, only what the threshold confidence value is. In other words, we are free to choose the accuracy parameter to obtain reasonable-looking confidence values, since the accuracy parameter changes the confidence but not the ordering. We choose to use $\left|{\rm{\Delta }}z\right|/\left(1+z\right)\lt 0.07$.

Figure 4 shows the results of implementing a confidence cut based on the NNC confidence levels for the CosmoDC2 sample. The confidence cut of 0.782 was chosen to retain 75% of the original sample, as this will be shown later to produce the highest FOM with this NNC. Since TPZ was trained on a nonrepresentative training sample with a maximum redshift lower than the maximum redshift of the application sample, the high-redshift portion of the application consists entirely of outlier photo-z estimates. The NNC successfully excludes a large portion of these major photo-z outliers, reducing the outlier fractions (defined as Δz/(1 + z) > 0.15) by about a third from f_out = 0.30 for the full sample to f_out = 0.20 for the NNC-selected sample. The Outlier NNC selects galaxies close to the one-to-one line representing accurate photo-z estimates when trained on a representative sample.

Zoom In Zoom Out Reset image size

We can also see in Figure 4, and later in Figure 5, that TPZ produces concentrations near z_phot ∼ 0.1 and z_phot ∼ 0.5. We would expect bands arising from degeneracies in galaxy colors to form diagonal bands, not horizontal ones, so this is likely not an effect of degeneracies in galaxy SEDs at different redshifts. It appears that TPZ has found some preferred redshifts. Investigation of the cause and mitigation of this effect is left for future work.

Zoom In Zoom Out Reset image size

One of the benefits of tomographic binning is that once a galaxy has been sorted into a bin, its individual redshift value is no longer important; angular correlation functions are computed for the population of galaxies within the bin. In this sense, it does not matter whether the photo-z estimate is highly accurate as long as it is good enough for most galaxies to be sorted into the correct tomographic bin. Following this logic, we implement a new method for training a "Misclassification" NNC. Instead of training it to assign low-confidence values to galaxies likely to be outliers, we trained this NNC to assign low-confidence values to galaxies likely to be sorted into the incorrect bin.

The Misclassification NNC has a more complicated training routine than the outlier NNC. With the Outlier NNC, we are free to choose the bin edges after applying the NNC and making the confidence cuts. However, since the definition of the correct bin will depend on the choice of bin edges, we must now choose the bin edges before training the NNC. Once photo-z estimates are determined for the application sample, we use the entire sample to define the bin edges, using equal Δz, equal Δχ, equal number binning, or the optimized binning determined by Equation (3). Then, objects in the NNC training sample are sorted into those bins by their estimated photo-z, and we determine whether the galaxy was sorted into the correct bin by comparing to the bin its true redshift would have placed it in. Galaxies in the correct bin are assigned an accuracy value of 1, while galaxies placed in any incorrect bin are assigned an accuracy value of 0.

We also train a separate Misclassification NNC for each bin, which is then applied only to objects sorted into that bin. As an example, a "bin 3 NNC" is trained on objects in the training sample that have been sorted into bin 3 and is applied only to objects in the application sample that are sorted into bin 3. Instead of identifying objects across the whole sample that are likely to be misclassified, this version of the Misclassification NNC identifies galaxies that are likely to be incorrectly sorted into each bin. This is an attempt to make it easier for the NNC to learn the photometric features associated with each bin, since the photometries associated with being sorted into the wrong bin are not necessarily consistent over all bins. This version of the Misclassification NNC performs better than the version trained on the entire sample and is the version we use going forward.

Figure 5 shows the sample selection using the Misclassification NNC. The NNC was trained on 12 equal Δχ bins, since this was the best performing of the base binning choices. The confidence cut was selected to retain 75% of the original sample, as in Figure 4. The sample selected by the Misclassification NNC looks quite different from the sample selected by the Outlier NNC. Boxy horizontal features are caused by the NNC only caring if the photo-z places the galaxy into the correct bin and correspond to the bin edges. In particular, in the equal Δχ binning case, the bins get progressively wider as the redshift increases; the highest redshift bin is quite wide, since there are not as many galaxies at those redshifts; and a modestly larger fraction of the outliers in the upper right of Figure 5 are kept in the sample compared to the Outlier NNC selection; this is consistent with the Misclassification NNC only caring about objects that shuffle between bins, as the highest redshift bin is broad.

We calculate the FOM resulting from the bins produced by various combinations of α and β in Equation (3). In order to do these calculations, we must make a choice for the redshift distribution, n(z) (equivalent to $\tfrac{{dN}}{{dz}}$ in Equation (3)). We have done these calculations for three versions of n(z): the photo-z n(z), made up of the point estimates of the photo-z for each galaxy, a version of the photo-z n(z) that has been smoothed with a Gaussian filter, or the true n(z). In all three cases, we find that slightly different combinations of α and β produce the optimal binning choice, but they all find that equal number binning is not the optimal choice. We show results for the true n(z) going forward.

Figure 6 shows the FOM values for a grid of different values of α and β for CosmoDC2. The maximum FOM of 124 is achieved at α = 0.25 and β = 2.0. We take this to be our optimized binning choice. The bin edges resulting from this choice of α and β are listed in Table 1. Buzzard reaches the maximum FOM at α and β corresponding to equal Δχ binning; see Appendix A.

Zoom In Zoom Out Reset image size

Since LSST will not be shot noise limited, removing some of the galaxies with poor photo-z estimates should improve the FOM. However, if too much of the sample is removed, we will reach the limit where shot noise increases sufficiently to drop the FOM below what it would be if we kept the entire sample.

We test the range over which the NNC sample cuts will improve the FOM and find the optimal fraction of the sample to retain (the retained fraction). Figure 7 compares the performance of the FOM as a function of the retained fraction for each type of NNC and binning choice for CosmoDC2. The results for Buzzard are shown in Appendix A. It can be seen that the Outlier NNC performs better at slightly higher retained fractions, while the Misclassification NNC obtains a higher FOM at lower retained fractions. Neither NNC improves the FOM for the Buzzard sample, but there are a range of retained fractions for which the Misclassification NNC does not hurt the FOM either. See Appendix A for further discussion. Figure 8 shows the sample selection by the Misclassification NNC with the optimized bin edges and retained fraction. The retained fractions achieving the highest FOM in each combination of bin edges and NNC sample selection are listed for CosmoDC2 in Table 1.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Both CosmoDC2 and Buzzard obtain a higher FOM with the Misclassification NNC than with the Outlier NNC, suggesting that the Misclassification NNC is preferred. However, as stated in Appendix A, the highest FOM obtained with the Misclassification NNC is the same as the full sample FOM for Buzzard. Interestingly, although there is not a large improvement in the FOM when the full CosmoDC2 sample of galaxies is sorted into equal Δχ bins versus the optimized bins, the Misclassification NNC greatly increases this improvement at the optimal retained fraction.

The Outlier NNC–selected sample shown in Figure 4, which retains 75% of the original sample, is sorted into equal Δχ bins, which is the best performing of the original three binning choices. The same Outlier NNC–selected sample is sorted into the optimized binning choice defined by α = 0.25 and β = 2.0. The Misclassification NNC–selected samples for equal Δχ binning (shown in Figure 5 with a retained fraction of 75%) and for the optimized binning choice, shown in Figure 8, each with a retained fraction of 45%, are sorted into equal Δχ bins and the optimized bins, respectively.

The top row of Figure 9 shows the binning results for equal Δχ binning, while the bottom row shows the results for the optimized binning choice. In the left column, we have binned the full sample, while the middle and right columns show the Outlier NNC and Misclassification NNC selections. Switching from equal Δχ binning to the optimized binning increases the FOM by ∼1%. The sample selection with the Outlier NNC improves the FOM by a further 3.3%, while the Misclassification NNC improves the FOM by 13.2% over using the full sample. The NNC sample selection process boosts the FOMs in the equal Δχ binning case by a similar amount: 3.0% for the Outlier NNC, and 8.5% for the Misclassification NNC. The overall improvement between the Misclassification NNC–selected sample sorted into the optimized bins and the full sample sorted into the equal Δχ bins is 14.2%. The Misclassification NNC slightly outperforms the Outlier NNC, and in Figure 9 there is visibly less overlap between the bins when the Misclassification NNC is used, particularly when compared to the full sample bins. A summary of the different binning methods and sample selections, along with the achieved FOMs, is included in Table 1.

Zoom In Zoom Out Reset image size

To compare how well the NNC sample selection can recover the ideal case of a representative training sample, we also test the binning method when TPZ and the NNCs have been trained with a fully representative training sample. When the galaxies are sorted into the optimized bin edges, we achieve an FOM of 186, an improvement of ∼50% over the nonrepresentative training sample. When the Misclassification NNC is applied in the nonrepresentative case, it is able to recover ∼25% of this difference. This indicates that the Misclassification NNC will be useful in attempting to re-create the ideal scenario of a fully representative spectroscopic training sample for LSST.

Given that the Tomographic Challenge already proposed methods for optimizing the tomographic binning strategy for LSST, albeit for the source sample of galaxies instead of the lens sample of galaxies as we have done here, we also compare how the Tomographic Challenge methods compare to the method described in this work. These results are described in Appendix B.

4.3.1. Difference between Binning Choices

The bin edges for both the optimized bins and equal Δχ bins are listed in Table 1. In the optimized case, the highest redshift bin is ∼40% wider than in the equal Δχ case, while most of the rest of the bins are slightly narrower than in the equal Δχ case. This could be a result of the highest redshift bin covering a range that is not well represented in our training sample, so the photo-z estimates are poorer. We gain constraining power by containing a larger fraction of those galaxies with poor photo-z's in one bin, allowing us to have narrower bins in regions with better photo-z's.

As can be seen in Figures 4, 5, and 8, galaxies at high true redshifts are assigned primarily lower photo-z values. In particular, there are bands at z ∼ 0.1 and z ∼ 0.5 where many higher-redshift galaxies are assigned. The lower band corresponds to bins 0 and 1, while the z ∼ 0.5 band falls around bin 5. In Figure 10, we show a closer look at this subset of the bins to illustrate some differences between the binning choices and sample selections.

Zoom In Zoom Out Reset image size

In the top rows of the upper left and right quadrants of Figure 10, the two lowest redshift bins have significant numbers of contaminating galaxies, where the true redshift places them well outside the bounds of the bin. In the middle and bottom rows of these quadrants, the Outlier and Misclassification NNC are able to remove many of these contaminating galaxies, with the Outlier NNC working better in the lowest redshift bin 0, and the Misclassification NNC doing a better job in bin 1. Additionally, the Misclassification NNC does a slightly better job when applied to the optimized bin 1 than the equal Δχ bin 1.

In the lower left quadrant of Figure 10, we show bin 5, which contains many of the galaxies assigned to the z ∼ 0.5 band. Similar to bin 1, the Misclassification NNC is better able to clean up the contaminating galaxies than the Outlier NNC, although in this case the Misclassification NNC performs about equally for both choices of bin edges.

In general, the Misclassification NNC removes contaminating galaxies better than the Outlier NNC. The two exceptions are the lowest and highest redshift bins, shown in the upper left and lower right quadrants of Figure 10, respectively. In the case of the highest redshift bin, this is likely because not many galaxies are truly misclassified into the highest redshift bin. In general, photo-z estimates are lower than the true redshift, not higher (see, e.g., Figure 8), so there are not a significant number of galaxies misclassified into the highest redshift bin from below. On the other hand, galaxies are not being misclassified from above either, since galaxies assigned into bin 11 that have true redshifts larger than their photo-z would have been assigned to the highest redshift bin anyway. It is not as clear why the lowest redshift bin should also do better with the Outlier NNC as opposed to the Misclassification NNC.

In the remaining bins, the Misclassification NNC performs better at cleaning up contaminating galaxies than the Outlier NNC. In 4 of the 10 remaining bins, the Misclassification NNC cleans up more contaminating galaxies in the optimized binning choice than in the equal Δχ choice. In another 4 bins it cleans up contaminating galaxies equally well in both bin choices, while in the remaining 2 it cleans up contaminating galaxies better in the equal Δχ case. Given that the Misclassification NNC performs better on the optimized case in a larger percentage of bins, this explains why the optimized bins are able to achieve more constraining power than the equal Δχ bins.

The visual indication that the Misclassification NNC is better able to clean up the contaminating galaxies is quantified by the contamination fraction. Bin edges are defined in photo-z space; we define the contamination fraction as the fraction of galaxies with z_true outside the photo-z range of the bin. When the full sample of galaxies is sorted into equal Δχ bins, the contamination fraction is 0.54; over half the galaxies have z_true outside the bin they are sorted into. This is not surprising; the maximum photo-z estimate is z ∼ 1.2, while the true redshifts extend to z ∼ 3. With the Outlier NNC–selected sample, the contamination fraction is reduced to 0.44. It is further reduced to 0.17 with the Misclassification NNC selection. In contrast, the contamination fraction for the full sample sorted into the optimized bins is 0.53. This is reduced to 0.43 with the Outlier NNC and to 0.17 with the Misclassification NNC. In both choices of bin edges, the contamination fractions are much lower when the Misclassification NNC is used to select the sample than when the Outlier NNC or no NNC is used. This can be seen visually in the upper right and lower left quadrants of Figure 10.

4.3.2. Impact of NNCs on n(z)

4.3.3. Combining NNCs