Data for GalaxiesML-Spectra

Morgan, Himes; Tuan, Do; Samiksha, Krishnamurthy

doi:10.5281/zenodo.16989593

Published August 28, 2025 | Version 2.0

Dataset Open

Data for GalaxiesML-Spectra

# GalaxiesML-Spectra Dataset

GalaxiesML-Spectra is a crossmatched dataset between DESI and HSC data containing images, spectra, and photometry for use in training a model to predict photometric redshifts. Within the directory there are 4 files used for model training.

DESI_HSC_64x64.hdf5

DESI_HSC_64x64_v2.hdf5

DESI_HSC_127x127.hdf5

DESI_HSC_127x127_v2.hdf5

The spectra-image hdf5 files contain 64 x 64 and 127 x 127 images, depending on the size required by the model it is training with. There is a v2 section of the data processing file that replaces artifacts and removes sources with significantly missing data. The original files and their changes are reflected below in the Data Format section.

## Dataset Summary

- Total objects: 134,533 galaxy spectra-image pairs

- Redshift range: z = 0 - 4 (90th pct: 1.581)

- File sizes: ~50 GB per HDF5 file

- Spectral coverage: 3600-9800 Å

- Image bands: grizy (HSC filters)

## Data Cuts

The first step is to identify the matches between HSC/DESI and build the catalog table. The following steps and the source count remaining after each step are shown below:

1. Load in entire DESI catalog (23,060,727)

2. Filter to zwarn = 0 (21,159,181)

3. Remove sources that have the same targetid but inconsistent redshifts (using combined error) (20,863,764)

4. Remove duplicates by taking the smallest redshift error (20,274,090)

5. Match to GalaxiesML, taking all matches within 0.5 arcsec (136,939)

6. Select best match in GalaxiesML by taking the best redshift agreement within 0.5 arcsec (135,082)

7. Remove sources DESI classifies as stars (135,005)

With the catalog in hand, we then download the spectra from DESI. The raw coadded HEALPix files are downloaded; then, the individual spectra corresponding to objects in our catalog are extracted and saved into individual HDF5 files. Once the spectra were downloaded, they were camera coadded. This process combines spectral measurements from DESI's blue, red, and near-infrared cameras in overlapping wavelength regions, resulting in a single, consistent spectrum for each observation.

## Data Format

### v1

The data used for training is stored in two HDF5 files corresponding to the image sizes (64x64 and 127x127). The HDF5 file structure is organized into datasets and groups:

DESI_HSC_{64x64 or 127x127}.hdf5/

├── DESI_fibermap/ (group)

│ └── BGS_TARGET, BRICKID, BRICKNAME, BRICKOBJID, COADD_EXPTIME, COADD_FIBERSTATUS, COADD_NUMEXP, COADD_NUMNIGHT, COADD_NUMTITLE,

DESI_TARGET, DESI_redshift, DESI_redshift_err, EBV, FA_TARGET, FA_TYPE, FIBERFLUX_G, FIBERFLUX_R, FIBERFLUX_Z, FIBERTOFLUX_G,

FIBERTOFLUX_R, FIBERTOFLUX_Z, FLUX_G, FLUX_IVAR_G, FLUX_IVAR_R, FLUX_IVAR_W1, FLUX_IVAR_W2, FLUX_IVAR_Z, FLUX_R, FLUX_W1,

FLUX_W2, FLUX_Z, GAIA_PHOT_BP_MEAN_MAG, GAIA_PHOT_G_MEAN_MAG, GAIA_PHOT_RP_MEAN_MAG, MASKBITS, MEAN_DELTA_X, MEAN_DELTA_Y,

MEAN_FIBER_DEC, MEAN_FIBER_RA, MEAN_PSF_TO_FIBER_SPECFLUX, MORPHTYPE, MWS_TARGET, NUMOBS_INIT, OBJTYPE, OBSCONDITIONS,

PARALLAX, PHOTSYS, PLATE_DEC, PLATE_RA, PMDEC, PMRA, PRIORITY_INIT, REF_CAT, REF_EPOCH, REF_ID, RELEASE, RMS_DELTA_X,

RMS_DELTA_Y, SCND_TARGET, SERSIC, SHAPE_E1, SHAPE_E2, SHAPE_R, STD_FIBER_DEC, STD_FIBER_RA, SUBPRIORITY, SV1_BGS_TARGET,

SV1_DESI_TARGET, SV1_MWS_TARGET, SV1_SCND_TARGET, SV3_BGS_TARGET, SV3_DESI_TARGET, SV3_MWS_TARGET, SV3_SCND_TARGET,

TARGETID, TARGET_DEC, TARGET_RA (datasets)

├── HSC_metadata/ (group)

│ └── coord, dec, g_cmodel_mag, g_cmodel_magsigma, i_cmodel_mag, i_cmodel_magsigma, r_cmodel_mag, r_cmodel_magsigma, ra, skymap_id,

specz_dec, specz_flag_homogeneous, specz_mag_i, specz_name, specz_ra, specz_redshift, specz_redshift_err, y_cmodel_mag,

y_cmodel_magsigma, z_cmodel_mag, z_cmodel_magsigma (datasets)

├── image/ (dataset)

├── object_id/ (dataset)

├── spectrum/ (group)

│ ├── flux (dataset)

│ ├── ivar (dataset)

│ └── wave (dataset)

├── targetid/ (dataset) </code></pre>

Here, the DESI_fibermap group stores all of the information from the DESI FIBERMAP HDU (see https://desidatamodel.readthedocs.io/en/latest/DESI_SPECTRO_REDUX/SPECPROD/healpix/SURVEY/PROGRAM/PIXGROUP/PIXNUM/spectra-SURVEY-PROGRAM-PIXNUM.html). Two additional datasets were added to the DESI fibermap (DESI_redshift, DESI_redshift_err). The HSC_metadata group stores the information from the GalaxiesML dataset. The image dataset stores the grizy images from HSC. The object_id dataset is the HSC object ID. The spectrum group stores the flux, wavelength, and inverse variance arrays of the camera-coadded spectra from DESI. The image dataset and flux array within the spectrum group are used for training a model with the images and spectra. Finally, the targetid dataset stores the DESI target ID.

### v2

Some changes were made to the HDF5 files in v2. Artifacts (large spikes in flux) were removed and replaced with the spectra median. Spectra with more than 80% missing data were removed and stored in separate files (DESI_HSC_64x64_discarded_v2.hdf5, DESI_HSC_127x127_discarded_v2.hdf5). The remaining spectra are stored in DESI_HSC_64x64_v2.hdf5 and DESI_HSC_127x127_v2.hdf5. The file structure is largely the same as v1, but one dataset was added to DESI_fibermap: num_missing_pixels (the number of missing pixels in each spectrum).