Preprocessed labelled CRISM data for training and validating interpretable classifiers.

The three files in this repository are pre-processed and labelled CRISM spectra. They are a subset of the CRISM Machine Learning Toolkit's Mineral Dataset compiled by Plebani et al. (2022). 

The Spectra were sampled from their respective Targeted Reduced Data Record (TRDR) infrared (IR) CRISM data records and then preprocessed to suppress noise and highlight revalent absorption features using the workflow developed by Plebani et al. (2022) and Convex Hull Removal, respectively. Finally, each spectra was normalized (i.e. converted to unit-vectors).

Classwise outliers in the dataset were identified and removed using the Local Outlier Factor (LOF) algorithm. The remaining spectra and their labels (i.e., dominant mineral species) and ancillary information were compiled into the first file - ReducedPreProcessedCRISMDataset.pkl. This ID included the ID (in the ML Toolkit) of the Image from which they were collected, the ID of the mineral outcrop (in the ML toolkit), as well as the Row and Column number of the pixel.

The spectra from each class - except High-Calcium Pyroxene and Hydroxylated Fe-sulfate - were divided into 1000 clusters each, using K-mediods clustering. The mediod of each cluster was sampled and compiled into a well-balanced training set (unified22MineralsinlierTrainingDatatset.pkl). This set also included all the spectra from High-Calcium Pyroxene and Hydroxylated Fe-sulfate.

The remaining spectra, i.e. non-cluster centres, were compiled into a separate, testing dataset - 22MineralsinlierValidationDatatset.pkl.