Table of Contents
Shortcuts

Evaluation

In addition to examining cross-validation training performance, model performance can be assessed with external dataset evaluation, and visualization of predictions across evaluation slides in the form of a heatmap.

Model evaluation

Once training and hyperparameter tuning is complete, you can test model performance on your held-out evaluation set using the evaluate function. Specify the path to the saved with the model argument. For example:

P.evaluate(
  model="/path/to/trained_model_epoch1",
  outcomes="category",
  filters={"dataset": ["eval"]}
)
slideflow.Project.evaluate(self, model, outcomes, dataset=None, filters=None, checkpoint=None, eval_k_fold=None, splits='splits.json', max_tiles=0, min_tiles=0, input_header=None, mixed_precision=True, **kwargs)

Evaluates a saved model on a given set of tfrecords.

Parameters

  • model (str) – Path to model to evaluate.

  • outcomes (str) – Str or list of str. Annotation column header specifying the outcome label(s).

  • dataset (slideflow.dataset.Dataset, optional) – Dataset from which to generate activations. If not supplied, will calculate activations for all project tfrecords at the tile_px/tile_um matching the supplied model, optionally using provided filters and filter_blank.

  • filters (dict, optional) – Filters dict to use when selecting tfrecords. Defaults to None. See get_dataset() documentation for more information on filtering.

  • checkpoint (str, optional) – Path to cp.ckpt file, if evaluating a saved checkpoint. Defaults to None.

  • eval_k_fold (int, optional) – K-fold iteration number to evaluate. Defaults to None. If None, will evaluate all tfrecords irrespective of K-fold.

  • splits (str, optional) – Filename of JSON file in which to log train/val splits. Looks for filename in project root directory. Defaults to “splits.json”.

  • max_tiles (int, optional) – Maximum number of tiles from each slide to evaluate. Defaults to 0. If zero, will include all tiles.

  • min_tiles (int, optional) – Minimum number of tiles a slide must have to be included in evaluation. Defaults to 0.

  • input_header (str, optional) – Annotation column header to use as additional input. Defaults to None.

  • mixed_precision (bool, optional) – Enable mixed precision. Defaults to True.

Keyword Arguments

  • save_predictions (bool or str, optional) – Either True, False, or any combination of ‘tile’, ‘patient’, or ‘slide’, as string or list of strings. Save tile-level, patient-level, and/or slide-level predictions. If True, will save all.

  • histogram (bool, optional) – Create tile-level histograms for each class. Defaults to False.

  • permutation_importance (bool, optional) – Calculate the permutation feature importance. Determine relative importance when using multiple model inputs. Only available for Tensorflow backend. Defaults to False.

Returns

Dictionary of keras training results, nested by epoch.

Return type

Dict

Heatmaps

To generate a predictive heatmap for a set of slides, use the generate_heatmaps() function as below, which will automatically save heatmap images in your project directory:

P.generate_heatmaps(
  model="/path/to/trained_model_epoch1",
  filters={"dataset": ["eval"]}
)
slideflow.Project.generate_heatmaps(self, model, filters=None, filter_blank=None, outdir=None, resolution='low', batch_size=32, roi_method='inside', buffer=None, num_threads=None, skip_completed=False, **kwargs)

Creates predictive heatmap overlays on a set of slides.

Parameters

  • model (str) – Path to Tensorflow model.

  • filters (dict, optional) – Filters to use when selecting tfrecords. Defaults to None.

  • filter_blank (list, optional) – Exclude slides blank in these cols. Defaults to None.

  • outdir (path, optional) – Directory in which to save heatmap images.

  • resolution (str, optional) – Heatmap resolution. Defaults to ‘low’. “low” uses a stride equal to tile width. “medium” uses a stride equal 1/2 tile width. “high” uses a stride equal to 1/4 tile width.

  • batch_size (int, optional) – Batch size during heatmap calculation. Defaults to 64.

  • roi_method (str, optional) – ‘inside’, ‘outside’, or ‘none’. Determines where heatmap should be made with respect to ROI. Defaults to ‘inside’.

  • buffer (str, optional) – Path to which slides are copied prior to heatmap generation. Defaults to None.

  • num_threads (int, optional) – Number of threads for tile extraction. Defaults to CPU core count.

  • skip_completed (bool, optional) – Skip heatmaps for slides that already have heatmaps in target directory.

Keyword Arguments

  • show_roi (bool) – Show ROI on heatmaps.

  • interpolation (str) – Interpolation strategy for predictions. Defaults to None. Includes all matplotlib imshow interpolation options.

  • logit_cmap – Function or a dict used to create heatmap colormap. If None (default), separate heatmaps are generated for each category, with color representing category prediction. Each image tile will generate a list of preds of length O, If logit_cmap is a function, then the logit predictions will be passed, where O is the number of label categories. and the function is expected to return [R, G, B] values. If the logit_cmap is a dictionary, it should map ‘r’, ‘g’, and ‘b’ to label indices; the prediction for these label categories will be mapped to corresponding colors. Thus, the corresponding color will only reflect predictions of up to three labels. Example (this would map predictions for label 0 to red, 3 to green, etc): {‘r’: 0, ‘g’: 3, ‘b’: 1 }

  • vmin (float) – Minimimum value to display on heatmap. Defaults to 0.

  • vcenter (float) – Center value for color display on heatmap. Defaults to 0.5.

  • vmax (float) – Maximum value to display on heatmap. Defaults to 1.

If you would like to directly interact with the calculated heatmap data, create a slideflow.Heatmap object by providing a path to a slide, a path to a model, and tile size information:

from slideflow import Heatmap

heatmap = Heatmap(
  slide='/path/to/slide.svs',
  model='/path/to/model'
)

The spatial map of logits, as calculated across the input slide, can be accessed through heatmap.logits. The spatial map of post-convolution, penultimate activations can be accessed through heatmap.postconv. The heatmap can be saved with heatmap.save('/path/').