miranda: Python utilities for climate data collection, conversion, and management

Contributors to this version: Travis Logan (@tlogan2000), Trevor James Smith (@Zeitsperre), Aslı Beşe (@aslibese).

Announcements

• miranda boilerplate code is now versioned with cruft and the Ouranosinc/cookiecutter-pypackage template.
• The miranda library now requires Python 3.10 or higher.
• The miranda library has undergone a significant refactoring to remove unused code and modules that were out of scope for the library.
  • The modules miranda.archive and miranda.remote have been removed and will be moved to a separate project.
  • Data treatment functions have been moved to miranda.treatments.
  • Pre-processing logic and functions have been moved to miranda.preprocess.
  • Several obsolete/nonexistent configuration files have been removed from miranda.convert.corrections.
• The miranda.validators submodule has become miranda.validate and now contains all controlled vocabulary and validation functions.
• miranda now leverages pooch to fetch and cache testing datasets from miranda-testdata:
  • miranda.testing.cassini is used to create an instance of Cassini for fetching testing data.
  • miranda.testing.registry.txt is a text file containing the list of datasets available in miranda-testdata.
• miranda has dropped black, isort and blackdocs, and now relies solely on ruff and flake8/flake8-rst-docstrings for linting and formatting.

New features

• Conversion for CanHomTv4 daily ECCC 4th generation of the adjusted and homogenized station.
• Conversions for variables in the GHCN-D weather station network dataset are now supported.
• Conversion support has been added for the ORRC, CaSR v3.1, and RDRS v2.1 datasets.
• ECMWF: Added support for the "era5-single-levels-monthly-means" project.
• ECMWF: Added support for ocean variables (sst, siconc), convective precipitation variables ('cp', 'cape'), and wind speed ('u', 'v').
• Aggregation operations now support more variables ('hur', 'hurs', 'huss', 'rlds', 'ta', 'tdp', 'ua', 'uas', 'va', 'vas')
• Minimum values of "0 kg m2 s-1" has been set for both 'tp' and 'sf' variables in ERA5 and ERA5-Land projects.
• Project user and developer documentation has been greatly expanded. All public functions and modules now have numpy-based docstrings.
• The miranda library now uses a src layout for better packaging and distribution.
• ruff checks and formatting standards have been adopted for the entire codebase.
• Added a new configuration for converting the NRCAN gridded climate dataset (NRCANmet).
• Conversion configuration JSON files are now validated against schema schemas.

Breaking changes

• Removed modules miranda.archive and miranda.remote (split into a separate project yet to be published).
• miranda.utils.show_versions has been moved to miranda.testing.show_versions. It now uses xclim.testing.show_versions to display the versions of all dependencies.
• Python 3.8 and Python 3.9 are no longer supported.
• The dev recipe now requires pooch (>=1.8.0).
• many dependencies have been updated to more modern versions, including:
  • numpy (>=1.25.0)
  • xarray (>=2023.11.0)
  • xclim (>=0.57.0)
• Logging has been significantly improved and standardized across the library.
  • Logging within modules has been standardized to use the miranda logger and never root.
  • Submodules no longer configure message logging to standard output and instead use the miranda logger.

Bug fixes

• Transformation docstrings are now only updated when the transformation is actually applied.
• Added a missing helper function to miranda.units (group_by_length) that was mistakenly removed in a previous change.

Internal changes

• miranda now has a security policy (SECURITY.md) for disclosing sensitive issues using secure communication channels. This has also been added to the documentation.
• miranda now applies the numpydoc documentation style to all publicly-exposed docstrings.
• GitHub Workflows now use commit hashes for both running GitHub Actions and installing Python dependencies from PyPI.
• miranda now has a CODE_OF_CONDUCT.md file for setting community standards and expectations.
• Now using the GitHub Ouranos bot for automatic version bumping via bump-version.yml GitHub Workflow.
• Adjusted calls using os.path to use pathlib for better cross-platform compatibility.
• Added new pytest fixtures for the new miranda-testdata repository:
  • cassini: pytest fixture for fetching local filepaths of cached testing data.
  • open_dataset: pytest fixture for one-off fetching and opening of a registered test data set.
  • era5_precip: pytest fixture fetching and opening a zip file containing a subset of the ERA5 precipitation dataset.
  • timeseries: pytest fixture for generating an artificial CF-compliant time series dataset using xclim and xarray.
  • multivariable_dataset: pytest fixture for generating an artificial xarray multivariable dataset.
• The tox.ini and pyproject.toml dependency pins have been synchronized.
• schema schemas have been defined for all conversion JSON files, and are now used to validate the JSON files as part of the testing suite.
• The code formatting now follows ruff standards, and black, isort, and blackdocs have been removed from the project. The pre-commit configuration has been updated accordingly. Line lengths have been increased from 88 to 150.
• pre-commit hook versions have been updated and new hooks have been added for checking variable spelling and security issues. Hooks for mypy and vulture have been staged for eventual inclusion in the CI testing suite.
• Allow some variables that are lacking a standard_name attribute to be converted if _standard_name is explicitly set as False.