Presentation Open Access
The "groupby" or the "split-apply-combine" paradigm is ubiquitous in scientific analysis, though it may be named differently e.g. "binning", "histogramming", "resampling", "compositing", or "climatology reductions". Xarray implements the groupby paradigm through a "GroupBy" object. Historically the underlying algorithm is not dask-aware, and tends to fail disastrously with large Pangeo-scale distributed workflows. Here I present "flox": a new package that explores effective strategies for groupby reductions at scale with dask. Ongoing work will plug this package in to xarray in a backwards-compatible manner, allowing the community to seamlessly benefit from significantly more efficient groupby computations.See https://flox.readthedocs.io for more.
|All versions||This version|
|Data volume||293.4 MB||293.4 MB|