Published November 17, 2021 | Version v1
Presentation Open

flox: Fast & furious GroupBy reductions with Dask at Pangeo-scale

  • 1. NCAR

Description

The "groupby" or the "split-apply-combine" paradigm is ubiquitous in scientific analysis, though it may be named differently e.g. "binning", "histogramming", "resampling", "compositing", or "climatology reductions". Xarray implements the groupby paradigm through a "GroupBy" object. Historically the underlying algorithm is not dask-aware, and tends to fail disastrously with large Pangeo-scale distributed workflows. Here I present "flox": a new package that explores effective strategies for groupby reductions at scale with dask. Ongoing work will plug this package in to xarray in a backwards-compatible manner, allowing the community to seamlessly benefit from significantly more efficient groupby computations.See https://flox.readthedocs.io for more.

Files

Cherian_2021-11-17.mp4

Files (65.8 MB)

Name Size Download all
md5:d5435a8e75714dad50c5114ca3d90f57
58.1 MB Preview Download
md5:8c6514269b2982834fb93f4d534b1df0
7.7 MB Preview Download