Presentation Open Access

flox: Fast & furious GroupBy reductions with Dask at Pangeo-scale

Cherian, Deepak


JSON Export

{
  "files": [
    {
      "links": {
        "self": "https://zenodo.org/api/files/de42b97d-e735-4bd4-bfdf-23f8df6fbc3f/Cherian_2021-11-17.mp4"
      }, 
      "checksum": "md5:d5435a8e75714dad50c5114ca3d90f57", 
      "bucket": "de42b97d-e735-4bd4-bfdf-23f8df6fbc3f", 
      "key": "Cherian_2021-11-17.mp4", 
      "type": "mp4", 
      "size": 58131046
    }, 
    {
      "links": {
        "self": "https://zenodo.org/api/files/de42b97d-e735-4bd4-bfdf-23f8df6fbc3f/Cherian_2021-11-17.pdf"
      }, 
      "checksum": "md5:8c6514269b2982834fb93f4d534b1df0", 
      "bucket": "de42b97d-e735-4bd4-bfdf-23f8df6fbc3f", 
      "key": "Cherian_2021-11-17.pdf", 
      "type": "pdf", 
      "size": 7699842
    }
  ], 
  "owners": [
    221183
  ], 
  "doi": "10.5281/zenodo.5772165", 
  "stats": {
    "version_unique_downloads": 23.0, 
    "unique_views": 70.0, 
    "views": 77.0, 
    "version_views": 77.0, 
    "unique_downloads": 23.0, 
    "version_unique_views": 70.0, 
    "volume": 293358458.0, 
    "version_downloads": 25.0, 
    "downloads": 25.0, 
    "version_volume": 293358458.0
  }, 
  "links": {
    "doi": "https://doi.org/10.5281/zenodo.5772165", 
    "conceptdoi": "https://doi.org/10.5281/zenodo.5772164", 
    "bucket": "https://zenodo.org/api/files/de42b97d-e735-4bd4-bfdf-23f8df6fbc3f", 
    "conceptbadge": "https://zenodo.org/badge/doi/10.5281/zenodo.5772164.svg", 
    "html": "https://zenodo.org/record/5772165", 
    "latest_html": "https://zenodo.org/record/5772165", 
    "badge": "https://zenodo.org/badge/doi/10.5281/zenodo.5772165.svg", 
    "latest": "https://zenodo.org/api/records/5772165"
  }, 
  "conceptdoi": "10.5281/zenodo.5772164", 
  "created": "2021-12-10T11:17:42.359183+00:00", 
  "updated": "2021-12-10T13:56:14.863937+00:00", 
  "conceptrecid": "5772164", 
  "revision": 3, 
  "id": 5772165, 
  "metadata": {
    "access_right_category": "success", 
    "doi": "10.5281/zenodo.5772165", 
    "description": "<p>The &quot;groupby&quot; or the &quot;split-apply-combine&quot; paradigm is ubiquitous in scientific analysis, though it may be named differently e.g. &quot;binning&quot;, &quot;histogramming&quot;, &quot;resampling&quot;, &quot;compositing&quot;, or &quot;climatology reductions&quot;. Xarray implements the groupby paradigm through a &quot;GroupBy&quot; object. Historically the underlying algorithm is not dask-aware, and tends to fail disastrously with large Pangeo-scale distributed workflows.&nbsp;Here I present &quot;flox&quot;: a new package that explores effective strategies for groupby reductions at scale with dask. Ongoing work will plug this package in to xarray in a backwards-compatible manner, allowing the community to seamlessly benefit from significantly more efficient groupby computations.See&nbsp;https://flox.readthedocs.io&nbsp;for more.</p>", 
    "license": {
      "id": "CC-BY-4.0"
    }, 
    "title": "flox: Fast & furious GroupBy reductions with Dask at Pangeo-scale", 
    "relations": {
      "version": [
        {
          "count": 1, 
          "index": 0, 
          "parent": {
            "pid_type": "recid", 
            "pid_value": "5772164"
          }, 
          "is_last": true, 
          "last_child": {
            "pid_type": "recid", 
            "pid_value": "5772165"
          }
        }
      ]
    }, 
    "communities": [
      {
        "id": "pangeo"
      }
    ], 
    "keywords": [
      "Pangeo", 
      "Xarray"
    ], 
    "publication_date": "2021-11-17", 
    "creators": [
      {
        "orcid": "0000-0002-6861-8734", 
        "affiliation": "NCAR", 
        "name": "Cherian, Deepak"
      }
    ], 
    "access_right": "open", 
    "resource_type": {
      "type": "presentation", 
      "title": "Presentation"
    }, 
    "related_identifiers": [
      {
        "scheme": "doi", 
        "identifier": "10.5281/zenodo.5772164", 
        "relation": "isVersionOf"
      }
    ]
  }
}
77
25
views
downloads
All versions This version
Views 7777
Downloads 2525
Data volume 293.4 MB293.4 MB
Unique views 7070
Unique downloads 2323

Share

Cite as