Software Open Access

Hickle: a HDF5-based python pickle replacement

Danny Price; Sébastien Celles; Pieter T. Eendebak; Michael M. McKerns; Eben M. Olson; Colin Raffel; Bairen Yi


JSON Export

{
  "conceptdoi": "10.5281/zenodo.2345648", 
  "conceptrecid": "2345648", 
  "created": "2018-12-17T06:53:06.612555+00:00", 
  "doi": "10.5281/zenodo.2345649", 
  "files": [
    {
      "bucket": "77fb06d4-9aeb-424f-be62-e3bb1b32c6d1", 
      "checksum": "md5:8883b55d676e64e775c2676c7a5a1ef2", 
      "key": "hickle-3.3.2.tar.gz", 
      "links": {
        "self": "https://zenodo.org/api/files/77fb06d4-9aeb-424f-be62-e3bb1b32c6d1/hickle-3.3.2.tar.gz"
      }, 
      "size": 43468, 
      "type": "gz"
    }
  ], 
  "id": 2345649, 
  "links": {
    "badge": "https://zenodo.org/badge/doi/10.5281/zenodo.2345649.svg", 
    "bucket": "https://zenodo.org/api/files/77fb06d4-9aeb-424f-be62-e3bb1b32c6d1", 
    "conceptbadge": "https://zenodo.org/badge/doi/10.5281/zenodo.2345648.svg", 
    "conceptdoi": "https://doi.org/10.5281/zenodo.2345648", 
    "doi": "https://doi.org/10.5281/zenodo.2345649", 
    "html": "https://zenodo.org/record/2345649", 
    "latest": "https://zenodo.org/api/records/2345649", 
    "latest_html": "https://zenodo.org/record/2345649"
  }, 
  "metadata": {
    "access_right": "open", 
    "access_right_category": "success", 
    "communities": [
      {
        "id": "zenodo"
      }
    ], 
    "creators": [
      {
        "affiliation": "Swinburne University of Technology", 
        "name": "Danny Price"
      }, 
      {
        "name": "S\u00e9bastien Celles", 
        "orcid": "0000-0001-9987-4338"
      }, 
      {
        "name": "Pieter T. Eendebak", 
        "orcid": "0000-0001-7018-1124"
      }, 
      {
        "name": "Michael M. McKerns", 
        "orcid": "0000-0001-8342-3778"
      }, 
      {
        "name": "Eben M. Olson"
      }, 
      {
        "name": "Colin Raffel"
      }, 
      {
        "name": "Bairen Yi"
      }
    ], 
    "description": "<p><code>hickle</code> is a Python 2/3 package for quickly dumping and loading python data structures to Hierarchical Data Format 5 (HDF5) files. When dumping to HDF5, <code>hickle</code> automatically convert Python data structures (e.g. lists, dictionaries, <code>numpy</code> arrays) into HDF5 groups and datasets. When loading from file, <code>hickle</code> automatically converts data back into its original data type. A key motivation for <code>hickle</code> is to provide high-performance loading and storage of scientific data in the widely-supported HDF5 format.</p>\n\n<p><code>hickle</code> is designed as a drop-in replacement for the Python <code>pickle</code> package, which converts Python object hierarchies to and from Python-specific byte streams (processes known as &#39;pickling&#39; and &#39;unpickling&#39; respectively). Several different protocols exist, and files are not designed to be compatible between Python versions, nor interpretable in other languages. In contrast, <code>hickle</code> stores and loads files from HDF5, for which application programming interfaces (APIs) exist in most major languages, including C, Java, R, and MATLAB.</p>\n\n<p>Python data structures are mapped into the HDF5 abstract data model in a logical fashion, using the <code>h5py</code> package. Metadata required to reconstruct the hierarchy of objects, and to allow conversion into Python objects, is stored in HDF5 attributes. Most commonly used Python iterables (dict, tuple, list, set), and data types (int, float, str) are supported, as are <code>numpy</code> N-dimensional arrays. Commonly-used <code>astropy</code> data structures and <code>scipy</code> sparse matrices are also supported.</p>\n\n<p><code>hickle</code> has been used in many scientific research projects, including:</p>\n\n<ul>\n\t<li>Visualization and machine learning on volumetric fluorescence microscopy datasets from histological tissue imaging.</li>\n\t<li>Caching pre-computed features for MIDI and audio files for downstream machine learning tasks.</li>\n\t<li>Storage and transmission of high volume of shot-gun proteomics data, such as mass spectra of proteins and peptide segments.</li>\n\t<li>Storage of astronomical data and calibration data from radio telescopes.</li>\n</ul>\n\n<p><code>hickle</code> is released under the MIT license, and is available from PyPi via <code>pip</code>; source code is available at <a href=\"https://github.com/telegraphic/hickle\">https://github.com/telegraphic/hickle</a>. Note: this text is modified from the hickle Journal for Open-Source Software paper, https://github.com/telegraphic/hickle/blob/master/paper.md.</p>", 
    "doi": "10.5281/zenodo.2345649", 
    "keywords": [
      "Python", 
      "HDF5", 
      "pickle", 
      "data format"
    ], 
    "license": {
      "id": "CC-BY-4.0"
    }, 
    "publication_date": "2018-12-17", 
    "related_identifiers": [
      {
        "identifier": "10.5281/zenodo.2345648", 
        "relation": "isVersionOf", 
        "scheme": "doi"
      }
    ], 
    "relations": {
      "version": [
        {
          "count": 1, 
          "index": 0, 
          "is_last": true, 
          "last_child": {
            "pid_type": "recid", 
            "pid_value": "2345649"
          }, 
          "parent": {
            "pid_type": "recid", 
            "pid_value": "2345648"
          }
        }
      ]
    }, 
    "resource_type": {
      "title": "Software", 
      "type": "software"
    }, 
    "subjects": [], 
    "title": "Hickle: a HDF5-based python pickle replacement", 
    "version": "3.3.2"
  }, 
  "owners": [
    29663
  ], 
  "revision": 3, 
  "stats": {
    "downloads": 14.0, 
    "unique_downloads": 13.0, 
    "unique_views": 298.0, 
    "version_downloads": 14.0, 
    "version_unique_downloads": 13.0, 
    "version_unique_views": 298.0, 
    "version_views": 343.0, 
    "version_volume": 608552.0, 
    "views": 343.0, 
    "volume": 608552.0
  }, 
  "updated": "2018-12-17T08:09:51.259920+00:00"
}
343
14
views
downloads
All versions This version
Views 343343
Downloads 1414
Data volume 608.6 kB608.6 kB
Unique views 298298
Unique downloads 1313

Share

Cite as