Data Liberation Toolkit (DLT)¶

The DLT is a set of open source tools made by Cauldron for the sustainability assessment community. They focus on data interoperability and dealing with well-known difficult challenges.

ecoinvent_interface¶

ecoinvent_interface is the (unofficial) missing API for ecoinvent. Get all ecoinvent information, including releases, reports, and process-specific data. This toolkit also harmonizes and fixes some ecoinvent data issues.

Authentication¶

ecoinvent_interface requires a valid ecoinvent license and agreement to the data privacy policy via the web browser. After that you can set your login identification via environment variables, at run-time, or via the permanent_setting function:

from ecoinvent_interface import Settings, permanent_setting
permanent_setting("username", "bob")
permanent_setting("password", "example")
# Secrets files read automatically
my_settings = Settings()

EcoinventRelease interface¶

In [1]:
from ecoinvent_interface import EcoinventRelease, Settings, ReleaseType
my_settings = Settings()
ei = EcoinventRelease(my_settings)

Database releases¶

To get a database release, we need to make three selections. First, the version:

In [2]:
ei.list_versions()
Out[2]:
['3.11',
 '3.10.1',
 '3.10',
 '3.9.1',
 '3.9',
 '3.8',
 '3.7.1',
 '3.7',
 '3.6',
 '3.5',
 '3.4',
 '3.3',
 '3.2',
 '3.1',
 '3.01',
 '2']

Second, the system model:

In [3]:
ei.list_system_models('3.11')
Out[3]:
['cutoff', 'consequential', 'EN15804', 'apos']

The ecoinvent API uses a short and long form of the system model names; you can get the longer names by passing translate=False. You can use either form in all EcoinventRelease methods.

In [4]:
ei.list_system_models('3.11', translate=False)
Out[4]:
['Allocation cut-off by classification',
 'Substitution, consequential, long-term',
 'Allocation, cut-off, EN15804',
 'Allocation at the Point of Substitution']

Finally, the type of release. These are stored in an Enum. There are six release types; if you just want the database to do calculations choose the ecospold type.

ReleaseType.ecospold: The single-output unit process files in ecospold2 XML format
ReleaseType.matrix: The so-called "universal matrix export"
ReleaseType.lci: LCI data in ecospold2 XML format
ReleaseType.lcia: LCIA data in ecospold2 XML format
ReleaseType.cumulative_lci: LCI data in Excel
ReleaseType.cumulative_lcia: LCIA data in Excel

See the ecoinvent website for more information on what these values mean.

Once we have made a selection for all three choices, we can get the release files. They are saved to a cache directory and extracted by default.

In [5]:
ei.get_release(version='3.11', system_model='apos', release_type=ReleaseType.ecospold)
Out[5]:
PosixPath('/Users/cmutel/Library/Application Support/EcoinventInterface/cache/ecoinvent 3.11_apos_ecoSpold02')

By default ecoinvent_interface will normalize release filenames to all fit the same pattern, and modify the ecospold files to add in the correct version numbers.

EcoinventProcess interface¶

This class gets data and reports for specific processes. It first needs to know what release version and system model to work with:

In [6]:
from ecoinvent_interface import EcoinventProcess
ep = EcoinventProcess(my_settings)
ep.set_release(version="3.11", system_model="apos")

You can then select a process by its filename, its internal ecoinvent id number, or by its attributes like name and reference product.

In [7]:
ep.select_process(dataset_id="40")

Basic process information¶

Once you have selected the process, you can get basic information about that process:

In [8]:
ep.get_basic_info()
Out[8]:
{'index': 40,
 'version': '3.11',
 'system_model': 'apos',
 'activity_name': 'rye seed production, Swiss integrated production, for sowing',
 'geography': {'comment': None,
  'short_name': 'CH',
  'long_name': 'Switzerland'},
 'reference_product': 'rye seed, Swiss integrated production, for sowing',
 'has_access': True,
 'unit': 'kg',
 'sector': 'Agriculture & Animal Husbandry'}

Process documents¶

You can use ep.get_file with one of the following file types to download process files:

  • ProcessFileType.upr: Unit Process ecospold XML
  • ProcessFileType.lci: Life Cycle Inventory ecospold XML
  • ProcessFileType.lcia: Life Cycle Impact Assessment ecospold XML
  • ProcessFileType.pdf: PDF Dataset Report
  • ProcessFileType.undefined: Undefined (unlinked and multi-output) Dataset PDF Report

ecoinvent_migrate¶

ecoinvent_migrate uses ecoinvent_interface to read the Excel change report, and combines that with its own examination of differences between release versions to generate randonneur data files which can added to randonneur_data

randonneur¶

randonneur is a library to make changes to life cycle inventory databases. Specifically, randonneur provides the following:

  • A data format for specifying life cycle inventory data transformations
  • Helper functions to create and validate data in this data format
  • Functions to apply the transformations to data

randonneur objects are self-contained JSON files.

randonneur metadata schema¶

The metadata scheme follows the datapackage.json Open Knowledge Foundation standard. We additionally require the folllowing:

  • mapping: A dictionary mapping the labels used in the transformation to data accessors.
  • graph_context: A list with either the string 'nodes', 'edges', or both 'nodes' and 'edges'. This defines what kinds of objects in the graph should be transformed.

We also recommend adding the following:

  • source_id: An identifier for the source dataset following the common identifier standard. Useful if the source data is specific.
  • target_id: An identifier for the target dataset following the common identifier standard. Useful if the target data is specific.

Defining a mapping¶

The mapping goes from the labels used within a randonneur file to accessors for the specific data attributes in the source data. It can be defined using XPath, JSONPath, or any other similar standard.

"mapping": {
    "source": {
      "expression language": "XPath",
      "labels": {
        "name": "//*:elementaryExchange/*:name/text()",
        "unit": "//*:elementaryExchange/*:unitName/text()",
        "uuid": "//*:elementaryExchange/@elementaryExchangeId"
      }
    },
    "target": {
      "expression language": "XPath",
      "labels": {
        "name": "//*:elementaryExchange/*:name/text()",
        "unit": "//*:elementaryExchange/*:unitName/text()",
        "uuid": "//*:elementaryExchange/@elementaryExchangeId"
      }
    }
  }

Change types¶

  • create
  • replace
  • update
  • delete
  • disaggregate

randonneur in action¶

randonneur is a standard and reference implementation. We have prepared a number of common migration files in randonneur_data.

In [9]:
import randonneur_data as rd
registry = rd.Registry()
In [11]:
registry.sample('ecoinvent-3.8-biosphere-ecoinvent-3.9-biosphere')
Out[11]:
{'delete': [{'source': {'uuid': '288c4e8a-d371-4bc1-b45b-c4b0cefd400e',
    'name': 'Methane, from soil or biomass stock'},
   'comment': 'indoor compartment deleted as not used'},
  {'source': {'uuid': 'c5c25aa6-d630-40bd-bed7-4e718c877ef4',
    'name': '[Deleted]Tri-allate'},
   'comment': 'should already have been deleted last time'}],
 'replace': [{'source': {'name': 'Iron',
    'formula': 'Fe',
    'unit': 'kg',
    'uuid': 'c983ad54-12e9-47a5-b747-5e5c448e58e2'},
   'target': {'name': 'Iron ion',
    'uuid': 'c983ad54-12e9-47a5-b747-5e5c448e58e2'},
   'comment': 'Flow attribute change not listed in change report'},
  {'source': {'name': 'Arsenic',
    'formula': 'As',
    'unit': 'kg',
    'uuid': 'dc6dbdaa-9f13-43a8-8af5-6603688c6ad0'},
   'target': {'name': 'Arsenic ion',
    'uuid': 'dc6dbdaa-9f13-43a8-8af5-6603688c6ad0'},
   'comment': 'Flow attribute change not listed in change report'}]}
In [12]:
registry.schema('ecoinvent-3.8-biosphere-ecoinvent-3.9-biosphere')
Out[12]:
{'source': {'expression language': 'XPath',
  'labels': {'name': '//*:elementaryExchange/*:name/text()',
   'unit': '//*:elementaryExchange/*:unitName/text()',
   'uuid': '//*:elementaryExchange/@elementaryExchangeId',
   'formula': '//*:elementaryExchange/@formula',
   'context': ['//*:elementaryExchange/*:compartment/*:compartment/text()',
    '//*:elementaryExchange/*:compartment/*:subcompartment/text()']}},
 'target': {'expression language': 'XPath',
  'labels': {'name': '//*:elementaryExchange/*:name/text()',
   'unit': '//*:elementaryExchange/*:unitName/text()',
   'uuid': '//*:elementaryExchange/@elementaryExchangeId',
   'formula': '//*:elementaryExchange/@formula',
   'context': ['//*:elementaryExchange/*:compartment/*:compartment/text()',
    '//*:elementaryExchange/*:compartment/*:subcompartment/text()']}}}
In [13]:
registry.sample('simapro-9-ecoinvent-3-context')
Out[13]:
{'replace': [{'source': {'context': ['Soil', 'agricultural']},
   'target': {'context': ['soil', 'agricultural']}},
  {'source': {'context': ['Emissions to soil', 'unspecified']},
   'target': {'context': ['soil', 'unspecified']}}]}
In [14]:
registry.sample('simapro-ecoinvent-3.10-cutoff')
Out[14]:
{'replace': [{'source': {'identifier': 'EI3ARUNI000011519610859',
    'name': 'Waste frit from cathode ray tube production {GLO}| market for waste frit from cathode ray tube production | Cut-off, U',
    'platform_id': 'E429BB7B-74E0-4B45-BADD-42D2514D5BC8'},
   'target': {'filename': 'b1b552af-0357-523c-b22f-a51080c47486_88dc8f7d-58d5-47c7-9165-428f977473ff.spold',
    'name': 'market for waste frit from cathode ray tube production',
    'location': 'GLO',
    'reference product': 'waste frit from cathode ray tube production'}},
  {'source': {'identifier': 'EI3ARUNI000011519617912',
    'name': 'Concrete, 35MPa {ZA}| concrete production, 35MPa, with cement, CEM II/B-V | Cut-off, U',
    'platform_id': 'F6CB3804-F435-4C2A-B029-2CDAD13C2C3E'},
   'target': {'filename': 'f7367209-2485-5ee9-9d42-c9d5785ac3d7_b1f74287-042b-4171-b90c-b1eb5739f30d.spold',
    'name': 'concrete production, 35MPa, with cement, CEM II/B-V',
    'location': 'ZA',
    'reference product': 'concrete, 35MPa'}}]}
In [15]:
registry.sample('ecoinvent-3.9.1-cutoff-ecoinvent-3.10-cutoff')
Out[15]:
{'replace': [{'source': {'name': 'treatment of dross from Al electrolysis, residual material landfill',
    'location': 'CH',
    'reference product': 'dross from Al electrolysis',
    'unit': 'kg'},
   'target': {'name': 'treatment of dross from Al electrolysis, residual material landfill',
    'location': 'RER',
    'reference product': 'dross from Al electrolysis',
    'unit': 'kg'}},
  {'source': {'name': 'treatment of hard coal ash, sanitary landfill',
    'location': 'CH',
    'reference product': 'hard coal ash',
    'unit': 'kg'},
   'target': {'name': 'treatment of hard coal ash, sanitary landfill',
    'location': 'GLO',
    'reference product': 'hard coal ash',
    'unit': 'kg'}}],
 'disaggregate': [{'source': {'name': 'treatment of wastewater, average, capacity 4.7E10l/year',
    'location': 'GLO',
    'reference product': 'wastewater, average',
    'unit': 'm3'},
   'targets': [{'name': 'treatment of wastewater, average, wastewater treatment',
     'location': 'CA-QC',
     'reference product': 'wastewater, average',
     'unit': 'm3',
     'allocation': 0.005618761409913913},
    {'name': 'treatment of wastewater, average, wastewater treatment',
     'location': 'CH',
     'reference product': 'wastewater, average',
     'unit': 'm3',
     'allocation': 0.011441849586346517},
    {'name': 'treatment of wastewater, average, wastewater treatment',
     'location': 'Europe without Switzerland',
     'reference product': 'wastewater, average',
     'unit': 'm3',
     'allocation': 0.4051017522415871},
    {'name': 'treatment of wastewater, average, wastewater treatment',
     'location': 'RoW',
     'reference product': 'wastewater, average',
     'unit': 'm3',
     'allocation': 0.5778376367621525}]},
  {'source': {'name': 'soda ash, light, crystalline, heptahydrate, to generic market for neutralising agent',
    'location': 'GLO',
    'reference product': 'neutralising agent, sodium hydroxide-equivalent',
    'unit': 'kg'},
   'targets': [{'name': 'soda ash, light, crystalline, heptahydrate, to generic market for neutralising agent',
     'location': 'RoW',
     'reference product': 'neutralising agent, sodium hydroxide-equivalent',
     'unit': 'kg',
     'allocation': 0.8039999999999998},
    {'name': 'soda ash, light, crystalline, heptahydrate, to generic market for neutralising agent',
     'location': 'RER',
     'reference product': 'neutralising agent, sodium hydroxide-equivalent',
     'unit': 'kg',
     'allocation': 0.19600000000000017}]}]}
In [ ]:
list(registry)

bw_simapro_csv¶

A robust library for parsing SimaPro CSV files. This format poses a number of challenges we have addressed (as far as possible). We will illustrate this with the Agribalyse import example.

Links¶

  • ecoinvent_interface
  • ecoinvent_migrate
  • randonneur
  • randonneur_data
  • bw_simapro_csv
In [ ]: