There is a newer version of this record available.

Software Open Access

metno/pyaerocom: Release version 0.8.0

Jonas; Jan Griesfeller; Hanna Svennevik

MARC21 XML Export

<?xml version='1.0' encoding='UTF-8'?>
<record xmlns="">
  <controlfield tag="005">20210224090747.0</controlfield>
  <controlfield tag="001">4159570</controlfield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="u">MET Norway</subfield>
    <subfield code="a">Jan Griesfeller</subfield>
  <datafield tag="700" ind1=" " ind2=" ">
    <subfield code="a">Hanna Svennevik</subfield>
  <datafield tag="856" ind1="4" ind2=" ">
    <subfield code="s">912625</subfield>
    <subfield code="z">md5:0ee7853bbdb742b24bae579c8a320969</subfield>
    <subfield code="u"></subfield>
  <datafield tag="542" ind1=" " ind2=" ">
    <subfield code="l">open</subfield>
  <datafield tag="260" ind1=" " ind2=" ">
    <subfield code="c">2020-10-30</subfield>
  <datafield tag="909" ind1="C" ind2="O">
    <subfield code="p">software</subfield>
    <subfield code="o"></subfield>
  <datafield tag="100" ind1=" " ind2=" ">
    <subfield code="u">Norwegian Meteorological Institute</subfield>
    <subfield code="a">Jonas</subfield>
  <datafield tag="245" ind1=" " ind2=" ">
    <subfield code="a">metno/pyaerocom: Release version 0.8.0</subfield>
  <datafield tag="540" ind1=" " ind2=" ">
    <subfield code="a">Other (Open)</subfield>
  <datafield tag="650" ind1="1" ind2="7">
    <subfield code="a">cc-by</subfield>
    <subfield code="2"></subfield>
  <datafield tag="520" ind1=" " ind2=" ">
    <subfield code="a">&lt;p&gt;Published here with one year delay, for completeness.&lt;/p&gt;
&lt;p&gt;This release comprises major improvements, changes and many new features compared to the last release (comprising about 10 months of development time). Thus, below we only summarise the most important changes. For a list of all changes, please see the changelog file of this release (in subdirectory changelog).&lt;/p&gt;
&lt;li&gt;&lt;p&gt;New sub-package &lt;code&gt;web&lt;/code&gt; (tools for high-level web processing)&lt;/p&gt;
&lt;li&gt;Contains frameworks and routines for high level analysis of data and computation of json files both for AeroCom evaluation and trends web interfaces.&lt;/li&gt;
&lt;li&gt;Includes 2 simple command line interfaces &lt;em&gt;pyaeroeval&lt;/em&gt; and &lt;em&gt;pyaerotrends&lt;/em&gt; for web processing&lt;/li&gt;
&lt;li&gt;Main classes:&lt;ul&gt;
&lt;li&gt;&lt;code&gt;AerocomEvaluation&lt;/code&gt; for processing of data displayed at &lt;a href=""&gt;;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;TrendsEvaluation&lt;/code&gt; for data displayed at &lt;a href=""&gt;;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Gridded reading (&lt;code&gt;ReadGridded&lt;/code&gt; class and methods used therein)&lt;/p&gt;
&lt;li&gt;&lt;code&gt;data_dir&lt;/code&gt; can be provided directly on input in &lt;code&gt;ReadGridded&lt;/code&gt; (e.g. for working locally with no database access). However, data files are required to be in AeroCom naming convention.&lt;/li&gt;
&lt;li&gt;og550gtaer is now primarily computed via od550aer-od550lt1aer&lt;/li&gt;
&lt;li&gt;easy file filtering for all attributes accessible via filename (e.g. model, variable, year, vertical type)&lt;/li&gt;
&lt;li&gt;option to compute variables during runtime for custom methods&lt;/li&gt;
&lt;li&gt;Clean up of outdated methods&lt;/li&gt;
&lt;li&gt;improved logic of processing work-flow, especially for computation of variables and handling of 4D files, e.g.&lt;ul&gt;
&lt;li&gt;use ModelLevel files if Surface is requested but not available&lt;/li&gt;
&lt;li&gt;Compute mass concentration fields (&lt;code&gt;concXX&lt;/code&gt;) from mass mixing ratios (&lt;code&gt;mmrXX&lt;/code&gt;) and density (&lt;code&gt;rho&lt;/code&gt;) fields&lt;/li&gt;
&lt;li&gt;Reading of climatological data (i.e. year 9999 in filename). Remark: tricky, because pandas cannot handle timestamps with year 9999  &lt;/li&gt;
&lt;li&gt;More flexible options for reading of iris cubes (&lt;code&gt;;/code&gt;)&lt;/li&gt;
&lt;li&gt;Improved check and correction of invalid time dimensions in source files&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Ungridded reading (Reading of observations)&lt;/p&gt;
&lt;li&gt;EBAS: implemented framekwork for computation of variables from variables that can be read (or computed)&lt;/li&gt;
&lt;li&gt;EBAS: evaluate and use flag columns (flagged data added to new flag column in &lt;code&gt;UngriddedData&lt;/code&gt; object)&lt;/li&gt;
&lt;li&gt;EBAS: support all occurring sampling frequencies (e.g. weekly, 2daily, etc.)&lt;/li&gt;
&lt;li&gt;EBAS: default now reads raw (i.e. as is in NASA Ames files), but writes all relevant information for filtering (e.g. datalevel, flags) into output &lt;code&gt;UngriddedData&lt;/code&gt; object, which can then be filtered flexibly after reading&lt;/li&gt;
&lt;li&gt;New reading routine for GAW ascii format&lt;/li&gt;
&lt;li&gt;New reading routine for data subset from trends paper by Aas et al.&lt;/li&gt;
&lt;li&gt;Updated EARLINET reading routine after major changes in format (Feb. 2019)&lt;/li&gt;
&lt;li&gt;More flexible handling of cached data objects in &lt;code&gt;ReadUngridded&lt;/code&gt; (cf. changes in caching strategy below)&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Data classes&lt;/p&gt;
&lt;li&gt;Merging of multiple instances possible (including metadata merging and handling of overlapping data)&lt;/li&gt;
&lt;li&gt;Removed attrs. stat_lon, stat_lat, stat_alt&lt;/li&gt;
&lt;li&gt;Support trends computation and visualisation&lt;/li&gt;
&lt;li&gt;Support profile data&lt;/li&gt;
&lt;li&gt;Support for flags and error data&lt;/li&gt;
&lt;li&gt;Outlier removal&lt;/li&gt;
&lt;li&gt;More flexible and robust conversion into StationData&lt;/li&gt;
&lt;li&gt;More advanced filtering and subsetting (e.g. extract single variable)&lt;/li&gt;
&lt;li&gt;Methods for merging of several instances&lt;/li&gt;
&lt;li&gt;Added &lt;strong&gt;iter&lt;/strong&gt; method (looping over data object -&amp;gt; returns StationData at each index, BETA)&lt;/li&gt;
&lt;li&gt;More flexible subsetting (e.g. sel method)&lt;/li&gt;
&lt;li&gt;Method to automatically infer surface level for 3D data&lt;/li&gt;
&lt;li&gt;Cleaned up attributes: now everything is stored within underlying cube (i.e. attr. &lt;code&gt;suppl_info&lt;/code&gt; is deperecated)&lt;/li&gt;
&lt;li&gt;Added CF attributes such as &lt;code&gt;standard_name&lt;/code&gt; and &lt;code&gt;long_name&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;WORK IN PROGRESS: altitude access for 4D fields via &lt;code&gt;get_altitude&lt;/code&gt; method (cleaned up and refactored old code due to below mentioned updates in mod &lt;code&gt;;/code&gt;)&lt;/li&gt;
&lt;li&gt;option to add metadata when converting to timeseries (&lt;code&gt;StationData&lt;/code&gt;) at distinct locations&lt;/li&gt;
&lt;li&gt;Updated I/O and naming conventions&lt;/li&gt;
&lt;li&gt;Region filtering&lt;/li&gt;
&lt;li&gt;time resampling&lt;/li&gt;
&lt;li&gt;All data classes contain many more helper and analysis methods and attributes, that are not explicitely mentioned here, for details see changelog&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Colocation: Improved flexibility and robustness of colocation routines (modules &lt;code&gt;;/code&gt;, &lt;code&gt;;/code&gt;), e.g.&lt;/p&gt;
&lt;li&gt;more control on individual outlier removal for both input datasets&lt;/li&gt;
&lt;li&gt;hierarchical resampling&lt;/li&gt;
&lt;li&gt;option for outlier removal&lt;/li&gt;
&lt;li&gt;option for unit harmonisation&lt;/li&gt;
&lt;li&gt;option for colocating time before downsampling&lt;/li&gt;
&lt;li&gt;option to ignore certain station names (for gridded / ungridded colocation)&lt;/li&gt;
&lt;li&gt;colocation with climatology data&lt;/li&gt;
&lt;li&gt;High level interface (&lt;code&gt;Colocator&lt;/code&gt; class) for automatic colocation, e.g. used in &lt;code&gt;AerocomEvaluation&lt;/code&gt; class for web processing.&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Other changes:&lt;/p&gt;
&lt;li&gt;Updated method &lt;code&gt;calc_statistics&lt;/code&gt;: biases (NMB, MNMB) and FGE are now computed only from positive values&lt;/li&gt;
&lt;li&gt;New modules &lt;code&gt;;/code&gt; providing custom unit conversion, e.g. for non-CF conform units in data files (e.g. sulphur specific mass concentration data: ug S m-3)&lt;/li&gt;
&lt;li&gt;Improved caching stragegy (now single variable instances of &lt;code&gt;UngriddedData&lt;/code&gt; are cached)&lt;/li&gt;
&lt;li&gt;Easier installation options&lt;/li&gt;
&lt;li&gt;Support for simple geographical calculations&lt;/li&gt;
&lt;li&gt;New helpers and processing methods in &lt;code&gt;;/code&gt;&lt;/li&gt;
&lt;li&gt;Support for more variables&lt;/li&gt;
&lt;li&gt;Advanced and unified time resampling in &lt;code&gt;TimeResampler&lt;/code&gt; class&lt;/li&gt;
&lt;li&gt;More CF-compliant (e.g. &lt;code&gt;units&lt;/code&gt; attr. in data classes)&lt;/li&gt;
&lt;li&gt;More flexible and unified handling (and sharing) of metadata among different data objects&lt;/li&gt;
&lt;li&gt;Methods for trends computation (class &lt;code&gt;TrendsEngine&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;Major improvements in ungridded caching using single variable cache files for I/O&lt;/li&gt;
&lt;li&gt;Bug fixes&lt;/li&gt;
&lt;li&gt;New class &lt;code&gt;TsType&lt;/code&gt; for handling and comparing temporal resolutions (in new mod &lt;code&gt;;/code&gt;)&lt;/li&gt;
&lt;li&gt;More flexible tests (using pytest markers that check access to database)&lt;/li&gt;
&lt;li&gt;Worked on implementation of vertical coordinate to altitude conversion methods (WORK IN PROGRESS, mod. &lt;code&gt;;/code&gt;)&lt;/li&gt;
&lt;li&gt;&lt;p&gt;API changes:&lt;/p&gt;
&lt;li&gt;&lt;code&gt;Station&lt;/code&gt; class is deprecated&lt;/li&gt;
&lt;li&gt;&lt;code&gt;ReadGriddedMulti&lt;/code&gt; is deprecated (but still works)&lt;/li&gt;
&lt;li&gt;sconc variables are deprecated (but still work): use conc instead (e.g. concso4 instead of sconcso4)&lt;/li&gt;
&lt;li&gt;Renaming of classes / modules:&lt;ul&gt;
&lt;li&gt;&lt;code&gt;AllVariables&lt;/code&gt; to &lt;code&gt;VarCollection&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;unit&lt;/code&gt; to &lt;code&gt;units&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Moved &lt;code&gt;GridIO&lt;/code&gt; class from &lt;code&gt;;/code&gt; to dedicated new module &lt;code&gt;;/code&gt;&lt;/li&gt;
&lt;li&gt;Global setup dictionaries for time conversion moved from &lt;code&gt;;/code&gt; to &lt;code&gt;;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Not finished / under development / coming soon&lt;/p&gt;
&lt;li&gt;Handling of vertical model coordinates&lt;/li&gt;
&lt;li&gt;Colocation of profile data&lt;/li&gt;
&lt;li&gt;Filtering by land / sea masks&lt;/li&gt;
&lt;li&gt;Computation of regional average time series in data objects&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Planned major changes for v0.9.0:&lt;/p&gt;
&lt;li&gt;API refactor: StationData based on xarray.Dataset (currently variable data can be either numpy array, pandas Series or xarray)&lt;/li&gt;
&lt;li&gt;Include filtering using land / sea masks (should work for &lt;code&gt;GriddedData&lt;/code&gt;, &lt;code&gt;UngriddedData&lt;/code&gt;, &lt;code&gt;ColocatedData&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;4D data (ModelLevel):&lt;ul&gt;
&lt;li&gt;conversion of vertical level coordinates to altitude&lt;/li&gt;
&lt;li&gt;profile colocation (would add additional vertical dimension to &lt;code&gt;Colocateddata&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;Retrieval of aerosol layer height (PRODUCT)&lt;/li&gt;
&lt;li&gt;Default vertical domains for vertical aggregation (particularly for web interfaces, e.g. 0-2km, 2-6km, &amp;gt;6km)&lt;/li&gt;
  <datafield tag="773" ind1=" " ind2=" ">
    <subfield code="n">url</subfield>
    <subfield code="i">isSupplementTo</subfield>
    <subfield code="a"></subfield>
  <datafield tag="773" ind1=" " ind2=" ">
    <subfield code="n">doi</subfield>
    <subfield code="i">isVersionOf</subfield>
    <subfield code="a">10.5281/zenodo.4139878</subfield>
  <datafield tag="024" ind1=" " ind2=" ">
    <subfield code="a">10.5281/zenodo.4159570</subfield>
    <subfield code="2">doi</subfield>
  <datafield tag="980" ind1=" " ind2=" ">
    <subfield code="a">software</subfield>
All versions This version
Views 11330
Downloads 60
Data volume 7.8 MB0 Bytes
Unique views 8626
Unique downloads 60


Cite as