The D-NET software toolkit: dnet-basic-aggregator
Creators
- 1. ISTI-CNR
Description
D-Net Software Toolikt
The D-NET Software Toolkit is a system that offers functionalities for the collection (“harvesting”), transformation, aggregation, and indexing of metadata records collected from an arbitrary number of data sources, complying with different protocols and data exchange formats. D-NET sets a workflow language, which developers can use to combine a variety of D-NET data management services, configure them to handle data according to given data models, and pipeline them into autonomic data processing workflows.
This software package is a simplified version of the D-Net toolkit and consists of a web application with a minimal set of services for:
-
Collection of metadata records in oai_dc format via OAI-PMH, FTP, local file system, HTTP.
-
Transformation of the collected metadata records into an internal format named DMF (Driver Metadata Format)
-
Indexing of DMF records in a Solr full-text index
-
OAI-PMH export of aggregated metadata records in DMF and oai_dc formats. More formats can be added at runtime by providing a dedicated XSLT from DMF to the desired target format.
Major changes in version 1.3.0
- OAI Publisher:
- fixed cache management
- fixed oai consistency (post feed) workflow branch
- fixed deletion of content when workflow of data sources are deleted
- D-Net enabling services:
- using cache for subscription access
- support only one subscription registry
- Mongo based services (mdstore, oaistore, wf logging):
- using API of mongo-java-driver 3.2.2, removed usage of deprecated methods
- tracking the number of stored records to possibly highlight the collection of records with the same identifier
- GUI:
- enabling deletion of APIs via GUI
- enabling editing of metadata_identifier_path
- more info available in the datasource section
- removed map of data sources (TODO: adapt to the new google map API)
- Metadata collection:
- handling HTML illegal entities in collected XMLs
- Indexing:
- default query operator for "bag of words" queries set to AND instead of OR
- Workflow manager
- do not launch workflows that were scheduled for execution during a pause of the aggregation system ("prepare for shutdown")
Official Web Site: http://www.d-net.research-infrastructures.eu
Need support? Contact us via email on dnet-team[at]isti.cnr.it
Files
dnet-team/dnet-basic-aggregator-1.3.0.zip
Files
(392.9 kB)
Name | Size | Download all |
---|---|---|
md5:119ee0ff5f82d26be1987fc607d3df38
|
392.9 kB | Preview Download |
Additional details
Related works
- Is compiled by
- Software: http://svn-public.driver.research-infrastructures.eu/driver/dnet45/modules (URL)
- Is documented by
- Software: https://github.com/dnet-team/dnet-basic-aggregator/tree/1.3.0 (URL)
- References
- Journal article: 10.1108/PROG-08-2013-0045 (DOI)
Funding
- OpenAIRE-Advance – OpenAIRE Advancing Open Scholarship 777541
- European Commission
- OPENAIRE – Open Access Infrastructure for Research in Europe 246686
- European Commission
- OpenAIRE-Connect – OpenAIRE - CONNECTing scientific results in support of Open Science 731011
- European Commission
- PARTHENOS – Pooling Activities, Resources and Tools for Heritage E-research Networking, Optimization and Synergies 654119
- European Commission
- OpenAIRE2020 – Open Access Infrastructure for Research in Europe 2020 643410
- European Commission
- OPENAIREPLUS – 2nd-Generation Open Access Infrastructure for Research in Europe 283595
- European Commission
- ARIADNE – Advanced Research Infrastructure for Archaeological Dataset Networking in Europe 313193
- European Commission