D4.3 – Final report on dataset integration

This deliverable describes activities carried out within Work Package 4 (WP4) up to Month 44 of the ARIADNEplus project. It describes the outcomes of dataset integration activity (Tasks 4.2 and 4.3) and updates Deliverable D4.1. It also includes an assessment of the support activities carried out in Tasks 4.1 and 4.6, and the digital libraries activity undertaken within Task 4.5. The implementation of the different application profiles carried out in Task 4.4 was reported in D4.2 and will be updated in Deliverable D4.4 at Month 48.

The document is structured according to the activities carried out by WP4. After an overall presentation of the main aims of WP4 in Section 1, Section 2 provides an overview of the AO-Cat conceptual model which was developed in order to facilitate interoperability across the broad range of data resources provided by partners. This was largely reported on in D4.1 but the core features are
repeated in this final deliverable for the sake of completeness, and enhancements introduced since Month 18 are highlighted in Section 2.2. Section 2 also presents the fundamental categories defined to classify the information in the Catalogue and make it easily available on the new ARIADNEplus Portal, according to the recommendations of the FAIR principles which inspired the model.

Section 3 presents the dataset integration process, and the principles which have been adopted to maximise interoperability. It describes the pipeline which has been established to support data integration.

Section 4 presents the tools developed and implemented in ARIADNEplus to facilitate data integration according to the AO-Cat model and to assist content providers in all phases of their ingestion, mapping, transformation, enrichment and publication activities. The 3M Mapping Tool developed by FORTH is fundamental as it allows us to define in detail the correspondences between the data
provider metadata schemas and the entities of the AO-Cat model, in order to implement an optimal level of integration. The FastCat tool, also developed by FORTH, has been designed for the rapid acquisition of information directly in the AO-Cat model. It has been made available to partners who do not use a pre-defined format for their metadata or who have a limited amount of data to be
provided.

Partners must also map their subject vocabularies to the Getty Art and Architecture Thesaurus (AAT), and the Vocabulary Matching Tool, developed by the University of South Wales (USW), provides a straightforward means of defining matches. It has been implemented as a publicly available ARIADNE Service in D4 Science. Temporal periods are defined using the PeriodO web publishing service which allows us to add From and Until dates in absolute years BCE and AD during the metadata enrichment
stage. However, in some cases From and Until dates may already exist in the source metadata, but are often represented in a variety of textual formats. In these cases the USW YearSpans tool is inserted into the workflow to extract numeric values for the From and Until dates. YearSpans automatically performs normalisation based on a wide set of date patterns commonly used in archaeology.

This section also presents Activity Dash, a dashboard system developed by FORTH which allows the aggregation task force to keep track of the multiple workflows, to raise queries, and to provide feedback to partners. Finally, we report on the use of the Helpdesk, a collaborative service provided within the ARIADNEplus D4 Science platform to assist content providers in all phases of the data
contribution and to provide assistance at every stage of the process, from preparation to the definition of mappings, up to fine-tuning the data harvesting and data acquisition mechanisms in the ARIADNEplus infrastructure. The service is based on the ticketing system and offers efficient interaction with the special team of experts set up to provide all the necessary information to foster
the process.

Section 5 documents the status of data integration, and shows how partners have adapted their data to the ARIADNEplus standards. We are able to report outstanding success in achieving a critical mass of high-quality archaeological data resources, available for research under an open licence, and easily discoverable via the ARIADNE portal, which represents a powerful research tool in its own right. As this Deliverable is submitted, we have over three million resources integrated in the portal, and anticipate that by Month 48 that figure will be closer to four million. The current resources derive from over 30 partner data providers (or “Publishers”) from as many countries, and covering three continents. We anticipate that by Month 48 over 40 data providers will be represented. Most significantly this includes data from over ten Associate Partners who have asked to supply data at their own expense, but with support of the ARIADNE Aggregation Task Force. These include the existing specialist multinational aggregation services ROCEEH and THANADOS. There is a clear appreciation of the importance of ARIADNE amongst the world archaeological community and an appetite from those beyond the original project partnership to make their data sets available via the ARIADNE platform.

Section 6 presents the activities aimed at linking the ARIADNEplus Data Infrastructure with repositories of scientific publications, exploiting, in particular, OpenAIRE and its open access digital archives and the links to individual journals such as Internet Archaeology. The gateway for Digital Humanities and Cultural Heritage, whose features have been updated and improved, is also presented. In addition, we outline the methodologies used by the ARIADNEplus text mining service and the results produced by its relational citation matching algorithm, used to improve the metadata generation of relevant textual resources and dynamically link them to the ARIADNEplus ecosystem.

Finally, the conclusions and an evaluation of the activities carried out in the project are presented in Section 7, along with plans for sustainability of data integration activities. ARIADNEplus has been outstandingly successful in bringing together a much broader range of records than the predecessor ARIADNE project, upon whose success it has been able to build. The intention was to integrate datasets ranging widely in space and time, and across a wider range of archaeological sub-domains. All these ambitions have been achieved. At Month 44 we have over 1120 resources which pre-date 1,000,000 BP, and 62,723 resources which post-date AD 1900, demonstrating the full range of 21st century archaeological research. We have resources from North and South America, Europe and Scandinavia, and Asia. The integrated data covers every ARIADNE resource type, from inscriptions and rock art to scientific analyses and dating resources. We have integrated over one million records for individual sites and monuments, over 500,000 artefacts and over 450,000 coins. Our catalogue includes over 250,000 unpublished fieldwork reports, and over 100,000 full fieldwork archives. This is the first time that many of these resources have been brought together, making it possible to address
some of the grand challenges of archaeology. In addition, the data integration approach and supporting infrastructure have proved their value and extensibility for use in other contexts, at national, regional or thematic level, allowing the same data resources to be presented in multiple showcases. In summary, what has been achieved in ARIADNEplus represents a massive digital research
infrastructure now available freely and easily as Open Access data to European scientists and researchers, teachers and students, as well as to interested members of the public.