Harvesting knowledge from cultural images with assorted technologies: the example of the ChIA project

: In recent years, cross-disciplinary collaboration for increased knowledge extraction from diverse data sources has been at the heart of interdisciplinary research and related fields, such as Digital Humanities. In particular, knowledge extraction and preservation from cultural heritage data has received increased attention. In this paper, we introduce the ChIA project, a cross-disciplinary Digital Humanities project that aims to perform knowledge design, knowledge extraction and organisation by applying semantic as well as Artificial Intelligence (AI) tools on a set of Europeana cultural food images. The collaborative endeavour aims to increase cultural knowledge access and analysis possibilities of images for different user groups and stakeholders, such as content providers or for educational purposes.


Introduction
Since the recent European Year of Cultural Heritage by the European Commission 1 , the role of science and technology for the benefit of cultural heritage has played a major role in both physical and digital realms. Not only preserving, but also capturing, organising and making accessible cultural knowledge to different user groups has been a widely addressed topic across disciplines in recent years (cf. Hardman et al., 2009;Behli et al., 2018). The ChIA project 2 is a current endeavour between the Austrian Centre for Digital Humanities and Cultural Heritage (ACDH-CH OeAW, AT) 3 , the Adapt Centre, Dublin City University (IE) 4 and the cultural content aggregator Europeana Local -Österreich 5 . The project, which is at an early stage after inception, focuses on integrating semantic technologies and image analysis to enhance the accessibility of cultural images. Through digitisation, cultural images are being transformed from tangible to digital intangible resources and become accessible via different platforms. This significantly increases their availability to the wider public. The transformation, however, mainly focuses on the conversion of the images into their digital representations with generic metadata (e.g. title, creator, caption), overlooking much of the content of the images themselves. Search and retrieval, thus become restricted only to the generic and associated metadata, failing to explain the relevance of the search results. Often, digital cultural images lack a systematic machine-readable description of the cultural and social aspects, as well as semantic enrichment and a well-defined interlinking of the cultural knowledge embedded. In the ChIA project, we aim to increase the knowledge that can be drawn from such images and enable its presentation in a structured way through the support of different digital tools and methods.

The method
The ChIA project draws on expertise from an interdisciplinary team, with backgrounds in semantic technologies, artificial intelligence, cultural heritage aggregation and linguistics. In our approach, recent advancements in technology, including deep learning and computer vision (CV), are applied to analyse and capture the content of a selected set of images from Europeana, using pattern matching algorithms to generate new metadata to enable better search and retrieval. Most of these methods, however, focus on automatic object detection typically in the form of predicted concepts and their probability, still with little coverage on the social and cultural aspects of the images. The ChIA system (see Fig.1) aims to combine these different tools and methods that support knowledge extraction and organisation. Semantic tools enable a unified representation of the entities and foster accurate interpretation; knowledge graphs can be generated by combining all metadata; visual search allows users to search for similar images and a proposed chatbot enables interactive communication exposing the data of Europeana in innovative and new ways. Another aspect that makes the project unique is its specific focus on food-related images (see Figure 2 for an example). The data, provided by Europeana, has previously been curated and provided to Europeana by cultural organisations, including museums, archives, libraries and galleries. As "food" is a rather flexible concept, we specifically concentrate on food that is edible by humans, and images that present it in a cultural setting, which typically involves persons, locations, objects or a combination thereof. Additionally, "culture" may also encompass depictions of family, societal traditions or customs.

The aim and scope of the study
The main content objective of ChIA is to develop knowledge design based on Europeana data derived via the Europeana API and explore & analyse cultural content in an experimental setting using AI. In this setting, we aim to enable access and analysis of cultural images by means of widening search capabilities using a combination of semantic technologies and augmented metadata, with interactive tools like chatbots, knowledge graphs and visual analysis. The project more generally functions as a testbed and playground for experimenting with Artificial Intelligence (AI) in a digital cultural context (cf. Schnapp, 2014). The new knowledge design models will be beneficial for both scientists and other actor groups, i.e. Europeana content providers or for educational purposes, allowing for much more complex searches than simple metadatabased solutions. Another objective concerns the establishing of an intermediate layer infrastructure with the aim to gain additional metadata from Europeana images, enhance the already existing ones and connect AI services from both industry and open source tools for comparative experimentation. Our approach is novel, in that the combination of proposed tools and the resulting knowledge organisation system, have not been applied to images in the framework of Europeana, a digital cultural heritage collection, before. In addition, the topic of choice, food, is an important aspect of mankind's tangible and intangible cultural heritage. Making cultural knowledge from images available beyond given metadata enables widerreaching access, but also increased possibilities for various analyses for both humans and machines (Abgaz et al., 2018). Given the time and available resource scope of the project, we aim to test different scenarios and provide test reports.
The resulting knowledge organisation system (KOS), the ChIA system (see also Figure 1), aims to unite the different aspects of semantic and AI technologies, and give users improved access to as well as improved interaction possibilities with Europeana cultural images depicting food concepts. On the one hand, the system will integrate thesauri with a special focus on food and descriptions of subjects represented in images. Available thesauri will be evaluated and finally reused in a new skosified and open ChIA vocabulary for image description. Thus resource discovery of cultural images that depict food related subjects shall be enabled and improved. The final ChIA food & culture vocabulary will be available in standard SKOS (Simple Knowledge Organization System) format and open for reuse via web services for cataloguing purposes. In line with this, domain-specific relationships between the SKOS concepts will be captured and represented using a separate OWL (Web Ontology Language) ontology. The ChIA food concepts and the ontology will also serve as input to Computer Vision concepts and for further user interaction and data enrichment. On the other hand, the ChIA system also integrates AI results, such as Computer Vision predicted concepts for the selected images analysed. By means of integrating a chatbot, users are also enabled to engage with the images in a more interactive way.

Conclusion and outlook
Finally, ChIA aims to draw on learnings and make use of larger existing European infrastructures such as DARIAH 6 and E-RIHS 7 . By using cultural images related to edible food from the Europeana database the project can draw on a considerable quantity of cultural heritage data derived from a great variety of cultural content holders (museums, archives, libraries, botanical gardens) across Europe. This will provide a vast starting point for analysis, but also shows the need to precisely target the research for dedicated user groups. In addition, the initial analysis pointed out the importance of the selection process for accumulating the optimal test set of raw data for ChIA purposes. It is envisaged that content analysis will start with investigating metadata richness within the first test sets accompanied by image analysis using computer vision functionality and artificial intelligence.