GenAI experiments: Extracting knowledge from educational materials
Authors/Creators
-
1.
Austrian Institute of Technology
Description
This collection contains the results of four ClimEmpower / MAIA GenAI experiments. These experiments aim to asess how and to what extent the Generative AI models can help knowledge curators extract knowledge from documents they need to analyse.
Concrete high-level research questions these experiments aim to resolve are:
RQ1: To what extent can the AI answers be used to formulate the final answers, without reading the whole document?
RQ2: Which types of questions are easier or more difficult for GenAI models to answer?
RQ3: How, and to what extent, can the answers be improved through prompt engineering?
RQ4: To what extent do the GenAI models follow instructions to base the answers (only) on the content provided in the document?
RQ5: How does the choice of GenAI model reflect in experiment results?
In addition, we were also interested in finding out the ways to further improve the SumQA, a Generative AI service that was developed in the MAIA project and supports batch-processing of documents.
Notes
ClimEmpower project team aims to provide the regional stakeholders with a library of “educational materials” that can be used for self-study by various types of regional stakeholders to increase their Climate Adaptation and Mitigation literacy. To achieve this goal, the project team has curated a large set of “educational materials”, with a main focus on materials that can be easily digested in a short time, such as educational videos and popular science texts. In order to make these materials easier to find, the project team aims to describe each of them according to the following schema:
- Document title.
- Short summary of the key messages.
- Reason for including this document in the collection (anticipated relevance and/or benefits for the regional stakeholders).
- Intended target audience of the document (e.g. decision makers, practitioners, scientists. . . ).
- Document type (e.g. scientific article, educational curriculum, popular science text. . . ).
- Hazards discussed in the document.
- Sectors and elements at risk discussed in the document.
- Threats, risks and impacts that are discussed in the document.
- Specific development pathways and/or solutions that are discussed in the document.
Although the task at hand may seem almost trivial, it required a lot of dedicated efforts from scientists working on the project and ensuring a coherent style and quality of the document analysis was challenging. This raises the question of sustainability of this development, as the progress has been slow so far and the efforts required to keep the library up to date and integrate new materials will be difficult to finance after the project ends.
With this in mind, we decided to find out if, how, and to what extent this process can be made more efficient with the help of the GenAI. ClimEmpower GenAI experiments presented in this data set are split in four distinct sub-experiments, each with their own set of input documents, AI questions and research questions. In all four sub-experiments, GenAI models are asked to analyse the input document(s) and indicate the document title, summarise the main messages, explain the relevance thereof for the stakeholders, and to categorise the input in several ways: open category, intended stakeholder target groups, hazards, risks/impacts. All four experiments were facilitated by SumQA batch processing service, which was developed in MAIA project.