Exploration of Scientific Collections with multimodal agentic RAG systems

In this poster, we explore using multi-modal agents based on Large-Vision-Language-Models (LVLMs) what a scholarly collections portal can be beyond a digital showcase of the university's collections. We focus on the interactive exploration of scientific collections. Collection data is valued differently from different perspectives. For the university administrators, it is an item to be included in budgeting and space management. Public relations uses it to highlight the achievements of the institution, while scientists use it as research data and see the publication as an opportunity to connect with other researchers in their field.

Traditionally, university collection portals reflect administrative structures or collection contexts. This is a perfectly valid approach, but not always conclusive for outsiders, as seen in the access statistics. Our contribution focusses on the perspective of collection data as research data, which, serve a research community as well as the interested public. By using more explorative layers for data linkage and retrieval, we open up new entry points for the collection data and its underlying relations.

Search in conventional portals requires some expertise in field. Enhancing it with an interactive agent, which can answer questions about the portal in general or the objects within, opens up an object-centered and intuitive approach to the data for users of the collection portal. The text-to-image similarity search provides direct retrieval of images by their content in addition to the conventional method, where they are only search using their associated metadata. In this contribution, we provide an overview of the components and interfaces and their integration into the microcosm of the collection portal.

Dieses Poster ist für die Tagung "Digital Turn. Sammlungen – Provenienzen – Märkte", die am 27. und 28. November 2025 an der Humboldt-Universität zu Berlin stattfand, entstanden.