Reference Set and MLLM Visual Information Extraction Prototype

Manually extracted data from figures and tables reported in pharmacology studies previously collected as part of an internal reference set. The dataset contains images from the visual elements and their corresponding values from 45 published pharmacology studies with distinct PubMed IDs. Multiple images could be sampled from each of these papers, and multiple values were often sampled from each image. Therefore, the reference set contains multiple rows with values from images of tables or from figures corresponding to graphs, plots, or charts. Dataset contains annotated information from 43 images of figueres and 40 images of tables. The visual elements contain data from any of eight different types of experiments, namely in vitro enzyme inhibition, induction, & kinetics, in vitro transporter inhibition, induction, and kinetics, as well as in vivo enzyme kinetics and in vivo interaction studies. The selected sample represents a wide range of styles, layouts, and structures for both figures and tables.

We also provide code from our MLLLMs Visual Information Extraction prototype using the Pydantic AI v1.25 Python module to connect with multiple models to perform VIE and produce a structured JSON output. Our pilot VIE system was used to process images from the reference set along with the rest of the annotated information to generate prompts. 

We have evaluated the following models.

Error corrections:

Within the "Manuscript Results folder" > "Tolerance Based ACC.ods" the calculation of Tolerance based accuracy for cells F12-J21 was incorrectly calculated by dividing the corresponding cell F1-J10 over 172 instead of 162. For example the correction for the value of cell F12 is to change  it's content from "=ROUND(F1/172,3)*100" to "=ROUND(F1/162,3)*100".