Promoting Open Science in times of Artificial Intelligence: Do we grasp the interplay?

Promoting Open Science in times of Artificial Intelligence: Do we grasp the interplay?

An awareness raising poster presented at the World Conference of Research Integrity 2024 that contrasts the logic and principles of Research Integrity (RI) and Research Ethics (RE) with the functionality of Large Language Models (LLM) (such as ChatGPT). We argue that these logics are too different to integrate LLMs into the RI-oriented research process without careful consideration. However, we consider it equally problematic to play them off against each other in the sense of an either/or.

What is it about? 

1. Training
Apart from fine-tuned and RAG-models, large generative LLMs, which are now used in research, are pre-trained on a large corpus of data from various (non-)scientific resources. The distinction between reliable, peer-reviewed, scientific etc. is not applied at this point (whereas researchers are encouraged to use scientific literature/data in the traditional research process).
Based on that training, responses are generated (from something to which we ascribe the attribute “BlackBox”) based on the process of tokenization and the tokens' statistical and probabilistic nature.
This functionality of probabilistic modeling of token sequences is the state of affairs with which we (also in research) go into prompting. And at the same time, it is the reason why, in our opinion, questions of reliability, accountability, factuality and traceability are irrelevant at this point – yet central to many scientific discussions. This is where the reasoning gets lost: of course, these principles are THE fundamental treasures of today's technology-based research. But in relation to the functionality described above, they are not placed correctly. 

2. Prompting
Prompting for well-researched topics (which have a relatively homogeneous structure in the training data set), such as the Theory of Relativity, might provide relatively reliable outputs. But what about topics that are not mainstream, appear less frequently and homogeneously in the training, such as innovations (and here we are with the task of research). 

3. The problem for RI and OS
The parameter for weighting probabilities is the datas' quantitative prominence. Open access publishing etc. is supportive here. At the same time, some (yet valuable) RI/RE principles and IPRs can make OS (= assumed reliable) data under-represented for AI training (building on existing data, reducing the likelihood of redundant studies, peer-review assessing originality and limiting similar publications, ethical reasons for withholding data, IPRs). We must conclude, that OS resources won't suffice to be THE main source for LLM training.

4. We have to discuss in order to deal with these fundamentally different logics: 
researchers must be aware of these limits and use generative LLMs with an appropriate reflectivity
developers should be aware that principles of good research practices matter!
we, as research community, must reflect on our processes and principles and readjust them if necessary.

5. We though conclude
Promoting OS as widely as possible could overcome the current under-representation, thereby refining the quality of knowledge that is freely accessible.