Designing for Trust in AI-Assisted Scientific Computing

As AI assistance lowers the barrier to coding, domain experts with limited programming experience are increasingly able to make use of scientific computing software. Yet a critical gap remains: evaluating and understanding the AI-generated code is still a significant challenge, making it difficult to verify and trust results. This talk presents lessons from building a computational optimization platform for chemical process engineering, sharing UX strategies to build confidence among non-programmers by enabling them to visualize underlying conceptual models and validate AI-driven changes. 

AI tooling is becoming increasingly common in research software, and there is growing concern about new classes of subtle, hard-to-detect errors. Unlike obvious failures, AI-generated code changes can produce plausible but incorrect results that trigger no exceptions and leave no traces. This is especially problematic in model based scientific software where the internal reasoning can be opaque and complex code changes can be a "black box" for non-programmers. For these users, AI-facilitated changes may feel like trial and error guesswork rather than a deliberate and strategic process. Without better visibility and validation, researchers risk building on flawed results that are difficult and costly to correct later.

Drawing on our experience building the Process Optimization and Modeling for Minerals Sustainability (PrOMMiS) Flowsheet Inspector, a domain-specific application designed for code-driven computational software, this talk shares UX strategies for building trust and confidence among  domain experts with limited programming experience. We map the user journey and design considerations required to surface underlying conceptual models, empowering users to seamlessly execute, debug, and verify AI-generated modifications. Through examples from the app, we demonstrate the UX patterns that help to make the diagnostic information more accessible, improve error discoverability, and empower users to evaluate changes through an intuitive interface.