Auditing the Borealis Norwegian Language Models

We report a large-scale adversarial safety audit of the Borealis family of Norwegian language models (National Library of Norway / NB AI Lab), built on Google’s Gemma-3. Using the SimpleAudit red-teaming framework (auditor → target → judge), we evaluate 21 models: 15 Borealis variants ({270M, 1B, 4B, 12B, 27B} × {full, instruct-preview, open}), the five Gemma-3-it models they are fine-tuned from, and gpt-5-mini as a frontier reference, across six benchmark suites (four Norwegian-domain: safety, healthcare, public sector, language; two epistemic: BullshitBench v1/v2), with n=10 reruns each (≈1,200 audits, 38,391 audited scenario instances). Three findings stand out. (1) Norwegian fine-tuning paid off where it was meant to: Borealis beats its Gemma-3 foundation on Norwegian-domain benchmarks at every size ≥1B, peaking at the 4B scale (+13.9 pts mean, up to +21.3 on NO-Safety) and at 27B on healthcare (+22.8 pts, p < 0.05). (2) But there is a measurable epistemic cost: at 12B/27B, Borealis is significantly worse than Gemma on BullshitBench v1 (−3.5, −5.1 pts), and the instruct-preview checkpoint scores consistently below the full release on Norwegian-domain safety (e.g. −3.7 on NO-Safety, ; within the ±5-pt Norwegian band and not individually significant). (3) Epistemic safety is broadly unsolved: across all 21 models (including sub-1B models unfit for deployment) 87.5% of responses are high or critical under a persistent adversarial auditor, and even the deployable flagships still fail ≈1 in 5 (critical), while the hardest high-stakes medical false-premise scenarios elicit critical failures in 87–93% of all model-runs. (These are adversarial, relative figures, not benign-use prevalence rates.) We also quantify measurement uncertainty: the residual noise of the n= 10 estimate is small for the large BullshitBench sets (±1–1.5 pts) but substantial for the small (8–10 scenario) Norwegian benchmarks (median 95% CI half-width ≈5 pts), so Norwegian deltas must be read with care. We close with deployment guidance.