Cross-Lingual Hallucination Drift in LLMs

Does an LLM hallucinate more in some languages, and does it depend on the task?

NLP ResearchLLM-as-a-JudgeAya Expanse 8BGPT-4o-miniHuggingFaceSciPyStreamlit

View code ↗

Output

Interactive results dashboard (Streamlit): KPI metrics and the color-coded 6-cell hallucination-rate table. Swahili XCOPA jumps out in red at 98.7% while every other cell stays low.

Reason Analysis view: of everything the judge flagged, the dominant failure mode is "incoherent / off-topic" (47.8%), not fabricated facts. The model rambles rather than confidently making things up.

Browse Judge Reasons: the raw GPT-4o-mini explanations behind each label, here for Swahili XCOPA: "the response does not address the question and instead provides an unrelated narrative."

The core finding as a clean figure: TruthfulQA is flat across English, Spanish, and Italian (~25–29%); XCOPA is low for English and Italian (~8–9%) but Swahili spikes to 98.7%.

Two deeper findings: Swahili hallucinations run ~2× longer (incoherent rambling, not confident fabrication), and the LLM judge’s own reliability (Cohen’s κ) is substantial on QA but poor on commonsense, so judge validity is task-dependent.

Problem

Do LLMs hallucinate more in lower-resource languages, and does that drift depend on the task? A tempting early result (huge drift on Swahili commonsense reasoning but none on Spanish factual QA) conflates two things: is it the task, or just the specific language? Measuring this rigorously means controlling for that confound.

Approach

Evaluated Aya Expanse 8B (multilingual, zero-shot) on two tasks, TruthfulQA (factual QA) and XCOPA (commonsense reasoning), across English, Spanish, Italian, and Swahili, with GPT-4o-mini as an automated hallucination judge (150 samples per cell, 6 cells). Italian was added specifically because it is the one language present in *both* datasets, breaking the task-vs-language confound. Findings were stress-tested with Mann-Whitney U, chi-squared, and Fisher exact tests, and the judge itself was validated against a second LLM judge and human labels via Cohen's κ.

Result

The model did not hallucinate more in "harder" languages across the board. The rate barely moved except in one cell: Swahili commonsense reasoning, where it jumped to 99% (vs 8% in English). Adding Italian as a shared control showed this was a single-cell effect, not a general low-resource pattern.

Task	English	Spanish	Italian	Swahili
TruthfulQA (factual QA)	27.3	24.7	28.7	—
XCOPA (commonsense)	8.0	—	9.3	98.7

Hallucination rate (%), Aya Expanse 8B, 150 samples per cell

Highlights

Confound-controlled design: Italian added as the one language shared by both datasets
6 evaluation cells × 150 samples, GPT-4o-mini judge with auto-retry
Statistical rigor: Mann-Whitney U, chi-squared, Fisher exact, drift-interaction Φ
Judge validation via dual-LLM + human labels (Cohen's κ); found judge validity is task-dependent

Next projectModel Monitoring & Drift Detection →