LLM Evaluation & Hallucination Analysis

An LLM-as-a-Judge framework that measures faithfulness and catches hallucinations.

LLM-as-a-JudgeNLPLangChainOllamaEvaluation

View code ↗

Output

Claim-level hallucination detection: each claim is verified against the context and labelled supported / contradicted / not-found, with the LLM judge's reasoning.

Benchmark dashboard: faithfulness and hallucination rate across 8 curated test cases (green = faithful, red = hallucinated).

LLM-as-a-Judge faithfulness scoring: a faithful answer scores high, a hallucinated one scores low.

Entity hallucination check (NLP-based): flags entities in the answer that never appear in the source context.

Overall benchmark summary: mean faithfulness, relevancy, grounding, and hallucination rate across all cases.

Problem

LLMs hallucinate in distinct ways: contradicting the source, adding unsupported facts, or inventing entities. Measuring this rigorously, rather than eyeballing outputs, is hard.

Approach

An evaluation suite combining LLM-as-a-Judge metrics (faithfulness, answer relevancy, context precision) with NLP-based heuristics. Hallucinations are detected at the claim level and classified as intrinsic (contradicts context), extrinsic (adds unsupported info), or fabrication (invents entities). Self-consistency checks across multiple samples and entity-level verification add robustness.

Result

A reusable benchmark that scores response grounding and flags hallucination type per answer: the evaluation backbone that makes a trustworthy RAG system measurable rather than anecdotal.

Highlights

LLM-as-a-Judge: faithfulness, answer relevancy, context precision
Hallucination taxonomy: intrinsic / extrinsic / fabrication
Self-consistency + entity-level verification
Curated benchmark test cases for repeatable scoring

Next projectCross-Lingual Hallucination Drift in LLMs →