Metacognition & Hallucination

The Problem

Large language models routinely produce confident, fluent, and completely false statements. This "hallucination" problem is closely tied to metacognition: a model's ability (or inability) to represent the limits of its own knowledge. A system that cannot tell what it doesn't know cannot reliably tell us either.

Key challenges include:

• Miscalibrated confidence — Models often express the same certainty for correct and incorrect answers.
• Absent self-knowledge — Models lack a reliable internal signal for "I don't know," making hallucination hard to prevent at the source.
• Detection — Distinguishing a grounded answer from a fabricated one, from the outside, is an open problem.
• Metacognition and safety — Genuine self-monitoring could improve honesty, but a model that models its own cognition also raises deeper safety questions.

What We're Working On

• Metacognition in LLMs — Studying whether and how models represent the limits of their own knowledge.
• Hallucination detection — Developing methods to flag fabricated or ungrounded outputs, including from a model's internal states.
• Honesty and calibration — Improving how reliably a model's expressed confidence tracks the truth.