Interpretability

The Problem

Neural networks are largely black boxes: we see inputs and outputs, but the internal logic connecting them is opaque. This is a core safety risk, if we don't know why a model made a decision, we can't be confident it will behave safely in novel or high-stakes situations.

Key challenges include:

• Mechanistic understanding — Reverse-engineering the circuits and algorithms a model learns internally.
• Feature attribution — Identifying which parts of an input drove a particular output.
• Representation analysis — Understanding what large and self-supervised models actually encode, and how to use it.
• Scalability — Making interpretability methods work as models grow to billions of parameters, without drowning in noise.

What We're Working On

• Mechanistic interpretability — Studying small-to-medium models to find circuits that generalize to larger systems.
• Representation & self-supervised learning — Analyzing what learned representations encode and how they transfer across tasks.
• Activation steering — Using internal representations to monitor and control model behavior at inference time.
• Verification — Using interpretability to confirm a model reasons correctly rather than exploiting dataset biases.