Model Evaluation
Systematic methods for measuring AI capabilities and detecting dangerous behaviours before deployment. Covers benchmark design, red-teaming, uplift evaluations for weapons and cyberoffence, and the challenge of eliciting capabilities that models may conceal.
Viewpoints
Greenblatt & Shlegeris: Control framework requires dangerous capability evaluations as trigger
Buck Shlegeris, Ryan Greenblatt
“Control protocols are designed to apply during a specific window: after dangerous capability evaluations start triggering (indicating models have become powerful enough to be dangerous) but before AI systems become too powerful for control measures to remain viable. During this controlled period, the strategy involves continuously evaluating dangerous capabilities, testing and implementing control protocols, using AI systems for useful work while they remain roughly human-range, and simultaneously developing solutions for handling more capable future systems.”
Key Moments
Karnofsky: Taxonomy of capability vs. alignment evals and meta-capabilities
Holden Karnofsky
“Model evaluation should be divided into capability evaluations (testing whether AI can perform dangerous tasks like bioweapon design or autonomous replication) and alignment evaluations (testing whether AI follows intended goals or has misaligned objectives). A critical subcategory is "meta-capability" evaluations, which assess abilities that make other capabilities hard to evaluate—such as unauthorized proliferation, where an AI helps humans build unrestricted AI systems that bypass safety controls.”
Dalrymple: Capabilities evals harder to game than mitigation evals
David Dalrymple
“Capabilities evaluations are significantly harder to game than mitigation evaluations when training follows standard autoregressive patterns with cross-entropy objectives. Mitigation evaluations resembling questionnaires are easily gamed and ineffective, while capabilities evals maintain reliability under standard training regimes. Automated red- teaming using AI systems that can perform gradient-based searches for adversarial inputs offers more promise for evaluating whether dangerous capabilities persist.”
Avin: Building regulatory infrastructure through near-term AI risks
Shahar Avin
“Establishing government oversight of AI systems requires building regulatory machinery through near-term, concrete risks like privacy and bias. This creates an interface where regulators can demand information and conduct audits, red- teaming, and evaluations—infrastructure that can later be extended to assess more advanced, existential risks as concrete measures for those risks are developed.”
What people have said about Model Evaluation
- Chained tasks only fail when the base success rate is too low — Sholto Douglas, Trenton Bricken
- Evals must catch capabilities that hide other capabilities — Holden Karnofsky
- Measuring AGI takes a broad battery of tests against human baselines — Shane Legg
- Memorization isn't intelligence, and ARC-AGI is built to show it — Francois Chollet
- Regulating today's AI harms builds the machinery for tomorrow's risks — Shahar Avin
Powered by Symmerai — a living index of public discourse. Request early access →
Related concepts
Other relevant clips
34 - AI Evaluations with Beth Barnes
Beth Barnes
“introduction I mentioned that you worked for model evaluation and threat research or meter like what is what is meter Yeah so basically War would you know basic mission is have the world not be taken by surprise by dangerous AI stuff happening so we're uh we d”
34 - AI Evaluations with Beth Barnes
Beth Barnes
“…ion fair enough so I guess so you're the you uh co-lead the model evaluations and threat research um we've talked a bit about evaluations but there's also kind of threat models of what we're worried about ai's doing and what we're trying to evaluate for um can”
47 - David Rein on METR Time Horizons
David Rein
“…s basically where um uh you know >> uh basically yeah model models still are are just like highly highly comp you know cost cost competitive. I totally imagine this changing at some point. um uh you know uh yeah just just trends in models kind of uh you”
Are Anthropic's AI safety policies up to the task? | Nick Joseph
Nick Joseph
“…. Coming up with new risk categories and threat models is something that anyone can contribute to. Rob Wiblin: What are the places that are doing the best work on this? Anthropic surely has some people working on this, but I”
26 - AI Governance with Elizabeth Seger
Elizabeth Seger
“…r enough in the last three months I'd have it down external model evaluation oh right yeah um yeah so enabling like external oversight and evaluation and I think it depends which one you're talking about so for example if we're talking about maybe okay let's s”
Are Anthropic's AI safety policies up to the task? | Nick Joseph
Nick Joseph
“…sing more compute or data to train any given AI model. The idea for RSPs has been around for a couple of years, and I think it was fleshed out maybe after 2020 or so. It was advocated for by this group now called”
26 - AI Governance with Elizabeth Seger
Elizabeth Seger
“…ts of Open Source to to safety and distributing control and model control and model control and model evaluation but you know if you open source away below that threshold the the net benefits are net benefits are net benefits are great gotcha so my next questi”
28 - Suing Labs for AI Risk with Gabriel Weil
Gabriel Weil
“…ability insurance requirements um so I could see a role for model evaluations both in uh deciding uh how much coverage like what what the coverage requirement is like a regulator could use a set of like dangerous capabilities evaluations to um decide how much”