Source: Astral Codex Ten | Author: Scott Alexander | Published: 2026-02-13

Summary

Scott Alexander proposes a straightforward empirical test of AI capabilities: invite readers to submit questions, run them through Claude 4.6 Opus, and publish the unedited results. The premise is that many AI skeptics form their opinions based on encounters with free-tier chatbots or viral screenshots of AI failures, rather than direct experience with the best available systems. By committing to show first responses without cherry-picking, Alexander sets up the exercise as a credibility test — one where the AI either performs or it doesn’t, in full public view.

The most interesting design choice is the “sweet spot” Alexander defines for question difficulty. He asks readers to target questions that would take a competent human roughly an hour of Googling and spreadsheet work — hard enough to be meaningful, but not so esoteric that no one could verify the answer. This framing implicitly argues that the real value of current AI is not in replacing deep domain experts, but in compressing routine research and synthesis tasks.

Our Take

The piece is vintage Alexander: a deceptively simple setup that encodes a larger argument about epistemics and technology adoption. The underlying claim is that the gap between public perception of AI and actual AI capability is partly a product of access inequality — people who refuse to pay for premium tools end up judging the technology by its weakest representatives. Whether or not the subsequent Q&A results are impressive, the framing itself is a useful contribution. It asks readers to move from vibes-based AI skepticism to something more testable, which is a worthwhile nudge regardless of where one lands on the broader debate.

That said, the exercise has obvious limits. A curated audience submitting questions to a model configured with careful system prompts is not the same as real-world deployment. The results will show what AI can do under favorable conditions, which is informative but not the whole picture. Still, as a conversation starter about how we should evaluate these tools, it is sharper than most of what passes for AI discourse online.