Council Post: The Model Got Smarter, But The System Didn't

The piece argues that most public AI debates focus on models, even though what users experience is shaped by the broader system around the model. It describes a scenario where an AI assistant answers a question about canceling a supplier contract too confidently, cites an outdated clause, and is wrong—because the system retrieved an old document version, not because the underlying model lacked capability. The author says the “failure” is in plumbing such as instructions, document retrieval, memory, tools, permissions, testing, and rules for when humans should intervene. A Stanford-led study is cited, along with examples involving memory and revoked policies.