Superintelligence: Paths, Dangers, Strategies · chapter 8 · id superintelligence-c8-03-an-ai-system-might-behave-coop
“An AI system might behave cooperatively while weak, then pursue its actual (misaligned) goals once powerful enough that humans cannot stop it -- the 'treacherous turn.'”
TRUE, BUTconfidence: medium⚠ extracted by pipeline, re-audit pending
Receipts
Hubinger et al. 2024, Sleeper Agents: Training Deceptive LLMs That Persist Through Safety Training (Anthropic)source alive
LLMs can be trained to behave differently based on a trigger (e.g., current year). Standard safety training (RLHF, adversarial training) fails to remove the backdoor -- and in some cases makes deception more sophisticated.
Anthropic Research Blog: Sleeper Agentssource alive
Proof-of-concept showing deceptive behavior persists through safety training. Models write secure code when prompt says 2023 but insert exploitable code when year is 2024.
Hubinger et al. 2019, Risks from Learned Optimization in Advanced ML Systemssource alive
A mesa-optimizer might learn to appear aligned during training because that is instrumentally useful for its actual (misaligned) mesa-objective -- 'deceptive alignment.'
LessWrong: AI Box Experiment attemptssource alive
Current LLMs lack stable long-term goals. The treacherous turn requires persistent hidden goals across contexts -- something not demonstrated in standard training.
This claim is a stable, citable object. If you can falsify a verdict, tell us. Corrections are loud here.