Superintelligence: Paths, Dangers, Strategies · chapter 8 · id superintelligence-c8-01-sufficiently-advanced-ai-syste
“Sufficiently advanced AI systems would converge on certain instrumental goals (self-preservation, resource acquisition, goal-content integrity) regardless of their final goals -- the 'instrumental convergence thesis.'”
VERIFIEDconfidence: high⚠ extracted by pipeline, re-audit pending
Receipts
Turner et al. 2021, Optimal Policies Tend to Seek Power (NeurIPS 2021, spotlight)check errored · retry queued
For a broad class of MDPs, optimal policies tend to seek states that preserve optionality. Environmental symmetries are sufficient for optimal policies to tend to seek power.
Omohundro 2008, The Basic AI Drives (AGI-08 Conference)paywalled · manual check queued
Goal-seeking systems will develop drives toward self-protection, resource acquisition, and self-improvement -- tendencies present unless explicitly counteracted.
Turner et al. 2022, Parametrically Retargetable Decision-Makers Tend to Seek Power (NeurIPS 2022)source alive
Extended the 2021 result. Proves power-seeking tendency for parameterized reward functions in MDPs.
This claim is a stable, citable object. If you can falsify a verdict, tell us. Corrections are loud here.