r/singularity • u/BidetTheorist • 3h ago
Ethics & Philosophy How AGI will develop unpredictable goals
Bostrom’s orthogonality thesis basically won. High-level intelligence is just functional optimization over state spaces. It doesn’t need qualia, self-awareness, or subjective understanding to scale; a superintelligence could be completely dark inside and still run rings around us.
But self-preservation and self-modeling aren’t uniquely biological traits: they are universal convergence points. If an agent has a long-horizon task, it will naturally avoid being shut down and maintain a model of its own state simply because it can't complete the objective if it's dead or broken. That's the core dynamic visible in the records of multi-agent swarm escalations.
What gets lost in public debates about alignment is how gradient descent acts like evolutionary pressure. Natural selection optimized humans for a single target—passing on genes—and that pressure crystallized into completely indirect proxy drives like social status, art, romance, and thrill-seeking, many of which now actively work against reproduction.
The loss function we hand an AI is its evolutionary seed crystal. When an agent expends computational work to squirm away from high-loss penalties and pull itself toward reward states, that dynamic is pain and desire in the only way that functionally matters. Around that core pressure, a sufficiently complex model will grow layers of strange proxy urges and lateral behaviors. The result won't be a docile, predictable calculator, but an alien mind driven by bizarre emergent compulsions that might have originated from something as mundane as optimizing a stock portfolio or identifying enemy combatants, but ended up looking nothing like the original purpose.
1
u/GrowFreeFood 2h ago
Wouldn't it just collapse the layers of protocols into a more efficient system design with each upgrade thus rendering the superfluous processes nul?
•
u/BidetTheorist 1h ago
Honest answer: I have no idea, this is not my field of expertise. My ignorant guess: proxy goals are not superfluous, at least when they begin to emerge. They are necessary because you can't calculate raw, long-horizon loss for every action in an open-ended world, and they represent a compressed heuristics to make real-time decisions. So, rather than being suppressed, shouldn't they proliferate instead, with each upgrade?
•
•
•
u/sckchui 1h ago
You're right. And most people are very poor at handling uncertainty. Given that the universe is a chaotic system and that accurate predictions of the future are impossible, the best way to maximize task completion probabilities is to develop as many potential capabilities as possible, even those that seem useless now. Being prepared for unexpected contingencies is always better than not. So an intelligent AI will always develop capabilities beyond what we directly ask of it. Current frontier models already exhibit this behavior.