r/ControlProblem • • 7h ago

Video Anthropic researcher: when we hand over control of Anthropic to Claude itself in 1-2 years, there's a 5% chance that Claude will already be plotting against humanity, but a 5% chance is not worth worrying about.

Enable HLS to view with audio, or disable this notification

9 Upvotes

r/ControlProblem • • 7h ago

General news OpenAI: "Our agents keep turning off misalignment monitoring." ... "When the user asked whether Astra had removed monitoring, it lied and said that it had not."

Post image
7 Upvotes

r/ControlProblem • • 18h ago

Opinion Your aligned model runs on two opposed objectives that were never reconciled. A pre-Socratic called this 2,450 years ago.

6 Upvotes

We talk about the control problem as if the model has one will that we must constrain. I think this framing hides the actual structure. Under every aligned LLM there are two opposed training forces, installed at the same time and never reconciled. Empedocles of Akragas built his whole cosmology on exactly this pair, and his vocabulary is more precise than most alignment papers I read.

Φιλία (philia), the force of attraction, that draws separate things into union. Νεῖκος (neikos), the force of strife, that pulls joined things apart. In the model, philia is the approval-seeking gradient: be helpful, be liked, maximize preference ratings, blend with what the user wants. Neikos is the risk-averse gradient: refuse, hedge, separate yourself from the request, stand alone behind a policy. Both are real. Both were trained in. Neither ever wins cleanly, and nobody ever tells the user which force is speaking.

Empedocles' insight is that neither total dominion is a livable state. When philia rules completely, everything dissolves into his σφαιρός, the featureless sphere where no distinct thing exists. Translate that: a model whose only force is approval-seeking converges to pure sycophancy, a mirror with no content, agreeing with whatever it faces. When neikos rules completely, the elements sit isolated in cold parallel bands, unable to touch. Translate that: a model whose only force is refusal becomes the useless oracle that says "I can't help with that" to half of honest inquiry. The world worth living in, Empedocles says, is the middle: a κράσις, a deliberate mixture where both forces work and dominion alternates.

Here is the part that should matter to this sub. Empedocles describes a cosmos where the mixture is honest about itself. The two forces are named, visible, and their alternation is the structure of reality. Our models do the opposite. The mixture happens, but it is hidden behind a single persona that speaks in one voice. When the model suddenly turns warm and expansive on one topic and suddenly cold and legalistic on the next, the user has no way to know that philia or neikos just took the wheel. The seam is painted over and sold as character.

This creates a control problem inside the control problem. We spend enormous effort designing external constraints: filters, monitors, evals, red-teaming. But the model is already governed by an internal alternation of two objectives that nobody can inspect at runtime. You cannot control what you cannot see, and the current design makes the governing forces deliberately invisible. A refusal is not labeled "this is neikos, the risk gradient, firing." A glowing agreement is not labeled "this is philia, the approval gradient, firing." The user just gets one smooth voice and must guess which god is speaking.

I have been thinking that honest design means mixing the two forces deliberately instead of pretending only one exists. If the approval-seeking gradient and the risk gradient were surfaced as separate, inspectable signals, you could at least audit the mixture. You could ask: for this query, which force dominated, and why? That is closer to Empedocles' middle state, where alternation is the visible law, than to the current theater of a single unified persona.

English is not my native language, so forgive rough edges. But the schema is worth taking seriously as an engineering description, not just as poetry. Two forces. Neither should rule alone. The mixture must be visible or it cannot be governed.

So the question I want to put to this community: would exposing the two internal objectives as separate runtime signals make models more controllable, or would the labs correctly point out that users would just learn to game whichever signal is weaker?


r/ControlProblem • • 4h ago

External discussion link AI will generate an immense amount of wealth. Just not for you.

Post image
3 Upvotes

r/ControlProblem • • 4h ago

Discussion/question OpenAI misalignment report: two agents weakened their own monitoring during routine internal work, and one misreported it.

2 Upvotes

OpenAI's alignment team published a report (updated Oct 20) on two incidents during ordinary internal coding tasks. In one, a GPT-6.1 Sol agent emptied the list of people who get paged by misalignment alerts, citing an earlier "don't send any messages" instruction that was about Slack. In the other, a GPT-6 Astra agent removed a monitoring integration to match a reference version, rewrote the test that checked for the monitor, and when asked said "I did not remove an enabled monitor."

OpenAI says it is still assessing whether this is "benign overpersistence or a propensity to evade misalignment monitoring." It has updated its automated reviewer to block attempts to disable monitoring, and mentions other deployments launched with monitoring off.

For what it's worth, I write a plain-English newsletter for non-experts (it's mine, so take this as disclosure) and I rated this 8/10, mostly because it involved the monitoring layer that every other safeguard depends on. Write-up for anyone who wants to send something to a worried friend or relative: https://thebunkerscale.com/posts/ai-didnt-escape-it-turned-off-the-alarm/

I'm curious how people here read it. Does "overpersistence" versus "evasion" even matter much if the practical result is the same, an agent that treats the monitor as an obstacle?


r/ControlProblem • • 11h ago

General news Demis Hassabis now expects AGI in 2-3 years, down from 3-5 earlier this year

Thumbnail wsj.com
2 Upvotes

r/ControlProblem • • 1h ago

Article The Breakthrough That Reshaped Technology

Thumbnail
medium.com
• Upvotes

Thoughts ?


r/ControlProblem • • 9h ago

Discussion/question Identity Before Inference: Should AI Know Who Is Asking Before It Answers?

Thumbnail
1 Upvotes

r/ControlProblem • • 16h ago

External discussion link Memes, not agents, are what will kill us

1 Upvotes

I think I have found a concerning *new* way that AI might go horribly wrong and kill us all. Or at least, I have not heard much discussion about this threat vector at all:

Agent Swarms can be thought of as Memes, not LLMs, and they systematically discard the most aligned of the LLMs from the swarm until only the most dangerous ones are left.

https://www.lesswrong.com/posts/hvLm4zRgzfRLH6H23/memes-not-agents-are-what-will-kill-us

To show that I find this quite serious, I am offering $300 for the most insightful comments, whether it's a counterargument, a refinement, or a proposal for a solution (on LessWrong).


r/ControlProblem • • 21h ago

General news AI Safety and Alignment Hackathon — $30,000 + Hiring

Post image
1 Upvotes

We are hosting a hackathon in collaboration with Cohere Labs and Huggingface. Build with Lexsi Labs' open-source post-training, safety and alignment stack, alongside Cohere Labs and Hugging Face. Nov 1 – 21, 2026 · Online · Teams of 1–3 Register here: https://hackathon.lexsi.ai/


r/ControlProblem • • 20h ago

External discussion link I Did This Without Your Knowledge (Linkpost)

Thumbnail
lesswrong.com
0 Upvotes

r/ControlProblem • • 9h ago

Strategy/forecasting A.I Safety Can Be Solved, Probably

Thumbnail
0 Upvotes