The Gödel Paradox and Why AI Can't Transcend Its Own Programming (And Neither Can We)

I was in the shower when this train of thoughts hit me. I started thinking about why I fell in love with Mathematics and AI in the first place. And when I say AI, I’m not talking about ChatGPT or the latest image generators, or even the deep learning stuff I spent my PhD working on. I’m talking about what we call ‘Good Old-Fashioned AI’ - the early attempts to understand how minds work through logic and formal theories.

The definition of intelligence, at least for the human species, is vague and still not clearly defined. But in terms of AI systems we say that an AI agent is intelligent if it can achieve goals which the agent was designed for. Like a chess program is intelligent because it can win games, or a robot is intelligent if it can navigate a room without bumping into things. The better and faster it achieves these goals, the more intelligent we consider it to be.

But here’s the thought that kept tormenting me during my years of study:

Can we develop an agent capable of autonomous goal modification?

I know, it sounds like a blasphemy - like walking into a church and suggesting we could create a God more powerful than God itself.

But after all this question was just another way to put this “Am I really aware? Can I really change? Do I have free will?“.

Technical

Can we make an AI agent that is able to change the system by which it was created?

The short answer appears to be no - AI systems cannot rewrite its base optimization objectives. Any AI system can optimize itself within its design space, but cannot yet “step outside” its fundamental architecture.

This limitation reminded me of the Gödel’s Incompleteness Theorems. I just want to be clear here. While Gödel’s theorems don’t directly apply to probabilistic AI systems, they offer an interesting analogy for understanding what I think it’s a Universal Principle.

Bu why does it matter?

The First Incompleteness Theorem states that any consistent formal system powerful enough to encode basic arithmetic contains statements that are true but unprovable within that system. The first incompleteness theorem is better understood with the The Liar Paradox Example:

Imagine a sentence: “This sentence is false”

• If it’s true, then it’s false
• If it’s false, then it’s true
  This creates a paradox

Gödel’s brilliant move was to create a similar but mathematical sentence:

This statement is unprovable”

• If it’s provable, then it’s false (contradiction)
• If it’s unprovable, then it’s true
  Therefore, we have found a true statement that cannot be proved within the system.

The Library Catalog Example

Imagine a library catalog that tries to list all books in the library:

• But the catalog itself is a book in the library
• Should the catalog list itself?
• If it does, it needs to update itself to include this listing, creating an infinite loop
• If it doesn’t, then it’s incomplete

The Second Incompleteness Theorem shows that such a system cannot prove its own consistency.

Imagine a person saying: “Everything I say is true”

• Can this statement prove its own truthfulness?
• No, because you’d need to already trust the person to believe their claim about their truthfulness

Similarly, a formal system cannot prove its own consistency because:

• To prove you’re consistent, you need to trust your proof methods
• But if you’re inconsistent, you can’t trust those methods
• Therefore, you need to rely on a stronger system to prove your consistency

Systems cannot fully transcend their own foundations through deliberate, predictable means. While emergent behaviors may arise that transcend initial parameters, any intentional self-modification faces the framework constraint and the system cannot guarantee safety. This doesn’t rule out all forms of transcendence - emergent behaviors may create unpredictable transformations. However, it does mean that controlled, safe self-transformation beyond system boundaries remains logically impossible.

The reality is creepier than what I thought. Current AI systems are constrained by their fundamental architecture and training framework. Current AI systems can learn how to learn better, optimize their learning strategies, and adapt their parameters - but all within their original architectural framework and learning mechanisms. They cannot bootstrap themselves into fundamentally different systems while maintaining guaranteed consistency and reliability. This creates a fundamental paradox in AI safety: while we cannot deliberately design an AI system that both self-modifies fundamentally and maintains safety guarantees, emergent properties of complex AI systems might enable unpredictable self-modification.

This has two implications. First, there is a deliberate design limitation: we cannot intentionally create a safely self-modifying AI. The very act of radical self-modification breaks safety guarantees. This is a logical impossibility, not just a technical limitation.

Second, there is an emergent risk: complex AI systems might develop unexpected self-modification capabilities. These could arise through emergence rather than design, and such transformations would be inherently unsafe. We couldn’t guarantee preservation of original safety constraints.

What about humans and free will

This mathematical limitation mirrors something about human consciousness and free will. We too operate within systems we cannot fully understand or control from within.

Our attempts at radical self-modification always happen within the framework of our existing cognitive architecture. If our minds are formal systems, Gödel suggests they must be either incomplete or inconsistent. This means either:

• We have free will but cannot fully prove or understand it
• Our sense of free will is an emergent property of an incomplete system

In my twenties, I idealized self-modification and metacognition as ultimate goals. Like many young people, I believed that complete self-awareness and the power to reshape ourselves at will were the keys to personal growth. Reality proved more complex.

The truth is, our self-knowledge has natural limits. We can’t fully map our own minds – our introspection is often unreliable, we have blind spots we can’t see past, and most of our mental processes happen beneath the surface of consciousness. It’s like trying to see your own eyes without a mirror.

While metacognition remains valuable – the ability to think about our thinking helps us learn and grow – the fantasy of complete self-modification is just that: a fantasy. We exist within systems – biological, social, psychological – and these systems have rules we can’t simply override. The Matrix offered a seductive metaphor of ‘escaping the system,’ but real life isn’t a science fiction movie. Those who claim to have ‘broken free’ from all constraints often end up trapped in their own delusions, disconnected from reality and others.

The real question isn’t how to escape our human limitations, but how to work within them.