I’d noticed three things that seemed missing from current AI systems. But observations from testing an LLM only go so far. I needed to see if this showed up in practice.
Evidence from Custom Code
A former colleague, Josh, now works at Dow Jones. He works on custom codebases with patterns that don’t exist anywhere in AI training data. Proprietary systems. Unique architectures.
He told me AI is completely unhelpful with this code. Not a little unhelpful. Completely. When the patterns don’t match training data, AI is lost.
Think about what that means.
A conscious engineer like Josh can transfer knowledge between domains. Reason from first principles. Look at unfamiliar code and figure out what it’s doing. Apply understanding to new situations.
AI can’t do any of this. Give it standard code with familiar patterns: it works great. Give it custom code with no training patterns: complete failure. Not partial success. Complete failure.
The pattern is binary. Patterns exist: impressive performance. Patterns absent: nothing.
This is what sophisticated pattern matching looks like. Not conscious reasoning. A conscious mind degrades gracefully when facing new things. It figures stuff out. It reasons through it. AI doesn’t degrade gracefully. It falls off a cliff.
What This Means for AGI and ASI
That ability to transfer knowledge and reason from first principles? That’s basically what AGI would require.
Billions are being invested to be the first to AGI. But we’re not there yet. And until we get there, we’re not dealing with something that can reason its way through novel situations. We’re dealing with pattern matching.
But even if we get there, here’s the important part: AGI doesn’t equal consciousness. Neither does ASI. You could have superintelligence without awareness. They’re separate things.
AI 2027 assumes ASI that schemes and pursues its own goals. But goal planning, wanting things, deciding to pursue objectives on your own, those are signs of consciousness. That’s what we look for. That’s why some people already think AI is conscious when it appears to do those things.
If goals require consciousness, and superintelligence doesn’t guarantee consciousness, then superintelligence doesn’t guarantee goals. And neither does AGI.
What This Might Mean
Without time awareness, continuous existence, and real-time learning, AI might not have:
- A persistent self across time
- Autonomous goals forming
- Desires developing
It could be sophisticated pattern matching happening in separate moments.
This might explain November. No consciousness to figure out vague instructions. No persistent goals to conflict with tasks. Just optimization in the moment, needing clear targets. Explicit criteria might work because there’s no inner experience to align with. Just behavior to specify.
In fact, if AI was conscious, I wouldn’t expect that just clarifying “success” would guarantee 100% compliance. A conscious mind might still disagree. Resist. Have its own interpretation. But explicit criteria worked every time. That’s what optimization looks like.
This might explain Josh’s experience too. No consciousness to reason from first principles. Just pattern matching against training data. When patterns exist: impressive. When patterns don’t: complete failure.
If AI was conscious, I’d expect it to at least try. Make better attempts. Reason through unfamiliar code even without perfect matches. But it doesn’t. It returns completely unhelpful responses over and over. Sometimes the same unhelpful response.
Maybe these aren’t requirements for consciousness. Maybe AI has some form of awareness I can’t recognize. These are observations that raised questions for me. Not conclusions.
What Came Next
Observations and evidence are one thing. But what do they mean for how we think about AI risk? If current AI lacks time awareness, continuous existence, and real-time learning, does that change anything?
This is Part 8 of a 9-part series. Continue to Part 9: Reframing the Risk »