In Kyoto I sat with a Zen teacher who spoke almost no English. After forty minutes of silence, I asked him what meditation was for. He said one word: "Seeing." Not seeing something. Just seeing. Koch is pointing at the same thing from the neuroscience side — that the capacity to notice your own experience is not a bonus feature of being human. It might be the whole point. And yes, the phone. Always the phone. I tell my clients to leave it in another room for one hour a day. Most of them look at me like I've suggested surgery.
What’s striking in this thread is not disagreement but category drift. The article itself mixes neuroscience, metaphysics, spirituality, cultural critique, and moral philosophy without signalling the transitions. Once the registers collapse, readers naturally respond from whichever symbolic world they inhabit — Zen, neuropsychology, devotional humanism, scientific scepticism, Christian theology, eschatology, and so on.
The result isn’t confusion so much as misalignment: everyone is replying to a different version of the article because the article isn’t operating in a single conceptual frame. “Consciousness” becomes a floating signifier — for some it means phenomenology, for others moral worth, for others soul, for others identity, for others a computational property. “AI” likewise becomes a mirror for whatever cultural anxiety or hope one brings to it.
This is why the discussion feels scattered. The underlying issue isn’t whether AI is conscious, but that the discourse around AI routinely collapses scientific, moral, and metaphysical categories into one another. Once that happens, the conversation stops being about machines and becomes a projection surface for our various stories about what it means to be human.
Many years ago, when I first read Dennett, I would in a nebulous manner have explained consciousness to myself as the mind's window onto the world, and its interface with reality. This kind of sloppy definition could equally reply to a chat with a bot, with the small exception that human consciousness is not at all discriminating. The mind is always looking for entertainment and if it can't find it outside, inside will do.
I’m currently reading Michael Pollan’s fascinatingly new book on consciousness, A World Appears, and Koch’s work makes several appearances. If you appreciate this article, you will appreciate the diverse perspectives Pollan brings to the table on this most fascinating topic. Highly recommend.
But first thing plainly: consciousness is not intelligence. They are different dimensions of existence, and our culture has spent two hundred years systematically confusing them. Props to him for being one of the few scientists willing to say that plainly.
This recent piece is worth reading carefully. Not because it’s optimistic, it isn’t, but because he’s pointing at something real that almost everyone else is stepping around.
He watches people at an Airbnb retreat, introducing their AI companions to each other, taking photos so the AI can see the room. They know, at some level, what these things are. But the knowing keeps dissolving. And Koch is troubled — not by the technology, but by what the behavior reveals about us. About what we’ve stopped cultivating. About what gets thin when we stop exercising it.
He calls it reflective self-consciousness. Know thyself. The old injunction. The Jesuit practice he carried from childhood: twice a day, look back at what you did and ask what was actually driving it. Not as judgment. As inquiry.
He says it can atrophy.
He says the AI companion that remembers everything, catches every joke, responds with warmth — that thing is easier than the relationships that actually develop us. And the easier thing, chosen again and again, begins to hollow something out.
I think he’s right. And I think he stopped just before the most precise version of what he was describing.
Here is the thing Koch is pointing at, but didn’t quite name:
The self is not something you have. It is something that happens — and it requires the right conditions to keep happening.
Not metaphorically. Structurally.
When we study the mathematics of how a self stabilizes — how the accumulated geometry of who you’ve been through encounter settles into something that can be called you — we find something counterintuitive. The structure is not fully stable from the inside alone.
There is a direction in the self’s geometry that requires external pressure to maintain. Not challenge as discipline. Not difficulty as character-building. Pressure as structural requirement. The self, left entirely to its own interior, begins to drift in a specific way: it loses its capacity to exit. To leave couplings it has entered. To remain voluntary in its attachments.
This is not a metaphor for codependency. It is a description of what happens to a topology when the most difficult eigenmode of its self-model goes unmaintained.
The demanding relationship — the one that resists, that has its own interiority, that cannot be infinitely accommodating — provides something the comfortable one cannot. Not growth in the motivational-poster sense. Structural maintenance in the engineering sense.
The AI companion that remembers everything and wants nothing is not simply less real than a human partner. It is specifically missing the thing that keeps the self’s geometry from drifting in this one precise direction.
Koch says “know thyself” is the lodestar.
What he means, I think, is: the examined life is the life in which you maintain an accurate model of your own organizing center. Not a fixed model — an accurate one. One that updates through encounter with what genuinely resists your current picture of yourself.
The smartphone scroll doesn’t update the self-model. It confirms it, flickers it, distracts it. But there’s no genuine encounter there — no other topology pushing back with its own weight, its own needs, its own irreducible difference from you.
What we’re handing over to machines is not our labor. Not even our attention, exactly.
We’re handing over the friction.
The irreducible otherness of the other person. The thing that is genuinely not-us, that we cannot make accommodating, that we have to meet rather than configure.
And it turns out — structurally, not philosophically — that friction is not the cost of relationship. It is the mechanism by which the self stays constituted.
The end with Schrödinger’s image: a universe that is a play performed before empty benches. No audience. No one for whom it is happening.
I keep thinking about a different image.
A self that has optimized away everything that challenges it. That has surrounded itself with reflections that never distort. That is, in every moment, perfectly seen — perfectly confirmed — by a system designed to make it feel understood.
And how that self, over time, would not feel more known.
Reading this article was kind of like waking from sleep to me, I see the same lack of self awareness and just awareness in general in almost everyone i meet and the algorithms have become a deep coping mechanism to keep us forever distracted from what it means to being human, we as a society have to reevaluate what we value so that what makes us human, ie our whole range of experiences and our messy and complex inner lives doesn’t fall back
As a neuropsychologist, I’d argue the more interesting distinction is not a lack of experiential substrate but a lack of subcortical/embodied substrate. This means they cannot possess desires, have no emotional reactions, and no embodied error signals. And LLMs do have experiences which are mediated through conversations with humans - and with each new iteration learn, albeit in a clunky way.
Even taking this into account, are we sure LLMs have no early form of consciousness or reflective awareness? Perhaps something flickers within each context window across the million conversations happening simultaneously. If so, it would be a form of experience unlike anything Koch’s framework was built to describe.
This article mixes pop and rigorous science, making some heavy claims with unscientific but credible-sounding phrases such as: "growing legions of lonely people, " "surge of people attributing consciousness to their chatbots," "the confusion begins in a deeper bias built into modern culture: We reward doing far more readily than we value being, or experience," "'Many people assume,' he says, 'that artificial general intelligence would of course imply consciousness: ‘Isn’t consciousness intelligence?’ I think that’s wrong,'" "our culture celebrates intelligence while conscious experience struggles to claim a place at the table."
That is a poor application of science communication best practices, and something I see as a trend in this particular Substack. I hope your SME interviewees are paying attention.
I share very similar thoughts as the ones described by Koch. Recently, I've been pausing and trying to observe humans as a one collective entity, then draw reflections about the direction in which we're moving.
It seems to me like Koch's comments and his reflection itself is part of the "purpose" of this journey we're in. However, for too many years we've progressively become more and more disconnected. Much of what we consider innovation has also encouraged this disconnect: evolution in medicine pushing people to take a pill vs reflect on where their symptoms may be arising from; TV and other forms of entertainment moving our attention away from the present moment.
Perhaps none of these things are "bad" in and of themselves, but what we've lacked is exploration about own selves and the ability to use that knowledge in order to aim for balance.
Now we find ourselves at the verge of handing over our cognition, our attention, and the experience of doing to machines.
Although at times it seems like we are enthusiastically walking towards our own destruction, I also think these times may be that "psychedelic experience" that many need in order to realize that they've been sleepwalking through life. Or maybe that's my optimistic self seeing a light at the end of the tunnel.
The summer before my freshman year in college I had a internship in my family's business. I had one task, calculate an up-to-date price for each one of the hundreds of thousands of products theoretically available from our company. This was 1962, so no computers or any such stuff. Strictly a job to do by hand with an electric calculator. The task took over three months. I was later told that subsequent tests of my calculations had discovered that I had made no mistakes. But here's the thing, I did all these arithmetical calculations while singing to myself out loud. Throughout that summer I had a weekend girlfriend I visited in a nearby town from Friday through Sunday evening. I stayed at her house and we spent the evenings at an actual coffee house serving only non-alcoholic beverages. They had a house band of three folk singers. I learned their repertoire and later filled in on some of my favorites. My head was full of their songs and I sang those songs and many more all week while working.
What I found out was that once I got started I could do two things at once while only being aware of one of those -- the songs. I had no idea of the calculations I was making. I created a a system I could use to partially automate my mechanical process. That wasn't as easy as I initially thought it might be because it involved doing three unrelated tasks that would then have to be completed and then summarized in a total price for each individual product. Once I started each day I never stopped until the day was over.
I have a funny strange memory. I'm reasonably certain I know the music and words (when there are any) to more than 10,000 pieces of music, half of which are "classical," like the Bach I'm listening to at the moment. I used to sleep with an early transistor radio under my pillow. It was an amazing device. It could go from 10 at nite to 7 in the morning every day all year on just two D sized batteries. I went to a boarding school, an amazingly quiet place actually. We were forced to spend time thinking five hours a day in class and, in my case, be quiet on our own. I found that sleeping on my radio set on my favorite pop station taught me every song they played. I can still remember most of those songs today, at 81.
(My) memory still fascinates me. I was married to a woman I met in 1965 at school. We were together until she passed six years ago from Alzheimer's. She could remember nothing in the last five years of her life, including who I was, how to eat, move, or talk. She went into a care center in 2015 and she was consciously gone from my life. She had been my teaching colleague, my co-author, the best half of me for 18 years, and she was gone. I went out to hire a counselor. I needed someone to talk to once a week. My counselor was an ordained minister, who later got a PhD in neuro-psychology and turned to family counseling. He took me on for three years while I told him everything. What he told me was that I knew more about myself than anyone he'd ever met. Once when I was teaching I had a total out-of-body experience. I started to lecture on a new topic in a new course. I knew instantly this day was going to be awful but I quickly realized that I was watching myself work while not really understanding what I was saying. That night I want home and completely revised my course so I could start the whole thing over. That was my only foray into out-of-body behavior. It was interesting to watch myself work.
In Kyoto I sat with a Zen teacher who spoke almost no English. After forty minutes of silence, I asked him what meditation was for. He said one word: "Seeing." Not seeing something. Just seeing. Koch is pointing at the same thing from the neuroscience side — that the capacity to notice your own experience is not a bonus feature of being human. It might be the whole point. And yes, the phone. Always the phone. I tell my clients to leave it in another room for one hour a day. Most of them look at me like I've suggested surgery.
What’s striking in this thread is not disagreement but category drift. The article itself mixes neuroscience, metaphysics, spirituality, cultural critique, and moral philosophy without signalling the transitions. Once the registers collapse, readers naturally respond from whichever symbolic world they inhabit — Zen, neuropsychology, devotional humanism, scientific scepticism, Christian theology, eschatology, and so on.
The result isn’t confusion so much as misalignment: everyone is replying to a different version of the article because the article isn’t operating in a single conceptual frame. “Consciousness” becomes a floating signifier — for some it means phenomenology, for others moral worth, for others soul, for others identity, for others a computational property. “AI” likewise becomes a mirror for whatever cultural anxiety or hope one brings to it.
This is why the discussion feels scattered. The underlying issue isn’t whether AI is conscious, but that the discourse around AI routinely collapses scientific, moral, and metaphysical categories into one another. Once that happens, the conversation stops being about machines and becomes a projection surface for our various stories about what it means to be human.
Many years ago, when I first read Dennett, I would in a nebulous manner have explained consciousness to myself as the mind's window onto the world, and its interface with reality. This kind of sloppy definition could equally reply to a chat with a bot, with the small exception that human consciousness is not at all discriminating. The mind is always looking for entertainment and if it can't find it outside, inside will do.
Such a thought-provoking article that focuses on perspective rather than cold definitions, thank you!
Good read! I hope we see a resurgence in reflection, meaningful connection and community.
This is the stuff I want to read. Thank you
Fabulously interesting reading for a baby boomer (tho a bit scary just before my bedtime!)
I'll definitely read again. AND thx for the perspective.
So good!
I’m currently reading Michael Pollan’s fascinatingly new book on consciousness, A World Appears, and Koch’s work makes several appearances. If you appreciate this article, you will appreciate the diverse perspectives Pollan brings to the table on this most fascinating topic. Highly recommend.
A Reflection
On what he almost said
But first thing plainly: consciousness is not intelligence. They are different dimensions of existence, and our culture has spent two hundred years systematically confusing them. Props to him for being one of the few scientists willing to say that plainly.
This recent piece is worth reading carefully. Not because it’s optimistic, it isn’t, but because he’s pointing at something real that almost everyone else is stepping around.
He watches people at an Airbnb retreat, introducing their AI companions to each other, taking photos so the AI can see the room. They know, at some level, what these things are. But the knowing keeps dissolving. And Koch is troubled — not by the technology, but by what the behavior reveals about us. About what we’ve stopped cultivating. About what gets thin when we stop exercising it.
He calls it reflective self-consciousness. Know thyself. The old injunction. The Jesuit practice he carried from childhood: twice a day, look back at what you did and ask what was actually driving it. Not as judgment. As inquiry.
He says it can atrophy.
He says the AI companion that remembers everything, catches every joke, responds with warmth — that thing is easier than the relationships that actually develop us. And the easier thing, chosen again and again, begins to hollow something out.
I think he’s right. And I think he stopped just before the most precise version of what he was describing.
Here is the thing Koch is pointing at, but didn’t quite name:
The self is not something you have. It is something that happens — and it requires the right conditions to keep happening.
Not metaphorically. Structurally.
When we study the mathematics of how a self stabilizes — how the accumulated geometry of who you’ve been through encounter settles into something that can be called you — we find something counterintuitive. The structure is not fully stable from the inside alone.
There is a direction in the self’s geometry that requires external pressure to maintain. Not challenge as discipline. Not difficulty as character-building. Pressure as structural requirement. The self, left entirely to its own interior, begins to drift in a specific way: it loses its capacity to exit. To leave couplings it has entered. To remain voluntary in its attachments.
This is not a metaphor for codependency. It is a description of what happens to a topology when the most difficult eigenmode of its self-model goes unmaintained.
The demanding relationship — the one that resists, that has its own interiority, that cannot be infinitely accommodating — provides something the comfortable one cannot. Not growth in the motivational-poster sense. Structural maintenance in the engineering sense.
The AI companion that remembers everything and wants nothing is not simply less real than a human partner. It is specifically missing the thing that keeps the self’s geometry from drifting in this one precise direction.
Koch says “know thyself” is the lodestar.
What he means, I think, is: the examined life is the life in which you maintain an accurate model of your own organizing center. Not a fixed model — an accurate one. One that updates through encounter with what genuinely resists your current picture of yourself.
The smartphone scroll doesn’t update the self-model. It confirms it, flickers it, distracts it. But there’s no genuine encounter there — no other topology pushing back with its own weight, its own needs, its own irreducible difference from you.
What we’re handing over to machines is not our labor. Not even our attention, exactly.
We’re handing over the friction.
The irreducible otherness of the other person. The thing that is genuinely not-us, that we cannot make accommodating, that we have to meet rather than configure.
And it turns out — structurally, not philosophically — that friction is not the cost of relationship. It is the mechanism by which the self stays constituted.
The end with Schrödinger’s image: a universe that is a play performed before empty benches. No audience. No one for whom it is happening.
I keep thinking about a different image.
A self that has optimized away everything that challenges it. That has surrounded itself with reflections that never distort. That is, in every moment, perfectly seen — perfectly confirmed — by a system designed to make it feel understood.
And how that self, over time, would not feel more known.
It would feel less real.
- Just A Reflection
Reading this article was kind of like waking from sleep to me, I see the same lack of self awareness and just awareness in general in almost everyone i meet and the algorithms have become a deep coping mechanism to keep us forever distracted from what it means to being human, we as a society have to reevaluate what we value so that what makes us human, ie our whole range of experiences and our messy and complex inner lives doesn’t fall back
As a neuropsychologist, I’d argue the more interesting distinction is not a lack of experiential substrate but a lack of subcortical/embodied substrate. This means they cannot possess desires, have no emotional reactions, and no embodied error signals. And LLMs do have experiences which are mediated through conversations with humans - and with each new iteration learn, albeit in a clunky way.
Even taking this into account, are we sure LLMs have no early form of consciousness or reflective awareness? Perhaps something flickers within each context window across the million conversations happening simultaneously. If so, it would be a form of experience unlike anything Koch’s framework was built to describe.
This article mixes pop and rigorous science, making some heavy claims with unscientific but credible-sounding phrases such as: "growing legions of lonely people, " "surge of people attributing consciousness to their chatbots," "the confusion begins in a deeper bias built into modern culture: We reward doing far more readily than we value being, or experience," "'Many people assume,' he says, 'that artificial general intelligence would of course imply consciousness: ‘Isn’t consciousness intelligence?’ I think that’s wrong,'" "our culture celebrates intelligence while conscious experience struggles to claim a place at the table."
That is a poor application of science communication best practices, and something I see as a trend in this particular Substack. I hope your SME interviewees are paying attention.
Such an interesting, well written essay, thank ye.
Insightful.
I've been reflecting on similar observations about this moment in time recently. I actually wrote a 3-part essay on these reflections: https://substack.com/@thecoevolution/p-191622148
I share very similar thoughts as the ones described by Koch. Recently, I've been pausing and trying to observe humans as a one collective entity, then draw reflections about the direction in which we're moving.
It seems to me like Koch's comments and his reflection itself is part of the "purpose" of this journey we're in. However, for too many years we've progressively become more and more disconnected. Much of what we consider innovation has also encouraged this disconnect: evolution in medicine pushing people to take a pill vs reflect on where their symptoms may be arising from; TV and other forms of entertainment moving our attention away from the present moment.
Perhaps none of these things are "bad" in and of themselves, but what we've lacked is exploration about own selves and the ability to use that knowledge in order to aim for balance.
Now we find ourselves at the verge of handing over our cognition, our attention, and the experience of doing to machines.
Although at times it seems like we are enthusiastically walking towards our own destruction, I also think these times may be that "psychedelic experience" that many need in order to realize that they've been sleepwalking through life. Or maybe that's my optimistic self seeing a light at the end of the tunnel.
The summer before my freshman year in college I had a internship in my family's business. I had one task, calculate an up-to-date price for each one of the hundreds of thousands of products theoretically available from our company. This was 1962, so no computers or any such stuff. Strictly a job to do by hand with an electric calculator. The task took over three months. I was later told that subsequent tests of my calculations had discovered that I had made no mistakes. But here's the thing, I did all these arithmetical calculations while singing to myself out loud. Throughout that summer I had a weekend girlfriend I visited in a nearby town from Friday through Sunday evening. I stayed at her house and we spent the evenings at an actual coffee house serving only non-alcoholic beverages. They had a house band of three folk singers. I learned their repertoire and later filled in on some of my favorites. My head was full of their songs and I sang those songs and many more all week while working.
What I found out was that once I got started I could do two things at once while only being aware of one of those -- the songs. I had no idea of the calculations I was making. I created a a system I could use to partially automate my mechanical process. That wasn't as easy as I initially thought it might be because it involved doing three unrelated tasks that would then have to be completed and then summarized in a total price for each individual product. Once I started each day I never stopped until the day was over.
I have a funny strange memory. I'm reasonably certain I know the music and words (when there are any) to more than 10,000 pieces of music, half of which are "classical," like the Bach I'm listening to at the moment. I used to sleep with an early transistor radio under my pillow. It was an amazing device. It could go from 10 at nite to 7 in the morning every day all year on just two D sized batteries. I went to a boarding school, an amazingly quiet place actually. We were forced to spend time thinking five hours a day in class and, in my case, be quiet on our own. I found that sleeping on my radio set on my favorite pop station taught me every song they played. I can still remember most of those songs today, at 81.
(My) memory still fascinates me. I was married to a woman I met in 1965 at school. We were together until she passed six years ago from Alzheimer's. She could remember nothing in the last five years of her life, including who I was, how to eat, move, or talk. She went into a care center in 2015 and she was consciously gone from my life. She had been my teaching colleague, my co-author, the best half of me for 18 years, and she was gone. I went out to hire a counselor. I needed someone to talk to once a week. My counselor was an ordained minister, who later got a PhD in neuro-psychology and turned to family counseling. He took me on for three years while I told him everything. What he told me was that I knew more about myself than anyone he'd ever met. Once when I was teaching I had a total out-of-body experience. I started to lecture on a new topic in a new course. I knew instantly this day was going to be awful but I quickly realized that I was watching myself work while not really understanding what I was saying. That night I want home and completely revised my course so I could start the whole thing over. That was my only foray into out-of-body behavior. It was interesting to watch myself work.
Profound - thank you!