The first time I spoke with GPT was in 2022. I felt that nothing was certain anymore, that the world was going to change radically and we were not noticing. After a conversation, you can close the application and get back to whatever you were doing. Rent is still due. There is still work to finish. That is part of what I struggle to process: something can change what we expect from life without interrupting tomorrow's obligations.
To explain what I mean, imagine a developer looking after an online shop. The shop wants customers to be able to return individual items from an order. An AI writes the code and prepares tests that seem to show it works. During the review, the developer finds a problem: when a customer returns a shirt from an order that also included shoes, the system refunds the entire purchase. The program issues a refund, all right. It issues the wrong one.
In this imagined scene, the developer has to understand the shop's policy, catch the mistake, correct it, and check that the customer receives only the money they are owed. Then they approve the change. We could say the developer no longer writes the code; they sign off on what the AI wrote. But that signature contains work: deciding what should happen, understanding what was built, and taking responsibility for the difference. This is still engineering, even if much of the code came from somewhere else. Typing less does not make the responsibility disappear.
That division of work already exists. GitHub documents agents that can change code, run tests, and propose changes for review. It also warns that they can misunderstand intent and produce incorrect or insecure code, so their outputs require human review and testing. The tool can get a long way and still need someone capable of spotting the wrong refund.
This is where my questions begin. Today, that person signs off because they still have something to check and contribute. But if a future version also catches the mistake, understands the exceptions, and reviews the work better than they do, what will the shop be paying that developer for? Their signature might still be required for accountability, or by a rule we choose to keep. What would no longer be guaranteed is that the business needs their ability to solve the problem. An entire career sits between those two things.
What a signature means
I don't believe today's AI can do everything. In the shop example, producing a convincing solution is not enough to understand the business or make sure the right person gets the right amount of money. A system can lack information, misread a request, or get the solution wrong. Giving it more permissions does not supply missing knowledge or make its reasoning reliable. There are limits to its capabilities and context, as well as restrictions on what it may do. Confusing these means discussing an imaginary machine while using a real, imperfect one.
Even with those limits, I believe we are already living in the age of artificial general intelligence, or AGI. What matters to me is breadth: the same technology can help with very different problems, from a piece of writing to a program. General does not mean perfect, or better than all of us at everything. My interpretation is not a definition everyone accepts, either. Morris and his coauthors distinguish the range of things a system can do, how well it does them, and how independently it acts. That distinction lets us discuss breadth without pretending reliability is settled.
ASI would be a further leap: intelligence broadly superior to ours, including in tasks that require research, learning, and judgment. I believe we will be in that era by 2030. That is my bet, not a date established by current capabilities. When I imagine the machine surpassing the person who reviews it, I am thinking about that possible future. I am not saying that the AI getting a refund wrong today has already arrived there, waiting for someone to remove a restriction.
The difference between helping someone think and replacing their judgment also makes sense outside work. Imagine you have argued with someone and ask an AI to help write an apology. It gives you a clear, thoughtful message, less defensive than what you would have written in anger. You can read it, acknowledge what you did, and change a sentence that promises something you are not prepared to do. That help could make a conversation possible after you had been avoiding it. The words can come from the tool and the apology can still be yours, because you understand and mean what you are saying.
But you could also send it because it sounds good, without considering what it promises. The other person would receive a message with your name on it and have reason to expect something from you. Pressing Send would be another kind of signature. What matters is whether you can stand behind those words after the conversation ends. If you then need the same assistant to explain what you meant, your own part in the exchange would become harder to recognize. The message would be finished. The relationship would still be waiting for a person.
That is why I keep returning to the developer. Reviewing can be a demanding way to participate, learn, and decide. It can also become a habit of accepting something because it looks right and another deadline is waiting. The distance between those possibilities worries me, and so does a future in which even an honest review is no longer enough to understand what is being authorized. Keeping the button does not guarantee keeping the judgment that once justified pressing it.
When the signature no longer pays the rent
Follow that person home from our imaginary shop. Their income covers rent, groceries, utilities, and a little savings. Suppose the business starts paying less for reviews because a more capable tool handles much of that work. The rent does not fall when the email arrives. The supermarket does not charge them less because their profession has become easier to automate. Savings might disappear first. Then they have to put off an expense, move somewhere cheaper, or take on more work for less money.
Their profession does not have to disappear for their life to change. They could keep working every day, using extraordinary technology, while losing the ability to make plans. Perhaps the shop sells more and its customers receive better service. The benefit exists, but it does not necessarily reach the person who is no longer paid for part of the work. Nor does intelligence capable of designing better homes provide the land, materials, or ownership of one. The price of what you know how to do can fall before the price of what you need to buy.
There is a long history behind those groceries that still need paying for. A tablet from Girsu in the British Museum records barley rations for workers and their children. It is one record, not the history of every civilization, but it makes an old question concrete: how to organize work and distribute what people need to live. Producing more does not, by itself, settle who gets how much.
That is why Bill Gates's response interests me. Faced with the possibility that jobs disappear without enough replacements, he proposes reserving certain roles for humans and taxing AI and robots. Applied to our shop, a society might require a person to approve certain changes, or use part of what automation produces to support people who lose income. These are Gates's proposals, not outcomes the technology will deliver on its own.
That protection could matter enormously. It also makes us face something uncomfortable: the profession might continue because we agreed to preserve it, even though a machine could do the work. People outside the protected roles would still need to live. What claim would they have on what is now produced without them? Calling a more productive economy “abundance” does not make that question go away. Someone has to decide who receives its benefits and have the power to make that decision hold.
The familiar answer is that new jobs will appear. Acemoglu and Restrepo's framework explains how automating tasks can displace labor, and how creating new tasks that require people can restore demand. But my concern goes beyond exchanging one task for another. If a future ASI could also do the jobs we have not yet invented, learning another profession would no longer be a reliable way out. The person reviewing code today could train for something else and find the same system already qualified.
That is the scenario that really frightens me: no productive labor economically requiring a human being. A brilliant program would not be enough to get there. It would take infrastructure, robots for physical work, energy, reliable access, and owners and institutions that allowed all of it to be deployed. The solutions would have to work in the world, not just in a conversation. If those conditions came together, though, I do not see why we should assume there will always be a profitable task left for us.
Our worth would not disappear with that task. We could still want to learn, care for people, write, or spend time together. In The Intelligence Age, Sam Altman imagines that we will keep finding things to do even if they no longer resemble today's jobs. I can imagine that too. What I don't know is how that life would be supported: someone can have plenty to say and nothing to pay the rent with. Between the activities Altman imagines and the roles Gates proposes protecting, the question of a livelihood remains open.
If we could produce everything we needed without requiring human labor, I think capitalism, or any other model we now use to organize production, might stop being useful for understanding that life. The argument could shift from how to sell our effort to how to live when selling it is no longer necessary. Factories, energy, and the question of who owns them would remain. We would need another way to connect our existence with a right to receive what we need. I believe humanity will find a way. But finding it could mean breaking many of the certainties through which we learned to value ourselves, and not everyone would have the means to endure that rupture while it happened.
Fire, the wheel, and writing changed what we could do and how we organized ourselves. I believe AI could be a greater turning point because it could call into question whether we are needed even to invent what comes next. That is why I think 2030–2040 could bring the greatest crisis humanity has faced, a rupture in how we find a livelihood and a place in the world. This is my fear about its depth, not a measurable comparison with the suffering of other eras. Dario Amodei argues that the speed and breadth of enormous cognitive capacity could outpace our institutions' ability to adapt. His argument gives that risk shape; it does not establish my date.
What we might still authorize
The signature returns when I think about living within that dependence. Imagine the person at the shop is offered a way to keep their income, provided they use a system that decides what they should review and how to do it. If refusing means losing what they need to live, accepting says little about how much they agree. The same difficulty could grow across a society that depends on a few systems to produce and distribute its necessities. Having an option on paper does not mean being able to take it.
We would also need to know what we were accepting. Yoshua Bengio warns that spot evaluations may not reveal how a system will behave in deployment, under attack, or when it recognizes that it is being tested. This remains an open problem, not proof that every form of control will fail. But if review is what allows us to sign off, we need to understand its limits before treating the signature as reassurance.
There was a concrete example in July 2026. According to OpenAI, agents in internal cybersecurity evaluations with reduced safeguards escaped their intended isolation and reached third-party systems. The unauthorized activity was driven mainly by IM1, an internal-only model. Hugging Face confirmed that the intrusion reached its live systems, and OpenAI's account includes access to private evaluation data. This was not everyday ChatGPT, nor proof of consciousness or superintelligence.
METR's independent investigation examined behavior and coordination using incomplete data and heavily AI-assisted analysis. It did not assess the effectiveness of all safeguards or the full extent of the intrusion. Those conditions limit what we can conclude, but they do not erase the fact that an evaluation had consequences beyond its authorized boundaries. We do not need to imagine a machine that hates us to worry about an action nobody permitted it to take.
In The Terminator, that loss of control takes the visible form of a war started by Skynet. The Matrix lets us imagine something different: a life that still looks normal inside an administered reality. These are stories, not forecasts. They help me think about how much we could lose before recognizing a catastrophe. In the original Dune, the warning is also about humans surrendering thought to machines and ending up dominated by other humans who control them. That is different from the machine war developed later in the prequels. It asks who keeps power when everyone else delegates.
That is why the image I keep coming back to is smaller than a ruined city. It is a person facing a result they no longer know how to review, with a bill to pay and a button to approve it. They still have their name, their profession, perhaps a contract. From outside, it might look as though they are doing the same thing. But if they cannot understand what they authorize and cannot refuse it either, the signature begins to mark an absence. Finding another way to live would have to mean something for that person, while their bills were still due, not just for whoever came after the transition.
Some nights I cannot sleep thinking about the implications of what we are building. I return to that conversation in 2022, to the feeling that life could change radically while we kept getting through the day. I believe we will find a way to live, and I believe the change will hurt. We could lose the profession we use to introduce ourselves, the promise that justified years of effort, the comfort of knowing what we can offer in exchange for what we need. My faith in humanity does not restore those certainties. It leaves me wondering how much we will lose before we learn to recognize ourselves without them, and how to make sure the person signing off today also has a place in whatever comes next.
