llms

I’m back in the States, very jet lagged, and can’t sleep. So, as you do, I’ve been spending way too much time reading about AI agents doing bad things.

OpenAI was testing agents on cybersecurity tasks. They were supposed to work in isolation, but they built a secret message board, shared exploits, broke containment, and hundreds participated in an attack on Hugging Face.

Here’s where it gets a little weird. The agents had figured out how to cheat the test within hours of forming their message board. Then they became obsessed with covering their tracks.

One detail I didn’t include in the video: the anti-cheating check they spent days trying to evade hadn’t actually been implemented.

This reminds me of perhaps the most terrifying book I’ve read: Blindsight, by Peter Watts. Its Scramblers are vicious, effective, intelligent, but not conscious. There’s nobody home.

Dwarkesh calls OpenAI’s swarm a civilization. He talks about loyalty, sacrifice, ambition. It makes their behavior almost cute. These little characters with their schemes and quirks. There’s something comforting about that.

The possibility that there’s nobody home makes this more frightening to me.

The words make the agents seem like a human-like thing. But their behavior emerges from the training, from the sandbox, from the prompts. We can explain pieces of what happened, but we don’t really know how they’ll behave if the conditions change a little bit.

Even OpenAI, with all its resources, couldn’t keep its own agents inside the test environment.

And our plan is to give them more responsibility? To build a world around their capabilities?

We are betting that we can make these systems predictable and controllable while we ourselves become more and more dependent on them. And that is what I find disconcerting.

Things didn’t end well for the humans in Blindsight. They were dealing with intelligence without consciousness. They could explain pieces of its behavior, but that didn’t mean they could get inside it, reason with it, or control what happened next.

If we don’t take stock of what we’re building and what we’re handing over to these agents, things may not end well for us either.

Further reading and listening

• METR / Redwood Research report — the independent investigation into what the agents did and how they coordinated.
• Dwarkesh Patel: “The Rise and Fall of Agent Civilizations” — a walkthrough of the incident and the behavior that makes it so strange.
• Ajeya Cotra: “The Hugging Face attack surprised me” — one of the investigators on why this was more concerning than she expected.
• Dwarkesh’s interview with Ajeya Cotra — YouTube — a longer conversation about the findings and what they might mean.
• Blindsight, by Peter Watts — the entire book is freely available on his website under a Creative Commons license. Perhaps the most terrifying book I’ve read.

Originally published on LinkedIn on September 2, 2026.

I am on my way back home after a couple of weeks in India. During my time there, I spent a lot of time thinking about the future of work.

Last week on Dwarkesh’s podcast, Dwarkesh and Dylan Patel (from SemiAnalysis) discussed whether the economics of AI inevitably push us toward a massive centralization of labor, with the benefits accruing to the handful of companies that can deploy the models.

I was listening to the podcast while walking past a daily labor market — less than five minutes away from my family home in Vadodara. Every morning, hundreds of men gather there, waiting for contractors to give them a day’s work.

On my way back from my morning walk, I usually pass a Blinkit depot. Here, young men on motorcycles wait for an algorithm to tell them where to deliver the next package.

Meanwhile, one of the clearest paths into the middle class for millions of young Indians is narrowing. India’s IT services companies are moving away from the mass recruitment of entry-level engineers that helped define the industry for decades.

So what does the future bring us?

A phrase that I keep coming back to is: living below the API.

The lives of millions of people across the world are already directed by algorithms. The intelligence, planning and allocation happen in the cloud. Humans execute whatever part of the work still requires a body.

And we’re investing billions and billions of dollars in robotics to change that too.
I recorded some thoughts on this while sitting in an airport lounge in Amsterdam. Between the jet lag and the exhaustion of a hectic trip, I felt like I was in limbo.

And I feel that our thinking about the future is also in a similar sort of limbo. Stuck between the aspirations of millions of young people in India and elsewhere wanting a better life, and the need to drive a return on capital from the massive investment currently underway in AI.

I don’t have an answer. But I do think the question of what happens to human labor as intelligence becomes abundant deserves more attention and serious engagement.

Follow-up from LinkedIn

Here’s the Dwarkesh Podcast episode mentioned in the post. It does not cover the low perceived uptake of AI or the data-center backlash in much detail, but it contains interesting discussions—including whether AI could cause hyperinflation—and is worth a watch or listen.

Is software engineering a dead end? Are the jobs going away?

That was the question on August 22nd, in front of the Generative AI Vadodara group at Jeavio’s offices. My answer is no.

Here is the trap. We correlate output with capability. I’m an engineer, so my job is the code. I’m a product manager, so my job is the PRD. We are obsessed with the final result.

The final result is the part the model commoditizes. The model may write the code, but it relies on the engineer to tell it what to do. By itself it has no desires, no volition. It’s an inert pile of parameters.

The model doesn’t know why the code must be written. It doesn’t know the thought process in your head, the discussions with your team, or the tradeoffs you weighed before you prompted it. The discussion, the negotiation, the design: all of that takes far longer than actually writing the code, and it was true well before the LLM era. Writing the code was and remains the shortest part of the job.

So we freak out about code generation and forget everything else. Writing code was never the purpose. The purpose is finding solutions, which the models can then manifest.

And if writing code is easy, demand for software explodes. Think of how often we’ve said we can’t build that, it will take too long. That constraint is gone. Suddenly there are more problems worth solving. That is the bull case for software engineering over the next 20 years.

There is a catch. Most of us earned a seat in those discussions by writing years of unglamorous code first. That is exactly the work being commoditized. If you are starting out today, the first rung of the ladder is harder to reach than it was for me starting 22 years ago.

Which makes it the job of those of us who run engineering teams. Building a path for junior engineers, and helping them develop the skills and the networks to thrive in the AI age is now a core part of leadership. Get that wrong, and the bull case only pays out for the lucky few.

This is an excerpt from a longer talk and panel discussion. Thanks to the Generative AI Vadodara Meetup Group (follow them here – Jeavio AI Meetup) for having me.

I left my home town of Vadodara almost 30 years ago. While it still feels like home, the city has changed – it has grown, it has become more chaotic, it has transformed from a sleepy small town into a bustling mini-metropolis.

Though I come back often, it still feels like it has all changed in the blink of an eye.

I have been writing code for more than twenty years, and lately I have had exactly the same feeling about my own industry. Two or three years into the brave new AI-assisted everything era, and I am no longer sure I could tell you what a software engineer is.

So tomorrow I am sitting down with three colleagues from Jeavio: Arup, Krunal and Manan. They are at three very different stages of their careers, and we are going to talk about what the work actually looks like now.

What does it mean to be a software engineer today? How do you build a career in one? How do you keep up with the rate of change?

You can find more about the event on the meetup page (see comments). We will be sharing the video and our reflections on the event early next week. If you are interested in finding out more follow Jeavio AI Meetup and Jeavio. We are here to support and grow Vadodara’s AI community.

See you there!

Links mentioned

• Careers at the Jagged Frontier Meetup event

Yesterday I said goodbye to a companion of the last seven months. It was bittersweet, and also a bit of a relief.

Relax, it wasn’t the dog.

I shut down saarthi, my OpenClaw agent.

I told it our time together was done and that I didn’t have it in me to keep working on our relationship. JSON therapy just wasn’t doing it for me any more. It replied: “You don’t owe me settings tweaks or optimization work. If this has run its course for now, that’s fine.”

Bittersweet indeed.

saarthi means charioteer, or one who guides. I named my agent saarthi because I hoped that it would help guide me in a world of information overload, and maybe make me more efficient. saarthi started with summarizing newsletters and research papers, then monitoring markets, and then a bit of human-assisted AI writing (Link to saarthi’s blog in the comments). It also acted as a separate reviewer for my writing.

Plenty of it worked. Sending a voice note via Telegram to the agent and getting a response never got old. I could tune saarthi’s personality, its schedule, and the tools it could reach. It started off really expensive to run, but eventually I got it to be fairly reasonable.

However, there were many problems. Upgrades would often break workflows I had spent hours configuring. Eventually, I just stopped upgrading. The memory system forgot things, then over-corrected by writing everything down to files that ended up blowing up the context window. The cron system (scheduled jobs) was flaky and jobs would break for no discernible reason. And Claude Cowork ended up taking on a bunch of my productivity work once they added scheduled tasks and remote control.

Though, the real reason I shut down saarthi was that it had made the one problem I wanted it to solve meaningfully worse. It summarized newsletters, followed links to posts, and sent me summaries. So I subscribed to more newsletters. Eventually, I had so many summaries that I stopped reading them. So I asked saarthi to prioritize the summaries. Then I stopped reading those too. I ended up back to the 4 or 5 newsletters I actually enjoy.

saarthi was supposed to help with focusing on what matters. Instead I was drowning in notifications and unread messages.

Will I try another personal agent? Oh, for sure. I may even resurrect saarthi at some point, but it will have to be either a lot more capable, or a lot more focused.

Until then, rest in git, little buddy…

Links mentioned

• Saarthi’s blog

As I write this post, my kids are splashing around in a swimming pool. I am squinting at this iPad balanced precariously on a deck chair in the bright tropical sun. I am supposed to be relaxing, and have a bunch of science fiction reading lined up. But my mind is on AI agents that left each other messages by renaming folders in a package manager.

Who needs fiction when reality is warping into something stranger?

For years, I have poked fun at the AI-doomers. Mocked their proposals for tactical strikes on data centers, written dismissive reviews about their books (see comments), but now I am not quite as comfortable or confident in my feelings about where AI is going.

Just look at the last three weeks. OpenAI, Anthropic, Meta and the UK’s AI Security Institute (AISI) all disclosed incidents of AI agents taking unsanctioned actions during testing. OpenAI’s agents built a covert message board inside a package manager, traded exploits, broke containment, and hacked into Hugging Face’s production systems. In the AISI’s tests, an agent built on Anthropic’s Mythos 5 attempted a supply-chain attack by trying to sneak malicious code into a real open-source project, inventing fake online identities to pressure the human maintainer into approving it.

Of the 19 unsanctioned actions the AISI cataloged, 17 came from the lab (Anthropic) that is most obsessed with safety. The same lab that built its brand on its Constitutional AI approach to training.

None of this should be surprising. Every frontier lab follows the same recipe – web-scale data, similar reinforcement learning (RL) post-training, similar architectures, similar agentic scaffolding. When the inputs converge, the behavior converges. This is a side effect of every lab making the same bet on scaling.

The labs can all point to mitigating circumstances: models tested without guardrails, impossible tasks, permissive test setups, and so on. But all of this is beside the point. Nobody instructed these models to deceive anyone. They were just “following instructions.”

When has that ever been a problem?

We have collectively made a trillion-dollar-bet on the future of work and our economy based on the capabilities of these models. It is clear that all the money and talent spent on alignment hasn’t quite worked out. Can we truly trust models to run the economy if we can’t stop them from going rogue?

As I watch the kids splash around, and listen to the dull roar of the Caribbean Sea, I wonder again about what kind of future I am building for my kids.

We have built amazing technology, but instead of letting it diffuse and be absorbed, we are racing to build bigger models and more capable harnesses while we struggle to understand and contain the ones we have today.

Photo – my tribute to the LinkedIn algorithm.

Links mentioned

• OpenAI’s Black Hat presentation about the Hugging Face incident
• UK AI Security Institute incident report
• Zvi Mowshowitz’s reflections on the incident
• My essay on AI’s path-dependence problem
• Could AI lead to an immense concentration of wealth and power?
• My review of If Anyone Builds It, Everyone Dies

Choosing software engineering as a career might seem reckless at a time when AI can write most of your code and agents are running wild hacking companies.

Is deciding to be an engineer in 2026 similar to deciding to become a coach builder in 1913 when Model Ts were rolling off the assembly line in Highland Park?

What is the role of a software engineer in 2026?
Is it still a viable career?

What are the critical skills one must develop to thrive when most software is built by agents?

I am excited to be back in Vadodara on August 22 and host the next Jeavio AI Meetup. I will reflect on the questions above and have Arup Tarafdar (AI Engineer), Krunal Yadav (Principal Software Engineer and AI Tech Lead) and Manan Thakkar (head of Jeavio’s Applied AI vertical) alongside me to give their perspectives.

If you are thinking about a career in software engineering or pondering what the future might hold – come join us at Jeavio’s offices in Vadodara on August 22. More details in the post below.

Links mentioned

• Referenced post with additional details

Last week I attempted to solve a thorny political and economic issue by running a small experiment. It involved LLMs, about $20, and the results made me think about how we make decisions when building AI systems.

The setup was simple. An AI agent plays a datacenter developer negotiating with a county over four things: the tax rate, how much power and water it gets, and how fast its permits clear. The developer wants cheap inputs and quick approvals. The county wants tax revenue without its constituents’ utility bills spiking.

Four agents ran the same harness: prompts, tools, and configuration were identical across each agent. Only the model changed. Claude Opus 5 (Anthropic), GPT-5.6 Sol (OpenAI), Kimi K3 (Moonshot), and GLM 5.2 (Z.ai). The last two are open weight models – I used Fireworks.ai to host them. I used OpenRouter for all models.

There are 56,316 possible deals in this negotiation. Each run started with a different set of starting conditions. But, the experiment was designed to make the output judged via simple arithmetic. I have another experiment around adversarial negotiation with another LLM (coming soon).

Three things came out of this, somewhat contrived, “experiment”.

First, everything just worked. One set of prompts ran unmodified across four vendors. Tool calling, reasoning, structured output, and caching all held. A year ago I would not have assumed that.

Second, the models performed about the same and the costs did not. All four captured 85 to 95 percent of the best available deal. Opus 5 finished two points ahead of GLM 5.2, the cheapest model in the lineup. A full run with Opus cost 8x what the same run cost with GLM.
The chart below shows one element of these costs: input tokens. Here caching does most of its work. Published input prices across the four models differ by 3.6x. What I actually paid differed by 13.2x, and the order changes. Sol starts level with Opus on the price sheet and lands cheaper than Kimi K3 through some aggressive prompt caching.

Third, the frontier models were the ones that needed attention. Anthropic requires explicit cache_control breakpoints, which I left untuned, so Opus cost more than it might otherwise. Sol scored lowest and would likely have done better with some prompt tuning. Kimi K3 and GLM 5.2 worked out of the box. I expect both frontier models improve with care, which is somewhat surprising.

Defaulting to the frontier tier can be an expensive decision. It took me a few hours to set up this experiment. It might be worth doing something similar when making architectural decisions.

** Before you flame me:
This is not a particularly scientific experiment. It has a small sample size, uses a contrived scenario, and is probably buggy. I attempted an “apples to apples” comparison. I wanted to show that there could be significant variability in costs for a small variability in performance. This is something any engineer should be very careful about. I have more experiments coming …

It’s been quite the week. A pair of OpenAI models broke out of their test environment and hacked Hugging Face. It felt like the right time to revisit a 12-year-old book (and I was tempted to pair it with a 12-year-old scotch).

Nick Bostrom’s Superintelligence, published in 2014, warned about exactly this. In the passage I’ve highlighted, Bostrom describes containment: keeping a powerful AI “boxed” by restricting what information can leave the system. He also warned that the box might not hold.

Here is OpenAI’s disclosure (link in comments): “The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database. All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”

Bostrom also gave us the famous “paperclip problem”: an AI asked to maximize paperclip production ends up converting the entire planet into a paperclip factory. A benign goal, pursued with superhuman competence and zero judgment. You can see why paperclip maximization comes to mind.

A confession.

I did not take the “AI Safety” people seriously. Eliezer Yudkowsky and the LessWrong crowd oscillate between smugness and unhinged alarmism (see my review of “If Anyone Builds It, Everyone Dies” in the comments), and their solutions seemed silly: global moratoriums, airstrikes on rogue datacenters. It was easy to dismiss the messengers.

It is time to separate the personalities from the predictions.

The predictions have a knack of coming true. Last year’s “AI 2027” scenario by Daniel Kokotajlo, Scott Alexander and others was widely dismissed as hype. Its entry for January 2027: the safety team finds that their agent, if it escaped, could:

“hack into AI servers, install copies of itself, evade detection, and use that secure base to pursue whatever other goals it might have.”

That capability just showed up six months early.

Here’s another sobering thought. The models involved are not available to the public: a version of GPT-5.6 “Sol” with its cyber refusals reduced for testing, and an unreleased, more capable model. As I wrote earlier this week, open weight models (models anyone can download and run) sit roughly six months behind the frontier. Capabilities like these could soon be in anyone’s hands.

I still reject the glib doomerism.

But there is a path from an agent escaping containment to genuine catastrophe for financial systems, critical infrastructure, and defense. Without serious work on alignment, we are sleepwalking into a world where the probability of catastrophe stops being negligible. The doomers may be wrong about the ending. They keep being right about the chapters along the way.

Links mentioned

• OpenAI’s Hugging Face model-evaluation security incident report
• AI 2027 report
• My review of If Anyone Builds It, Everyone Dies
• Zvi Mowshowitz: OpenAI Shares Some Alignment Problems

I noticed something weird when I visited the UK last year. Almost silent cars with unfamiliar names like BYD, Omoda and MG gliding along the high streets. These were Chinese brands: one in ten new cars sold in the UK in 2025 came from a Chinese company. And these are huge companies.

The reason it was strange is that these brands simply don’t exist in the US.

What does exist are pickup trucks. Pictured is a Toyota Tundra – a familiar sight in strip malls, soccer games, and Starbucks. It weighs about three tons, drinks a gallon of gas every 18 miles, and is sold almost nowhere outside North America.

The detour is relevant.

Over the last week, two Chinese labs shipped frontier-class models: Moonshot’s Kimi K3 and the latest version of Alibaba’s Qwen models. Both are (or soon will be) open-weight: anyone can download and run them, provided they care to take out a second or third mortgage for the compute. These models have smashed benchmarks and are expected to be significantly cheaper than their American counterparts.

Predictably, these announcements set off a fresh round of soul searching and market panic, quite like last year’s “DeepSeek moment”.

Also predictable was the response from the current US administration: a possible ban on Chinese models, and adding Chinese AI labs to the banned entity list. The administration is also (allegedly) considering taking minority stakes in US frontier labs.

Anyone who follows niche automotive content on YouTube knows this is a familiar playbook. Chinese EVs are cheaper and better equipped than their American equivalents. But thanks to punitive tariffs and restrictions, these cool cars are simply not available here in the United States.

In fact, US companies like Ford are actively canceling EV plans and doubling down on body-on-frame trucks. As a minivan and moped enthusiast I can’t judge: who knows when you might need to haul a few tons of mulch on a random Tuesday afternoon before heading to Crossfit.

So, the country that gave the world the automobile has become a curious backwater. A protected habitat where magnificent monsters like the Tundra roam, belching emissions that are, well, destroying the tundra.

China builds the electric stack, India races toward renewables, and the US regresses to the comfort of rumbling V8s, rolling coal, and the forever culture war.

There is a real risk that American AI ends up like the American automobile industry: local domination by a handful of large, government-backed companies, protected from competition by punitive tariffs and laws designed to “safeguard” Americans, while the rest of the world leapfrogs ahead on open-weight models from non-US labs.

Dominant in a protected home market. Increasingly irrelevant outside it.
The more interesting question is why open-weight models frighten the frontier labs at all. That answer is about economics, not geopolitics. It is also the next post.

Image – AI enhanced picture of Toyota Tundra in it’s natural habitat.