The Spigot

I keep thinking about how we ended up here.

My kids are three and five. I have been in this industry for twenty two years. You would think that somewhere along the way I would have put together a coherent mental model of how the technology industry works, and make sense of what is happening. Something that I could wrap up as wisdom, or advice, or maybe just a pointer as my kids think about their future in a few years time.

Instead, I have been reacting. Lurching from one cycle to another. Mobile, Cloud, Crypto, AI and whatever comes next. As I navigate each cycle, like a befuddled tourist, the scale gets bigger. Each time, it seems that fewer people are thinking and the explanations for what is happening and why are more confused.

So I have been reading. Postman. Karen Hao’s Empire of AI on Sam Altman and OpenAI. Sebastian Mallaby’s The Infinity Machine on Demis Hassabis and DeepMind. Kyla Scanlon on the strange economics of the moment. And I have been trying to put together a through-line. The outline of a semi-coherent narrative that gestures towards why we end up here – arguing about data centers, fretting about AI, as the world literally burns.

I think it starts with a banner ad.

October 27, 1994

Bill Clinton was President, and Marc Andreessen had just released the Mosaic web browser just a year earlier. On that date, Wired Magazine’s digital spinoff, HotWired, launched its website with about a dozen paid advertisements. One of them was for AT&T. It was 476 by 56 pixels. It asked: “Have you ever clicked your mouse right HERE? You will.” Clicking it took you on a virtual tour of the world’s great museums.

The world’s first banner ad

By today’s standards, it was quaint. I suspect it was quaint even by the standards of 1994. But it was the beginning of something that would birth an enormous range of products and services: from social media to AI slop. But at the time, nobody involved thought they were the catalyst to the creation of a giant industry.

Within a year of HotWired’s launch, it was followed by Lycos, Excite, InfoSeek, and others – each company created with advertising as their primary business model. By the time Google created AdWords in 2000, the model was established. By 2014, Google alone was generating $60 billion a year in ad revenue. In 2025, Google brought in nearly $295 billion.

Nobody planned this. Andrew Anker, the former investment banker who wrote HotWired’s business plan, settled on advertising because it was, as an oral history of the era describes the only logical revenue stream he could envision. So, the humble beginnings of the business model that today underpins Meta,Google, and one that upstarts like OpenAI are staking their future on. An incidental decision, that became possibly the most profitable business model ever invented.

The same decision was also the inflection point from where technology went from being the domain of a handful of nerds and dreamers in a sleepy suburb of San Jose to the dominating social and economic force it is today. It is also where the trouble starts.

The Spigot Opens

Digital advertising became the canonical business model as the world went online. Technology meant zero marginal costs, global reach, and rapid dissemination. It generated staggering amounts of surplus capital concentrated in a handful of companies. Google, Facebook, Yahoo before them and many others to follow. The venture capitalists who were early investors in these companies generated mind boggling returns. Serving targeted ads at scale became a money printer.

All that surplus capital needed somewhere to go.

Capital chases returns. When you have that much money looking for a home, the bar for “this might work” drops dramatically. The entire Venture Capital business model is predicated on finding that one unicorn that would generate massive returns. You can see multi-billion dollar bets on businesses leasing office space for a loss while wearing the accoutrements of a technology company were justified.

For a VC, sitting on uninvested funds means your fund underperforms, your investors look to greener pastures. So you fund whatever has the ceremonial language of disruption. You fund NFTs, you fund fancy juice makers, because the alternative would be irrelevance.

The Juicero (RIP) – disrupting the fruit squeezing industry

The post-pandemic crypto boom seems like a fever dream now. Billions raised on the promise that pixelated pictures of digital apes would one day sublimate into collectible and unique art. Investors poured money into automated money-making machines based on complicated DeFi (Decentralized Finance) protocols that required a PhD in applied mathematics and the willingness to suspend disbelief.

I wasn’t immune. I spent way too much time trying to understand how Bitcoin works (and wrote a terrible science fiction story trying). I spent my hard-earned money on staking out some space on the blockchain. Money that could have gone to my kids’ 529 plans. Sorry girls. I wasn’t the outlier though. People way smarter than me poured millions into something that was, when you stripped away the jargon and the vibes, the world’s slowest and most expensive database.

How did this happen? How did so much capital get allocated (and is still allocated!) on something that is so obviously broken?

It is downstream of the spigot. There is so much money in the system, generated by that initial accident of online advertising, that it has to go somewhere. And the job of any ambitious entrepreneur is to provide it a plausible channel. Once the money starts flowing, it becomes self-reinforcing. More money validates the narrative. The narrative attracts more believers. More believers attract more capital. FOMO reigns supreme. The cycle runs until something breaks the spell.

Technopoly

Neil Postman wrote Technopoly in 1992, before the Internet was really a thing. I finished it a few weeks ago and it’s been rattling around in my head ever since.

Postman, a famously skeptical and uncompromising media critic, defined a technopoly as a society that has surrendered its decision-making to technology. Not in a “I welcome our robot overlords” way, but through a blind submission to metrics and statistics of dubious value. His examples – IQ tests, a man “drowning in a river that is, on average, four feet deep”. A society where technology becomes dominant through a self-perpetuating loop where it drives investment in itself, regardless of the impact on human-well being. Does this sound familiar?

Postman didn’t believe resistance was futile

I am also reading Sebastian Mallaby’s The Infinity machine. What comes through in his account of Demis Hassabis, Elon Musk, Larry Page and other industry titans is just how each of them operates with a profoundly different view on Artificial Intelligence – set to be the dominant technology of the 2020s and beyond. Hassabis comes across almost monk-like – viewing AI as the means to understand the nature of existence. Page as a transhumanist who sees the body as a shell to be discarded and the machine as the vessel for eternal life. Musk as a self-appointed guardian of a very particular vision of humanity – presumably with him as some sort of God-Emperor. These are the people who are deciding how and where the most consequential technology in a generation is deployed.

Consider the data center buildout. Trillions of dollars of private capital are being deployed to construct AI infrastructure at a pace that makes the railroad boom look modest. Even adjusted for inflation.

The rationale behind the investment seems to follow a circular logic. AI requires compute for training and inference, so you build data centers and fill them with compute. You fill them with compute which makes rapid deployment of AI possible and sparks massive competition between foundation model companies and hyperscalers to build, fundamentally, the same things. That in turn drives further demand for compute, and so on.

And AI is not just another speculative bet sitting alongside the economy. In 2026, it is the economy. Kyla Scanlon put it plainly: as AI swallows more and more capital, it has become the stock market and the economy simultaneously. The same companies – Microsoft, Google, Amazon, NVIDIA, etc. bankrolling the AI infrastructure buildout are also its biggest customers. Data center construction and investment in AI is driving GDP growth without growing jobs. Scanlon calls it a “jobless expansion“.

The prosperity exists in balance sheets and in the giant data centers sprouting up along state highways in places like Northern Virginia and Tennessee. It does not exist in communities that have been hollowed out as manufacturing evaporated and a career meant becoming the meat interface of a faceless algorithm directing you to the next gig.

Postman would have recognized this instantly. Nobody is in the driver’s seat. No single actor decided this was a good idea for society. No democratic process approved it. The technology cycle itself is driving societal change without any significant discussion. We have abdicated to the technology itself.

I wrote this in my notes on the book: “FOMO-driven investment in AI data centers with little to no prospect of broad societal benefits. We could have spent this money on climate change remediation or education or universal healthcare but here we are.”

Here we are.

The Arbitrage Trap

So there is a clear sense of a backlash brewing against technology. Try posting a pro-AI take on Threads or Bluesky and see the reaction. It’s not just the keyboard warriors who are ready to fight. A young man threw a molotov cocktail at OpenAI CEO Sam Altman’s house last weekend. Politicians who oppose moratoriums on data center constructions are threatened. There is a sense of rage as a new generation emerges into the post-ChatGPT landscape of disappearing knowledge work and the deep sense of careers and meaning being stolen by algorithms and trillion-parameter models.

But there is a key point that is missing in the public discourse. When Bernie Sanders “debates” Claude, it isn’t a meeting of equals. Bernie is an influential Senator and can call for moratoriums, hearings, and regulation. But Anthropic just raised billions of dollars in private capital. Claude runs not on the public dime, but on VC dollars. If Sanders pushed for a ban on data centers in Vermont, they will just build them in North Dakota. It doesn’t really matter where the data center sits. If the capacity is built out, compute will be deployed, and that compute will reshape the day-to-day work of people whether they want it or not.

Hope Bernie has a Claude Max plan

Jasmine Sun, who writes one of the sharpest Substacks on AI and Silicon Valley culture, spent time in DC and San Francisco earlier this year tracking what she calls “AI populism.” Her observation is stark: there is a widening chasm between the people who are building and funding this technology and the people who will live with its consequences. Politicians gearing up for the 2026 midterms are scrambling to design their AI agendas. Labor unions, environmentalists, social conservatives are all rushing to come up with a position.

But the backlash lacks a mechanism to break the capital flow. Moratoriums work only if they are coordinated and enforced. Otherwise you just get regulatory arbitrage. Capital exits to friendlier jurisdictions, and the places that resisted end up with neither the investment nor the jobs. That is the trap.

And if you want to see the arbitrage logic taken to its absurd conclusion, look no further than Elon Musk’s push for building data centers in space. As they say, in space, no one can hear your strident demands for a datacenter moratorium.

Spending a trillion dollars to deploy millions of GPUs in space seems insane, but it is also internally consistent with the incentive structure.

The Ratchet

The AI investment cycle can be best described as a ratchet. It moves only in one direction before locking in place. We seem to be committed to seeing where this cycle plays out – even if it ends in tears.

Sun points out the worst case scenario in a recent post – “One nightmare is a future where we get AI that’s good enough to wreak social and economic havoc, but not yet good enough to cure cancer / solve climate change / deliver 10% GDP growth. In that world… who pays?”

I work with AI every day. I run teams that build with it. I can see the utility. I have watched it compress weeks of work into hours and deliver real value for my clients. I am no old man shaking my fist at the clouds. But I also know that when the backlash arrives in full force, when the torches are lit and the pitchforks come out, the distinction between “I used AI thoughtfully” and “I profited from AI” will not matter. I worked in investment banking in 2008, I know what it means to be a social pariah.

People like me, who have bet careers on this technology being useful, will be caught in the same sweep as the people who bet billions on it being transformative. Everyone on the ratchet moves in the same direction. And yet, apart from the odd paper, there are little to no concrete suggestions from the same billionaires about how to make technology work for everyone apart from some vague gesturing to super-intelligence and to abundance.

And what feels like willful delusion rather than mere miscalculation is that the ratchet keeps clicking forward even as the world around it deteriorates. The United States, Israel, and Iran are in an active military conflict with direct strikes and counter-strikes. The Strait of Hormuz, through which a quarter of the world’s traded oil passes, is under threat. Russia’s invasion of Ukraine grinds on. These are the kinds of events that should be sending capital fleeing to safety. Instead, the markets shrug and carry just .. carry on?

Trying and Failing to Understand the World

I started this year realizing that I did not have a coherent mental model to explain what was happening. I read Postman, Hao, Sun and Scanlon to try and see how others made sense of a world so utterly dominated by technology and the eccentric billionaires who control it.

I ended up with a series of explanations that seem to involve some sort of hardware. Spigots, ratchets, and data centers in space. But I do not think I have a mental model. I do not think I can predict what comes next except a vague feeling that we will continue to spin faster until the whole edifice comes crashing down or we ascend to the singularity.

Postman’s views on the subservience of humanity to technology appeal to me because they seem to be manifest everywhere I look. From people scrolling aimlessly on their phones to pouring out their darkest secrets and deepest fears into the maw of a trillion-parameter language model. It is grim stuff.

But Postman also offers a solution. He calls it a “thoughtful rebellion.” Maybe the movement to touch grass, the surging sales of physical books and vinyl are signs that there is a genuine desire to disengage from digital technology. But how much of that movement is itself driven by mimetic desires pushed by algorithms, through BookTok and the like?

Perhaps technology provides its own means of meaningful disengagement. And maybe that is the only, if unsatisfactory, answer.

My kids will inherit a world shaped by decisions nobody consciously made, funded by a torrent of money nobody voted to spend, run on infrastructure nobody asked for. The best I can do is to show them the machinery that drives the world. And hope that they can find a way.


What if ASI Leads to Stasis?

I recently read and reviewed Nick Harkaway’s Titanium Noir, a noir detective novel set in a world ruled by Titans, humans made immortal and superhuman through a drug called T7. Harkaway sets the book in a world that is static with technological progress frozen, and controlled by a tiny elite – the Titans. 

The Titans have every incentive to keep it that way. If you intend to live forever, you want predictability. You suppress black swan events. You prevent anyone else from accessing the technology that made you powerful.

Like all good science fiction, Titanium Noir made me think of the current moment – about ASI and what the impact of a powerful new technology might have on society.

The Accelerationist Promise

The dominant narrative around ASI assumes dynamism. 

Ray Kurzweil’s singularity. Dario Amodei’s “Machines of Loving Grace,” which imagines AI compressing a century of scientific progress into a decade. The promise is exponential takeoff: once we build superintelligent systems, growth compounds, scarcity dissolves, and we enter a post-scarcity future.

The doomers share this assumption of exponential takeoff, just with the sign flipped. Eliezer Yudkowsky’s scenarios (as laid out in “If Anyone Builds It, Everyone Dies“) and reports like AI 2027 project rapid, destabilizing change. Whether utopia or catastrophe, the shared premise is acceleration.

But is this a foregone conclusion? What if the incentives point elsewhere?

Infrastructure Investment

Trillions of dollars are being invested right now in AI infrastructure: data centers, chips, power plants. Microsoft is signing 20-year power purchase agreements. NVIDIA’s market cap rivals the GDP of mid-sized nations. The US has imposed export controls on advanced chips to China. This is concrete capital deployed by a small number of companies with the resources to play at this scale.

AI is constrained by compute, which is constrained by power, which is constrained by massive capital investment and regulatory approval. The entities building this infrastructure are building moats. And a sufficiently powerful AI system, controlled by a sufficiently small group, creates interesting incentives. 

Does it make sense to continue to invest trillions of dollars in compute? At what point is the investment enough and are the returns justified?

The Stasis Thesis

Consider an alternative scenario. ASI emerges, powerful but without agency. Think of it as a super-powered, general purpose Claude Code – but without consciousness or autonomous goal-seeking behavior. 

I think this is as plausible as the scenarios involving goal-oriented or “selfish” behavior that keep AI safety researchers up at night. 

The ASI systems in this scenario are transformative, but also controllable, and controlled by those who built and own the infrastructure.

What do they do with it?

Titanium Noir suggests an alternative: freeze the world. 

A small elite controls compute and power. A large population lives in stasis, perhaps supported by something like a Universal Basic Income, pacified and surveilled by these AI systems. 

The technology that could enable abundance instead enables control. Growth stops because those in power benefit from predictability. Black swan events are suppressed. The world becomes static.

This is dystopia in the mundane sense. A world where nothing much changes, ever, because change threatens the position of those who own the infrastructure.

AI and Capital

In late December 2025, Philip Trammell and Dwarkesh Patel published “Capital in the 22nd Century,” arguing that while Piketty was wrong about the past, he may be right about the future. 

Their thesis: once AI and robots can fully substitute for human labor, the economic logic that has historically raised wages breaks down. Capital accumulates indefinitely to those who own it. Wealth concentrates. The gains flow upward without limit.

This is pessimistic, but it still assumes dynamism. Growth continues; it just accrues to the owners of capital. 

The stasis thesis is more pessimistic. What if those who control ASI don’t want continued growth at all? Does generating shareholder returns actually matter when economic growth becomes a non-factor?

ASI could be a technology capable of suppressing and controlling everything. Those who control the infrastructure now have a tool to assert complete dominance. And if maintaining control means ASI induced stasis, then it might be a price worth paying.

The Titans of Titanium Noir froze their world because immortality makes you conservative. Infinite time horizons make you risk-averse. You stop wanting change and start wanting control. Growth itself becomes a threat.

Trammell and Patel worry about inequality spiraling upward forever. I wonder if the ceiling is lower and harder: a world frozen in place by those who got there first.

Roadtrippin’

Fifteen hours alone in a minivan will take your mind to strange places.
Last weekend, as I drove from the Gulf Shore back home, mine wandered from gas station hot dogs to the future of AI.

📎 It feels like we’re already in a “paperclip maximization” loop.
Each new model is just good enough to justify ongoing jaw-dropping investments in data, compute, and talent. Data center construction now seems to be propping up a flagging US economy. But the benefits of AI don’t yet show up in the numbers.
Is the point of AI simply… to build more AI?

🙏🏾 AI research has often been overtly religious undertones. Kurzweil imagined post-singularity AI as an omnipotent God — the Old Testament kind: awesome, inscrutable, alien.
But maybe we don’t get that.
With so many teams building frontier models, maybe we get something closer to the Hindu pantheon — a whole cast of deities, each with their own agendas. Some awe-inspiring. Others… a little kooky.

🎭 Calling a startup a “ChatGPT wrapper” used to be an insult.
Now I think we’ve all become AI wrappers — sometimes just the meat-interface for LLMs.

🐉 On vacation, my kids and I made up stories:
Pink glitter dragons.
Mean unicorns.
Friendly witches.
Fearsome fairies.
I tried asking ChatGPT for stories, but even the most expensive model couldn’t match my three-year-old’s chaotic creativity. That made me hopeful — because what are humans, if not storytellers?

💀 There’s probably a billion-dollar business in a “dead man’s switch” for chatbots.
An app that erases your entire chat history when you die.
Because I’d rather not be remembered as the guy who once asked ChatGPT why the minivan’s doors wouldn’t close.
(It was a switch. Of course it was.)

The Future is Here..

It’s just not very evenly distributed ..

This thought-provoking quote by William Gibson has been on my mind recently. The frantic pace of AI development contrasts sharply with the casual indifference of friends and family who do not care about cutting-edge technology.

Most people outside the tech community may have heard about ChatGPT, LLMs, or other “autonomous” technology in passing.

However, we will increasingly see these worlds intersect. Take, for example, this amusing video of a San Francisco police officer attempting to reason with a wayward Waymo car.

The cop steps before the slow-moving vehicle, commanding it to stop and stay like an errant puppy. He then lights a flare in front of the car, hoping the smoke would make it stop.

The video is funny but is also a cautionary tale of the types of issues that we will face when introducing autonomous agents to the broader public.

Just like the bewildered cop, we will have to deal with users who do not understand the capabilities and limitations of new technology.

Designing effective User Interfaces and Experiences for these complex new technologies will be critical to broad and safe adoption.

Google, Microsoft and the Search Wars

A demo cost Google’s shareholders $100bn dollars last week. Why?

Google’s Share Price after the Bard event

Google has dominated search and online advertising for the last twenty years. And yet, it seems badly shaken by Microsoft’s moves to include a ChatGPT-like model in Bing search results. 

Why is this a threat to Google?

1️⃣ Advertising: Google’s revenues are driven by the advertisements it displays next to search results. The integration of language models allows users to get answers – removing the need to navigate to websites or view ads for a significant subset of queries.

2️⃣ Capital Expenditure: Search queries on Google cost around $0.01 (see link in the comments for some analysis). Integrating an LLM like ChatGPT *could* cost an additional 4/10th of a cent per query since the costs of training and inference are high. Even with optimization, integrating LLMs into Google search will increase costs in running search queries. According to some estimates, this directly impacts the bottom line to almost $40bn. 

3️⃣ Microsoft’s Position: Bing (and, more broadly, search) represents a small portion of Microsoft’s total revenues. Microsoft can afford to make search expensive and disrupt Google’s near-monopoly. Indeed Satya Nadella, in his interviews last week, said as much (see comments). 

4️⃣ Google’s Cautious AI Strategy: Google remains a pioneer in AI research. After all, the “T” in GPT stands for Transformer – a type of ML model created at Google! Google’s strategy has to sprinkle AI in products such as Assistant, Gmail, Google Docs, etc. While they probably have sophisticated LLMs (see LaMDA, for example) on hand, Google seems to have held off releasing an AI-first product to avoid disrupting their search monopoly. 

5️⃣ Curse of the demo: Google’s AI presentation seemed rushed and a clear reaction to Microsoft’s moves. LLMs are known to generate inaccurate results, but they didn’t catch a seemingly obvious error made by their BARD LLM in a recorded video. This further reinforced the market sentiment that Google seems to have lost its way.

References and Further Reading

Ben Thomson’s “4 Horsemen of the Tech Recession”

In the last month, we have had huge layoffs across technology, yet the “real economy” seems robust. What is going on?

Meta is making 2023 ‘a year of efficiency’. Microsoft, Alphabet, and many other companies have stated economic headwinds as the reason for letting thousands of people go. 

However, last week, the US posted the lowest unemployment numbers in 50 years(!) while adding half a million jobs. 

Ben Thomson discusses this in this week’s excellent Stratechery article. 

He points to 4 factors that are causing this disconnect:

1️⃣ 😷 The COVID Hangover -> Companies assumed COVID meant a permanent acceleration of eCommerce spending. Customer behavior has reverted (to a certain extent) to pre-pandemic patterns

2️⃣ 💻 The Hardware Cycle -> Hardware spending is cyclical. After bringing forward spending due to the pandemic, customers are unlikely to buy new hardware for a while.

3️⃣ 📈 Rising interest rates -> The era of free money is over. Investing in loss-making technology companies in anticipation of a future payout is no longer attractive.

4️⃣ 🛑 Apple’s Application Tracking Transparency (ATT) -> ATT has made it difficult to track the effectiveness of advertising spending. This caused enormous problems for companies like Meta, Snap, etc. that rely on advertising.

The Limits of Generative AI

AI is having a moment. The emergence of Generative AI models showcased by ChatGPT, DALL-E, and others has caused much excitement and angst. 

Will the children on ChatGPT take our jobs? 

Will code generation tools like Github Copilot built on top of Large Language Models make software engineers as redundant as Telegraph Operators? 

As we navigate this brave new world of AI, prompt engineering, and breathless hype, it is worth looking at these AI models’ capabilities and how they function. 

Models like the ones ChatGPT uses are trained on massive amounts of data to act as prediction machines. 

I.e., they can predict that “Apple” is more likely than “Astronaut” to occur in a sentence starting with: “I ate an.. “.

The only thing these models know is what is in their training data. 

For example, GitHub Copilot will generate better Python or Java code than Haskell. 

Why? Because there is way less open-source code available in Haskell than in Python. 

If you ask ChatGPT to create the plot of a science fiction film involving AI, it defaults to the most predictable template. 

“Rogue AI is bent on world domination until a group of plucky misfit scientists and tough soldiers stops it.” 

Not quite HAL9000 or Marvin the Paranoid Android. 

Why? Because this is the most common science fiction film plot.

Cats and Hats

Generative AI may generate infinite variations of a cat wearing a hat, but it has yet to be Dr. Suess. 

AI is not going to make knowledge work obsolete. But, the focus will shift from Knowledge to Creativity and Problem-Solving. 

Between Rock and a.. podcast?

Just because you can do it doesn’t make it a great business model. Take music streaming, for example.

Image by Chloe Ridgeway on Unsplash

Spotify, the world’s most popular streaming service, has been the target of some Internet ire in the last week or so. Neil Young, the creator of the legendary Pono digital media player (apparently he made some music too?), decided he didn’t want anything to do with Spotify. 

Why all the righteous indignation?

Spotify pays Joe Rogan, a media personality / MMA commentator / master of “doing his own research,” over $100m to have exclusive rights to his wildly popular podcast. 

Apparently, Mr. Rogan has some interesting ideas around COVID, vaccinations, and horse de-worming medication. Not particularly controversial topics 😬. 

Why is this a big deal for Spotify?

Music streaming is a terrible business. Spotify has been bleeding cash for years and only recently turned a meager profit. The company had an operating margin of 1.4% in the first nine months of last year. No hockey sticks in sight.

The reason? It has to pay royalties to music labels for each music stream. The value from streaming accrues to the music companies, not to the streamers or artists.

Spotify makes its money not from streaming but from selling subscriptions and advertising. 

This is where podcasts come in. Spotify pays millions to Joe Rogan because he brings in a massive audience in the highly desirable 18-34 demographic. Spotify offers targeted advertising on podcasts to its most important customers, advertisers. This makes much more economic sense than making tiny margins on each stream of, let’s say, “Rockin’ in the Free World.” 

The risk to Spotify in this, slightly ridiculous, situation is not losing access to rock & roll; its not being able to monetize their investments in podcasting. 

Spotify would rather you come for the music and stay for Elon Musk smoking some fine herb  with his buddy Joe Rogan. 

They have set up expectations for their users that they can stream any song at any time. So they have to double down on more economically viable content like the Joe Rogan Experience. 

I am sure there is a Neil Young song about rocks and hard places..

On Roblox’s Outage

Roblox is one of the world’s biggest game platforms. With over fifty million daily users, it is a wildly popular platform to build and play games. 

In October last year, they had an outage where the entire platform was down for over 72 hours. This was all over the news at the time..

Today, Roblox published a post mortem about the incident. It is fascinating reading for anyone interested in distributed systems, DevOps, and Engineering (link below). I will write up a more detailed note in a couple of days.

Summary
– The outage was due to an issue in their service discovery infrastructure which is implemented in Consul
– Roblox is deployed on-premise(!!) on 18,000 servers which run 170,000 service instances
– These services rely on Consul (from HashiCorp) for service discovery and configuration
– An upgrade to Consul and the resulting switch to the way services interact with Consul lead to a cascading set of failures resulting in the outage

Some Initial Thoughts
– Distributed systems are hard, and the use of service-oriented architectures come with costs of coordination and service discovery
– Microservice architectures do not reduce complexity, just move it up a layer of abstraction
– The complexity of the modern software stack comes not just from your code, but also from your dependencies. 
– Leader election is one of the hardest problems in Computer Science 🙂 

On Forgiveness in UX Design

As engineers and designers, we need to focus on building products that have empathy and forgiveness for their users. 

Software is eating the world, but as it optimizes for engagement and retention, it leaves behind confused and exhausted users. 

Companies raise millions of dollars at billion-dollar valuations. With those valuations comes a drive to add new features. With the move to SaaS for everything, user interfaces and modes of interaction seem to change overnight.

Perhaps we could take inspiration from the consumer packaged goods industry. 

As a new father, I have changed diapers in various circumstances. In the dark, in the park, trying to mitigate a full-on meltdown and sometimes just trying to stem an avalanche of 💩. 

And yet, the diaper works as intended. Forgiveness is built into the design. I can operate it one-handed if I have to, and it gives some protection even when not used correctly. I can be confident that the design won’t change dramatically in the next iteration.

So, dear UX designer, next time you fire up Figma, think of the humble diaper, and a poor sleep-deprived dad dealing with a poop 🌋 at 3am. 

Think of the mistakes a user may make and design your application to forgive them and not punish them when they make those mistakes when addled, distracted, or simply exhausted.