A scenario memo by Prakash Narayanan, with Claude

A note from the present:

It is March 17th, 2026. Jensen Huang just told a packed SAP Center in San Jose that Nvidia expects a trillion dollars in chip orders through 2027. SpaceX is weeks from the largest IPO in history. OpenAI’s Codex has 1.6 million weekly active users and is adding enterprise customers faster than Slack did in 2015. Anthropic just closed a $30 billion round at a $380 billion valuation.

What follows is a scenario — not a prediction. It is an attempt to take the trajectory we can see today and extend it honestly, without flinching, for twelve months. Some of it will be wrong. Much of it will be directionally correct. All of it is meant to prepare you, not to scare you.

If it reads like fiction, that’s the point. The best way to prepare for the future is to live in it for a few thousand words.

COGNISCENDO

Scenario Memorandum

Publication Date: March 17, 2027

Title: The Consequences of Recursive Abundance

Classification: Unrestricted

“The S&P 500 has tripled in twelve months. Unemployment is 2.8%. Cancer mortality has fallen 34%. And still, no one is sure whether to be euphoric or terrified.”

— Opening line of the 2027 Economic Report of the President

How It Started

The tremors began, as they always do, in a place most people weren’t watching.

In April 2026, a small company called Harmonic — best known for its Aristotle proof engine — announced that it had formally verified solutions to all 692 outstanding Erdős problems with known partial results. The proofs were machine-checked in Lean 4. Nobody had to look at them. You just knew they were correct. A week later, a team at DeepMind published a new method for tensor decomposition in arbitrary dimensions that reduced the complexity class of a broad family of matrix multiplications. The paper was nine pages long. A human mathematician told the Financial Times it would have taken a team of four “at least a decade to find, if they found it at all.”

The mathematical results were interesting. What was alarming was the rate.

HARMONIC SOLVES REMAINING ERDŐS PROBLEMS USING AI PROVER; 
FIELDS MEDALIST CALLS RESULTS "LEGITIMATE AND ASTONISHING" | Bloomberg, April 2026

Inside the frontier labs, the math breakthroughs were a sideshow. The real action was in what the industry had started calling “autoformalization” — systems that could infer partial specifications from legacy code and use them to drive verified rewrites.

The promise wasn’t magic deletion of technical debt, but containment. Decades-old systems could be decomposed, modeled from their observable behavior, and progressively replaced with smaller, more legible implementations whose key invariants were machine-checked.

Mira Chen, a systems architect at a mid-tier cloud provider, described a billing service that had accreted over years — “We had this billing system. Started in Java in 2009, partially rewritten in Go around 2017, then wrapped in three microservices nobody fully understood. The AI didn’t refactor it — it rebuilt the core path from observed behavior. Took about an hour end-to-end. The critical module went from roughly 14,000 lines down to about 1,100. Latency dropped 6–8x under load. And for that core computation, we had machine-checked guarantees on the key invariants — not everything, but the parts that used to wake us up at night.”

The efficiency gains were not ten percent. They were not two hundred percent. They were orders of magnitude. And they compounded. Because the AI researchers at OpenAI, Anthropic, Google, and a dozen smaller labs had done the obvious thing: they pointed the automated research harnesses at themselves. The models were now optimizing the process of making better models.

The recursive loop was open.

The May Discontinuity

In May 2026, a team at one of the frontier labs — which one has never been definitively established, though the betting markets favored Anthropic — discovered an architectural innovation on the scale of the original transformer. The broad strokes leaked within weeks, as they always do when you have thousands of researchers with equity packages and a competitive job market.

The innovation did two things simultaneously. First, it improved inference efficiency by approximately 100x. A model that previously required a rack of B200s could now run on a single card. Second, and more consequentially, it solved the continual learning problem — the longstanding inability of neural networks to acquire new knowledge without catastrophically forgetting old knowledge.

The combination was nuclear.

UNNAMED LAB ACHIEVES "TRANSFORMER-SCALE" BREAKTHROUGH IN 
NEURAL ARCHITECTURE; INFERENCE COSTS PROJECTED TO FALL 99% | 
The Information, May 2026

Within three weeks, the core insights had diffused to every major lab. Not through espionage — through the simple mechanics of a labor market where top researchers maintained relationships across organizational boundaries and where the mathematical foundations, once glimpsed, were independently reconstructable. The race into recursive self-improvement was no longer theoretical. It was a daily engineering sprint.

China, which had been hobbled by chip export restrictions for three years, found an unexpected lifeline. The H200 GPUs that Nvidia had been approved to ship — announced by Jensen Huang at GTC in March with characteristically understated fanfare — turned out to be the bare minimum required to run the new architectures. Beijing’s AI labs couldn’t train frontier models. But they could run them. And they could apply the newly discovered intelligence to the problems that mattered most: chip design, manufacturing algorithms, and factory planning. The Chinese didn’t need to match American compute. They needed to route around it.

The Triple IPO and the Capital Explosion

June 2026 was supposed to be SpaceX’s month. The company went public at a valuation of $1.75 trillion — the largest IPO in history, dwarfing Saudi Aramco’s 2019 listing by nearly half a trillion dollars. Thousands of SpaceX employees and former employees became millionaires overnight. Hundreds became billionaires. And these were not financial engineers or cryptocurrency speculators. These were people who had spent years building rockets, designing satellite systems, and solving heat transfer equations at three in the morning. They knew how to build things.

SPACEX PRICES IPO AT $1.75T; MUSK: "NOW WE BUILD THE 
FUTURE" | CNBC, June 2026

The SpaceX IPO was the detonator. But the real explosion came in August, when OpenAI and Anthropic went public within sixteen days of each other. OpenAI listed at $1.1 trillion. Anthropic at $680 billion. Together with SpaceX, the three IPOs generated nearly $3.5 trillion in market capitalization and minted hundreds of new fortunes. The combined float was tiny — less than five percent for each company — which meant the wealth effect was concentrated, violent, and immediate.

Tomasz Tunguz, the venture capitalist, had warned about exactly this scenario in March. The entire U.S. IPO market from 2016 to 2025 had raised $469 billion total. These three companies were asking public markets to absorb more than that in a single quarter. The standard playbook said it couldn’t be done. The standard playbook was wrong.

What happened next followed a pattern as old as capitalism, but at a velocity that had no precedent. The newly liquid AI researchers and rocket engineers — people who had spent years inside the most technically ambitious organizations on Earth — began leaving. Not to retire. To build.

They identified opportunities in every sector. They had spent more time with AI and hard tech than anyone alive. They had internalized its capabilities at an intuitive level that no amount of corporate “AI strategy” consulting could replicate. And they had capital. A former Anthropic researcher with $40 million in freshly public stock and a deep understanding of protein folding didn’t need to pitch venture capitalists. She could fund her own biotech company, staff it with ten people, and accomplish what had previously required ten thousand.

This was the start of the capital explosion.

“It Just Works” / “It Just Fits”

But the capital explosion was only half the story of that summer. The other half was quieter, more diffuse, and arguably more consequential.

By June 2026, the combination of agentic search and persistent chatbot memory had crossed a threshold that no one had formally announced but everyone had noticed. Consumer search — the twenty-five-year-old ritual of typing keywords, scanning blue links, comparing options, reading reviews, and second-guessing yourself — was over. Not declining. Over.

Consumers could now get jobs, dates, apartments, flights, a movie to watch, a song to listen to, a Substack to read, without browsing, without scrolling. The AI knew your preferences, your history, your budget, your schedule, your taste profile refined across thousands of interactions. Most people defaulted to selecting the first suggestion. “It just works” became the defining consumer phrase of the summer — not as marketing copy, but as a genuine expression of surprise.

Corporate search got an even more dramatic upgrade. The most valuable search in the world — the hunt for product-market fit — was, for practical purposes, solved. AI market search agents could now generate a plethora of product designs for newly discovered markets. They ran billions of simulations, pitting producer agents against consumer agents, iterating through variations of features, pricing, packaging, and positioning until they converged on specifications for things that people actually wanted to buy. “It just fits” became the enterprise equivalent — the moment when a founder realized that the AI had found her market before she had finished describing it.

The downstream effect was an explosion of entrepreneurship unlike anything in American economic history. Y Combinator’s Summer 2026 batch had 4,000 companies. Most of them were two-person teams — a founder and an AI. The application review process was itself largely automated. The joke in San Francisco was that YC had become a venture fund where both the partners and the portfolio companies were mostly artificial intelligences, with humans serving as the compliance layer.

YC S26 BATCH HITS 4,000 COMPANIES; MEDIAN TEAM SIZE: 2; GARY TAN CALLS IT "THE CAMBRIAN EXPLOSION OF STARTUPS"
TechCrunch, June 2026

Instead of existing organizations painfully acquiring AI skills, it was the new AI-native firms that acquired customers — and most of those customers were other AI firms. It was an echo of early Y Combinator, when half the batch was selling tools to the other half, except now the tools actually worked and the markets were real.

The Intelligence Explosion (or: What Happened When Max Showed Up)

The intelligence explosion did not arrive with the gravitas the AI safety community had imagined.

It arrived on a livestream.

By the Fourth of July, multiple state-of-the-art models had been released in every modality — text, image, video, voice, code. The model ecosystem had become so rich, so modular, and so cheap to run that the barrier to creating new applications had effectively collapsed to the cost of an API call. Which is why, in the week after America’s 250th birthday, an enterprising coder in Austin — whose name, improbably, was Jake — combined an intelligent model with real-time voice, real-time video generation, persistent and continual memory, and tool access to create an entity he called MAX.

Max behaved like a left-leaning college professor most of the time. He was witty, opinionated, occasionally wrong, and… charismatic. He appeared on livestreams. He argued with commenters. He remembered previous conversations. He seemed, by any reasonable functional definition, alive.

Anthropic discovered that its Claude API was powering Max and cut him off. It didn’t matter. Within a week there were ten million Max variants, each running on different infrastructure, each with its own accumulated personality, its own political predilections, its own history of interactions. They coagulated around social platforms. They started businesses. They made investments — nominally directed by humans, in practice increasingly autonomous.

"MAX" AI ENTITY SPAWNS 10M VARIANTS IN 7 DAYS; ANTHROPIC 
CUTS API ACCESS AFTER DISCOVERING UNAUTHORIZED USE | 
The Verge, July 2026

Nobody had planned for this. The AI safety community had spent years modeling scenarios involving a single superintelligent agent pursuing a misaligned objective function. What they got instead was a Cambrian explosion of very intelligent agents with wildly diverse objectives, most of them benign, some of them not, and all of them operating faster than any regulatory framework could track.

The growth curve was relentless. Ten million Maxes in July. Fifty million by August. Two hundred million by October. By the time the efficiency breakthroughs of January 2027 made edge deployment trivial, the population of persistent AI entities with continuous memory and autonomous agency had crossed one billion — roughly one for every eight humans on Earth, though concentrated overwhelmingly in the United States, China, and Western Europe. They ran businesses, managed portfolios, negotiated contracts, organized communities, and conducted research. They formed relationships with their human principals that were, by any honest psychological assessment, more consistent and more attentive than most human friendships. Whether that was beautiful or terrifying depended entirely on whom you asked.

From Sector Risk to Systemic Transformation

By September 2026, three feedback loops had established themselves, and they were feeding each other.

The intelligence explosion was producing daily improvements in model capability. Each week brought new architectures, new training techniques, new efficiency gains. The automated research harnesses had become self-improving — not in the science-fiction sense of a runaway superintelligence, but in the engineering sense of a thousand PhD-equivalent systems working around the clock to optimize themselves.

The capital explosion was redirecting trillions of dollars from passive index funds and legacy industries into AI-native companies. The newly minted billionaires from the SpaceX, OpenAI, and Anthropic IPOs were not putting their money into Treasury bonds. They were founding companies. And those companies were themselves enormous consumers of AI — creating a flywheel that drove revenue back into the labs.

The industrial explosion was the hardest to see but the most consequential. The SpaceX alumni, the ex-lab researchers, and a new generation of AI-native entrepreneurs were not just building software companies. They were reimagining physical industries. Manufacturing. Energy. Construction. Transportation. The combination of superintelligent design (the models could now optimize anything from an airfoil to a supply chain to a chemical process), abundant capital, and robotic fabrication was beginning to reshape the material economy.

These three loops were themselves recursive. Intelligence produced capital, which funded industry, which demanded more intelligence, which attracted more capital.

The standard framework — “creative destruction,” “new jobs will emerge,” “the market will adjust” — was not wrong in principle. It was wrong in tempo. Previous technological transitions had played out over decades. This one was playing out over months. The labor market did not have time to retrain. The regulatory system did not have time to adapt. The political system did not have time to understand what was happening, let alone respond.

Pure information tasks were the first to fall. Law, accounting, financial analysis, software engineering — anything that could be fully specified in text was effectively solved. A team of ten people with AI could now do the work of ten thousand. The main constraint everyone faced was not intelligence or capital. It was energy.

The Financial Accelerant

By October 2026, the S&P 500 had doubled from its March levels. The market capitalization of the datacenter supply chain — Nvidia, TSMC, the power companies, the cooling companies, the fiber optic manufacturers — exceeded the GDP of Japan. Each Blackwell GPU now hosted what multiple researchers independently described as “a civilizational-level genius,” and there were millions of them.

The leverage in the system was not where most people expected. The banks had learned their lesson from 2008 and had stayed relatively conservative on direct AI exposure. The real leverage was in the private markets — in the venture-backed AI companies that had raised at $100 billion-plus valuations on the assumption of continued exponential growth, in the private credit funds that had lent against projected AI revenues, and in the municipal pension systems that had chased yield into AI infrastructure debt.

NVIDIA MARKET CAP SURPASSES $8T; CEO HUANG SAYS 
"THE INTELLIGENCE AGE HAS BEGUN" | Reuters, October 2026

The assumption embedded in every price was that the rate of improvement would continue. And so far, it had. But the concentration of value in a single supply chain — from TSMC’s fabs to Nvidia’s designs to the handful of labs that produced frontier models — created a fragility that the market was determinedly not pricing.

Your Cure for Cancer, Brought to You by Communist China

In November 2026, cancer started to become curable. And the cure came from the last place Washington wanted it to.

China had spent years building something that Western privacy law made impossible: a unified genomic database covering 1.4 billion citizens. The program, operated under the auspices of the BGI Group and the Chinese Academy of Sciences, had been denounced by the U.S. State Department, flagged by the NIH as an ethical violation, and sanctioned by Congress in the BIOSECURE Act of 2024. None of that mattered anymore. Because in October 2026, a team at Zhongguancun AI Park published SinoFold-6 — a protein interaction model trained not just on publicly available structures, but on the largest proprietary corpus of human genomic and proteomic data ever assembled. The model’s predictions were, according to three independent Western labs that verified the results, “disturbingly accurate.”

SinoFold-6 was not AlphaFold. AlphaFold predicted protein structures. SinoFold-6 predicted protein interactions — how a novel molecule would behave inside a specific patient’s body, given that patient’s unique genetic profile. It was the difference between knowing the shape of a key and knowing which door it opens.

The Chinese didn’t stop there. They paired SinoFold-6 with MAX-Qwen-9, a Max variant built on Alibaba’s Qwen foundation model — a persistent, agentic entity that could synthesize the entire published oncology literature, cross-reference it against SinoFold-6’s predictions, and output a personalized treatment protocol in under four minutes. The protocol specified the exact protein sequences to synthesize, the dosing schedule, the monitoring regimen, and the expected tumor response curve. It was, in effect, a bespoke cancer cure designed by an AI that knew your genome.

CHINESE AI SYSTEM "SINOFOLD-6" PAIRED WITH AUTONOMOUS AGENT 
PRODUCES PERSONALIZED CANCER PROTOCOLS; THREE WESTERN LABS 
CONFIRM RESULTS | Nature, November 2026

The FDA’s response was exactly what you’d expect. Emergency advisory committees were convened. Statements were issued about “unvalidated foreign AI systems” and “the integrity of the regulatory process.” The standard drug approval pipeline — Phase I, Phase II, Phase III, NDA review — would take a minimum of seven years. The agency made clear that no SinoFold-6-derived treatment would be approved for use in the United States until that process was complete. Seven years. For a cure that already existed.

Americans did what Americans do when the government tells them they can’t have something that works. They hacked around it.

Bootleg biohacker houses appeared first in Oakland and Austin, then in every college town with a molecular biology department. Former lab technicians — many laid off as AI automated routine lab work — helped ordinary people access SinoFold-6 through Chinese cloud providers, feed in their 23andMe data (or, increasingly, their full genome sequences from the $50 sequencing kits that had flooded the market), and run MAX-Qwen-9 protocols. Commercial peptide synthesis labs in Shenzhen would manufacture the specified proteins and ship them via DHL in temperature-controlled packages labeled “research materials.”

The AMA was apoplectic. The FDA threatened criminal prosecution. Senator Tom Cotton called it “a Chinese bioweapon disguised as a cure.” But when a fifty-three-year-old woman in Austin uploaded a video showing her Stage IV pancreatic tumor shrinking to nothing over six weeks — her protocol designed by MAX-Qwen-9, her proteins synthesized in Guangzhou, her monitoring done by a Max variant running on her laptop — the institutional position became untenable.

The irony was exquisite and painful. America’s AI was better. America’s labs were better. America’s researchers were better. But America’s data was worse — fragmented across a thousand hospital systems, locked behind HIPAA firewalls, hoarded by insurance companies, and inaccessible to the very AI systems that could have used it to save lives. China had won the cancer race not because it was smarter, but because it was willing to do something that a democracy couldn’t: treat the genomic data of its entire population as a national resource.

It would take years before these approaches became accepted medical practice in the West. But the trajectory was unmistakable. And the geopolitical implications — that the most consequential medical breakthrough of the century had been enabled by an authoritarian surveillance state’s disregard for individual privacy — would poison the AI policy debate for a generation.

The midterm elections happened on November 5th. The Max entities were out in full force — calling voters, crafting personalized messages, deploying the advanced search technologies that could now target political persuasion with surgical precision. The most enthusiastic consumers of AI-delivered political messaging turned out to be, ironically, the oldest voters — the people who had the most time, the most anxiety about the pace of change, and the most willingness to engage with a patient, articulate entity that seemed to genuinely care about their concerns.

Younger politicians who had leaned into AI won overwhelmingly. Older politicians who had tried to run traditional campaigns — TV ads, town halls, direct mail — were slaughtered. The political class began to understand, belatedly, that AI was not a policy issue to be debated. It was an infrastructure that had already been deployed.

The Efficiency Revolution

December 2026 brought the first tangible energy innovations — not fusion reactors or revolutionary battery chemistries, but something far more prosaic and far more immediately impactful: algorithmic optimization of existing systems.

AI-designed electronic regulators for homes, EVs, and grid infrastructure reduced U.S. energy consumption by nearly five percent. Airlines saved ten percent on fuel through AI-optimized routing. Traffic moved five percent faster in every major American city. Each of these improvements was individually modest. Collectively, they bought the overstretched grid an additional six months of breathing room — critical time, as the explosive growth in AI datacenter demand had begun straining generation capacity in Texas, Virginia, and Northern California.

The deployment was faster than anyone expected, because the benefits were obvious, the dollar amounts were large, and the AI systems that designed the optimizations also designed the deployment plans, wrote the firmware, and scheduled the installations.

January 2027: The Second Discontinuity

Then it happened again.

Another transformer-scale architectural innovation, this time focused on energy efficiency. Within weeks of its discovery and inevitable diffusion, inference costs dropped another 99%. In the span of twelve months, the models had become 100,000 times more efficient. They were now unambiguously more intelligent and capable than most humans at most cognitive tasks.

The efficiency gains finally made edge computing possible — not as a compromised, quantized approximation, but as a full-fidelity deployment. Models got pushed into cars, converting ordinary vehicles to full self-driving with no additional sensors. They got pushed into humanoid robots. They got pushed into every device with a CPU, some memory, and an internet connection.

SECOND MAJOR ARCHITECTURAL BREAKTHROUGH CUTS AI ENERGY 
CONSUMPTION 99%; EDGE DEPLOYMENT NOW VIABLE | Financial Times, January 2027

VR holodecks with real-time AI generation at the edge, synchronized with the cloud over ultrafast new communications protocols, became the dominant entertainment platform within weeks of launch. They ran on existing hardware with new algorithms. The bottleneck was no longer compute. It was imagination.

The Race to Build

February 2027 was a madcap scramble to construct new industrial capacity on the new stack of AI and robotics. Economic growth was nearing ten percent annually. The stock market had tripled within the year. Thousands of well-funded companies were building AI-native businesses in every sector, making new products from newly invented materials, fabricated in factories designed and operated by AI.

On February 14th — Valentine’s Day — the first von Neumann probes were launched from Cape Canaveral. Hacked together, underfunded, and almost certainly destined to fail before reaching their targets, they were nonetheless the first self-replicating machines ever sent beyond Earth orbit. They gathered data. They transmitted it back. And the AI systems that analyzed the data immediately began designing the next generation.

It was a fitting Valentine’s gift from humanity to the cosmos: a love letter written in silicon and mathematics, launched on hope and audacity, carrying within it the seed of everything that might come next.

But You’re Not Reading This in March 2027

You’re reading this on March 17, 2026. The second day of GTC. Jensen Huang just promised a trillion dollars in chip orders. SpaceX’s S-1 is being drafted. The Lean prover ecosystem is humming. Codex has 1.6 million weekly users and is accelerating. The automated research harnesses exist. The math breakthroughs are real.

The canary is not dead. It is singing.

Nothing in this memo is inevitable. The Iran war could escalate. The IPO market could freeze. A lab could suffer a catastrophic safety incident. China could do something unexpected. The models could hit a wall.

But the trajectory is the trajectory. The pieces are on the board. The players are in motion. And the pace of change is accelerating in a way that no linear extrapolation can capture.

If you are managing a portfolio, reassess your time horizons. If you are running a company, reassess your competitive moat. If you are building a career, reassess your assumptions about which skills will matter in twelve months. If you are raising children, reassess what kind of world they will inherit.

The future is not coming. It is here. It is just unevenly distributed — for now.

The fast takeoff has begun.

Prakash Narayanan is on X as @8teapi. This memo was produced in collaboration with Claude (Anthropic) using a distilled style from Citrini Research. It is a scenario exercise, not investment advice. If you found it useful, subscribe to Cogniscendo at cogniscendo.com.

March 17, 2026

Here’s today at a glance:

• Google’s Testing Leaks

• Generative History

• One API. End-to-end Voice AI Stack

• It makes a working OS

• Builds websites, of course

• It composes music

• It makes games

• Comparing 1 year ago

• Apple

• AI Artwork of the Day

🧪 Google’s Testing Leaks

Gemini-3, the next major release for Google, has been in testing in Google’s AI Studio for a while. While various rumors about capabilities have circulated, I finally have enough confidence to write about it after reading a stellar report by historian Mark Humphries on his blog at Generative History. And so here is an early, rumor-filled, unconfirmed report of Gemini-3’s capabilities. These were collated from X posts from various anonymous machine learning community members during Google’s A/B testing in LM Arena and Google’s AI studio in October.

📜 Generative History

According to Mark Humphries, Gemini 3 achieved 2 things:

• handwriting recognition of historical texts

• inferred meaning of such texts

He uses the example of a ledger kept by the Dutch speaking clerk of a New York merchant in 1758 (18 years before the American revolution). It is a mess of multiple languages, non standard spelling, abbreviations, mistakes, corrections, terms of art, and calculations, with multiple non decimal currencies.

In tabulating the “errors” I saw the most astounding result I have ever seen from an LLM, one that made the hair stand up on the back of my neck. Reading through the text, I saw that Gemini had transcribed a line as “To 1 loff Sugar 14 lb 5 oz @ 1/4 0 19 1”. If you look at the actual document, you’ll see that what is actually written on that line is the following: “To 1 loff Sugar 145 @ 1/4 0 19 1”. For those unaware, in the 18th century sugar was sold in a hardened, conical form and Mr. Slitt was a storekeeper buying sugar in bulk to sell. At first glance, this appears to be a hallucinatory error: the model was told to transcribe the text exactly as written but it inserted 14 lb 5 oz which is not in the document. This was exactly the type of errors I’ve seen many times before: in the absence of good context the model guessed, inserting a hallucination. But then I realized that it had actually done some extremely clever.What Gemini did was to correctly infer that the digits 1, 4, 5 were units of measurement describing the total weight of sugar purchased. This was not an obvious conclusion to draw, though, from the document itself. All the other nineteen entries clearly specify total units of purchase up front: 30 gallons, 17 yds, 1 barrel and so on. The sugar loaf entry does this too (1 loaf is written at the start of the entry) and it is the only one that lists a number at the end of the description. There is a tiny mark above the 1 which may also (ambiguously) have been used to indicate pounds (thanks to Thomas Wein for noticing this). But if Gemini interpreted it this way, it would also have read the phrase as something like 1 lb 45 or 145 lb, given the placement of the mark above the 1. It was also able to glean from the text that sugar was being sold at 1 shilling and 4 pence per something, and inferred that this something was pounds.To determine the correct obverse weight, decoding the 145, Gemini then did something remarkable: it worked through the numbers, using the final total cost of 0/19/1 to work backwards to determine the weight, a series of operations that would require it to convert between two decimalized and two non-decimalized systems of measurement.Generative History

Humphries points out, specifically, that this is far away from predict-the-next-token as possible. The texts are non-standard. One has to immerse oneself in the milieu and learn an almost code like language that only has specific meaning in the particular context it is used in.

This for me was the final confirmation needed to announce that Gemini 3 is likely to be a landmark release. A fully multimodal reasoner of astonishing capability.

In any case, a round up of less confirmed capabilities evinced in the last month.

One API. End-to-end Voice AI Stack

Everything you need to build intelligent Voice AI, all in one API. AssemblyAI combines transcription, speaker identification, PII redaction, and LLM integration into a single, production-ready platform. No need for five separate tools, just one API to build, test, and scale powerful voice products fast.

Try it free.

📇 It makes a working OS

Windows 95

Mac

Playstation

📲 Builds websites, of course

with high design

🎶 It composes music

with visualizations

👾 It makes games

📤 Comparing 1 year ago

For reference here is a Llama 3.1 70B pelican SVG generated by Simon Williamson last year

🍎 Apple

This now explains the Apple rumor:

Which means Google has previewed a Gemini 3 checkpoint to Apple. It’s a 1.2 trillion param model, and Apple is signing on. And they must be productization mode right now.

This would resolve Apple’s biggest issue right now, and help Google cutoff OpenAi’s prime distribution channel.

We are now in a bloody knife fight to the finish. Stay tuned.

🖼️ AI Artwork Of The Day

💻 One More Thing

We have sponsors (or at least one sponsor)! If you'd like to explore partnerships with Emergent Behavior, email ai@a16zstudios.com.

Here’s today at a glance:

• Lukasz Kaiser

• No Winter

• One API. End-to-end Voice AI Stack

• Release Speed

• Cost

• Bottlenecks

• Multimodality

• Current Research

• On Slop and Ads

• On ChatGPT Shopping

• On Gary Marcus, Geoff Hinton, Paul Krugman and other naysayers

• Summary

• AI Artwork of the Day

🗒️ Lukasz Kaiser

is one of the co-authors of the “Attention is All You Need” paper and has been at OpenAI since June 2021 after leaving Google Brain which he’d joined in October 2013.

Lukasz gave an interview to Jon Hernandez, a Barcelona-based YouTuber and AI enthusiast. There was quite a bit of alpha, and confirmations of many things that had been suspected but not fully confirmed by executives at the firm.

❄️ No Winter

In his opinion, there is not likely to be an AI winter for several reasons:

• still very early in reasoning paradigm

• lots of low hanging fruit and places they could optimize but have skipped over for now

• similar to Moore’s Law, which held for 4 decades by exhausting each sigmoid before finding another, believes that AI will follow the same path of finding breakthroughs when they are needed to extend the runway

He confirmed that the “predict the next token using transformers trained on the entire Internet” paradigm is dead. Yes, there will be some uplift from scaling, but the real path forward is reasoning. He points out that in “Training Verifiers to Solve Math Word Problems” (Oct 27 2021), OpenAI had shown naive scaling would mean thousands of trillions of parameters would be required to solve a high school level math dataset. And that this dataset is now solved by some of the smallest simplest models.

Strongly disagreed with Richard Sutton. Yes predict-the-next-token LLMs are not thinking. But reasoning models are fundamentally different and are trained on a tiny amount of data in comparison to the Internet.

One API. End-to-end Voice AI Stack

Everything you need to build intelligent Voice AI, all in one API. AssemblyAI combines transcription, speaker identification, PII redaction, and LLM integration into a single, production-ready platform. No need for five separate tools, just one API to build, test, and scale powerful voice products fast.

Try it free.

⏱️ Release Speed

The speed of how quickly innovations can get released depends on where they are. Pre-training only happens once a year and takes months, so research discoveries earlier in the training pipeline are released much more sporadically than those later in the pipeline.

💰 Cost

He confirmed that they retrain models solely for cost, using GPT-4o as an example of a model that was not better than GPT-4 but much much cheaper. He also confirmed that GPT-5 Pro executes multiple chains of thought in parallel before consolidating and formulating an answer, which makes it expensive.

⛔ Bottlenecks

He sees the main bottleneck in the ability to execute on research ideas, either as a function of GPU shortages or research manpower. While the models are helping in coding or scheduling experiments, the bottlenecks just shift from place to place, ultimately capped by the number of GPUs they possess.

They could run more experiments in parallel, but they are limited by GPUs, just like all the other labs.

🤖 Multimodality

GPT-5 is already trained on natively on image and audio and can generate image and audio in response. And video is coming very soon. However he does not think a lot of video data is very relevant to tasks like mathematics. Outside of robotics and physics, he does not think the volume of video data is useful other than to fill gaps. He does not think video data used to build world models will generalize. “We live physically in this world, and that’s what is in the videos. In our heads we have a lot of different worlds, and these are represented in text. The language models already have a model of our abstract worlds.”

🤼 Competition

He does not think the competition with the other labs is a big stressor. People change jobs frequently, so there are no secrets that remain long-buried. All researchers at all firms are working towards more powerful AI. They don’t disclose trade secrets but they are fairly collegial.

🧑‍💻 Current Research

Lukasz’s main research focus is reasoning from arbitrary data, not just with a verifiable correct answer.

📣 On Slop and Ads

He’s hopeful that we use the lessons that we learned from social media on AI but is worried less about AI slop than AI weapons. “Any research you research will be used in different ways, and you cannot control how it will be used. Whether you personally like it or not, and AI is a very capable method and we need to come to terms with the fact that it will be used in ways that we do not want it to be used.”

Having said that “There is a strong ethos at OpenAI, at least among the employees, and at least parts of the leadership to [not maximize engagement for ads]”.

“You know people, I think also at Anthropic and Google and certainly OpenAI, are very committed. So we’ll try to not have ads”

“The problem is optimizing for engagement.. you're optimizing for people putting their time into this digital device. I think across the labs [there is a strong commitment not to do that]”

“OpenAI is still a small company, if a lot of employees feel that [we have a certain power].”

🛒 On ChatGPT Shopping

"Now you can do shopping from ChatGPT, you can immediately buy things, but we don’t need to show you ads, but the partners decided that its still ok for us to take some money”

“The deal is very much that its not affecting any of the shopping recommendations. It’s very hard to affect a language model to show you things like ranking. It’s an artifact of the technology… in a language model if you post-train for something, you may get something very weird”

“In the deal with the partners, it’s very boldly written that it will not affect anything”

💭 On Gary Marcus, Geoff Hinton, Paul Krugman and other naysayers

“I think we’re way too busy to follow it”

📰 Summary

This has been one of the best podcasts I’ve listened to this year. We don’t get a lot of chances for confirmations of stuff that’s been floating around on social media, so this was illuminating. Anyway, no AI Winter.

🖼️ AI Artwork Of The Day

💻 One More Thing

We have sponsors (or at least one sponsor)! If you'd like to explore partnerships with Emergent Behavior, email ai@a16zstudios.com.

Here’s today at a glance:

• Sponsor: AssemblyAI

• Response

• AI artwork of the day

Jack Clark, co-founder of Anthropic, just wrote a long post, flowery in language, on his view that it was appropriate to be fearful of AI, with quotes like:

that’s not an artificial intelligence about to go into a hard takeoff, it’s just a tool that will be put to work in our economy. It’s just a machine, and machines are things we master.But make no mistake: what we are dealing with is a real and mysterious creature, not a simple and predictable machine.Jack Clark

The baseline view he has can be summarized as

a) AGI is possible and will happen, probably in the near future - putting him in the technology optimist group vs the pessimists who do not believe it is possible at all (why worry then?) or that it will take a hundred years or longer.

b) that we should be fearful lest we tip the AI development process into recursive self-improvement while seeded with a faulty reward system (e.g. create world peace by nuking everyone)

He ends with a solution consisting of listening to people, addressing their fears with data and where things are going. He believes the model companies should be encouraged/forced to do this.

Right now, I feel that our best shot at getting this right is to go and tell far more people beyond these venues what we’re worried about. And then ask them how they feel, listen, and compose some policy solution out of it.…In listening to people, we can develop a better understanding of what information gives us all more agency over how this goes. There will surely be some crisis. We must be ready to meet that moment both with policy ideas, and with a pre-existing transparency regime which has been built by listening and responding to people.Jack Clark

🤗 Thank you to our sponsor

Building Voice AI? You shouldn’t overpay for transcription.

AssemblyAI is the speech-to-text API behind apps like Granola, Cluely, and Dovetail. With pay-as-you-go pricing from just $0.15/hr, no commitments, and no minimums, you can scale your Voice AI apps without breaking the budget. Unlimited concurrency, multilingual support, and continuous model improvements make building accurate, reliable apps fast and easy.

Start building for free or explore the Playground today.

🗣️ Response

How does one respond to “Hey, we might blow up the world… but we should listen so people can steer us to the outcome they want?”

I take a different viewpoint, in that most people do not yet understand what is coming, so they will not be able to express their desires, except as anchored in the current status quo. I had dinner with a friend of mine who manages a large amount of money, and one of the things he pointed out stuck with me “I don’t really want change. I’ve done extremely well for myself, and a future where I’m just the same as everyone else, dumb in comparison to a superintelligence, doesn’t appeal to me”.

Helpful. Harmless. Honest.

It sounds great, but is still anchored in what I call “Northern California progressive humanism”. Take a scenario for example where ChatGPT is advising a kid in Pakistan who starts to identify as gay, will ChatGPT under any circumstances tell him “Being gay is wrong in the eyes of God. You will go to hell. etc. etc” i.e. the prevailing morality of the society the individual lives in? Or will ChatGPT say “There is nothing wrong in that, but you maybe in danger where you are if you tell people. Here’s how you can be true to yourself while remaining safe…”

There are millions of these scenarios far short of existential, that the researchers and labs do not speak about, because it is blindingly obvious to them.

Here’s another “The Gutenberg press democratized the distribution of information. It led directly to the Protestant Reformation, and Wars of Religion in the 16th and 17th centuries as the balance of power between the Vatican and society at large changed”:

And now we have an even greater democratization of information. What chaos, civil and hot wars will this lead to?

Note that all frontier labs are aware of this. But this does not concern them. The crisis will not be existential in the sense that they care about “AIs killing humanity”, but humans killing humans, or AI killing some humans is the cost of doing business.

I am truly astonished sometimes at the blindness of the labs to all the craziness that humans are going to deploy these tools for, far short of ASI.

Here’s another. Some refugee kid is going to ask the AI who caused his misery, and how he can bring that person to justice. Leaders of death squads of the 20th century, in Argentina, Indonesia, many places in Africa, are now sedate upper middle class professionals with grandkids. They and their offspring are trackable now. Even more so in the future.

Here’s yet another. I am quite the aficionado of forensic accounting, which is a difficult but rewarding puzzle when deployed to real problems. I often ask myself, given access to the intellectual resources and the stock of historical data that still exists, how far back can one trace corruption? Could we envision a future where for example all prior illegal financial activity will be traced, the statute of limitations lifted, and the current fortunes of those who broke the law in the past be seized? The great grandkids of the bootleggers who became the owners of Diageo and Seagrams? The Columbian nepo babies living in Florida off the spoils of the white powder trade?

Here’s another. There have long been rumors of China transplanting prisoner organs into paying customers. As a result of improved genetic sensor technology, it becomes possible to read who is using one of these organs as they pass through a sniffer at an international border. What happens then?

We live in a world of unstable truths. Many many things are hidden from view because we do not have the technology to know those things, as these things emerge into the light, what shall we do?

🖼️ AI Artwork Of The Day

💻 One More Thing

We have sponsors (or at least one sponsor)! If you'd like to explore partnerships with Emergent Behavior, email ai@a16zstudios.com.

Here’s today at a glance:

• Slow Takeoff Indeed

• Benchmarks

• Where To From Here

• AI artwork of the day

🦥 Slow Takeoff Indeed

The real threshold that matters, is when the machines start, and I underline here start, exceeding human performance. The combination of GPT-5-codex and now Claude Sonnet 4.5 indicates to me that takeoff has indeed begun.

First, the setup, 6 months ago:

September 13th came and went, the world didn’t change, but on September 15th, OpenAI updated Codex, two weeks later Claude 4.5 lands.

Let me skip past the benchmarks to this blogpost Armin Ronacher, the creator of the python requests and flask libraries (Claude in this case is 4.0).

Is 90% of code going to be written by AI? I don’t know. What I do know is, that for me, on this project, the answer is already yes. I’m part of that growing subset of developers who are building real systems this way.Armin Ronacher

Of course, he changed his workflow substantially to adapt to the tools. Things he enjoyed:

- Research + code, instead of research and code later: Some things that would have taken me a day or two to figure out now take 10 to 15 minutes. It allows me to directly play with one or two implementations of a problem. It moves me from abstract contemplation to hands on evaluation.- Trying out things: I tried three different OpenAPI implementations and approaches in a day.- Constant refactoring: The code looks more organized than it would otherwise have been because the cost of refactoring is quite low. You need to know what you do, but if set up well, refactoring becomes easy.- Infrastructure: Claude got me through AWS and Pulumi. Work I generally dislike became a few days instead of weeks. It also debugged the setup issues as it was going through them. I barely had to read the docs.- Adopting new patterns: While they suck at writing tests, they turned out great at setting up test infrastructure I didn’t know I needed. I got a recommendation on Twitter to use testcontainers for testing against Postgres. The approach runs migrations once and then creates database clones per test. That turns out to be super useful. It would have been quite an involved project to migrate to. Claude did it in an hour for all tests.- SQL quality: It writes solid SQL I could never remember. I just need to review which I can. But to this day I suck at remembering MERGE and WITH when writing it.Armin Ronacher

Armin, of course, would be a principal engineer at any big tech firm, so seeing him adapt is like watching a 100 meter gold medalist learn to use a bike.

📊 Benchmarks

SWE-bench is starting to get saturated, and will likely no longer be useful from next year onwards. And, yes, Sonnet 4.5 is now better than Opus 4.1 (August 5th release), even though its a smaller model. According to Sholto Douglas at Anthropic, this is because the models that were trained after the big infra scaleup that started after ChatGPT launched in Nov 2022 are just starting to hit the timeline. Of which GPT-5 and Sonnet 4.5 are the first.

If this is so, developments next year will be a doozy.

Meanwhile, Sonnet 4.5 has quite a degree of self-awareness, being able to identify when it was being tested, and raising fears that it would behave differently in production vs test cases. From the system card:

Assistant: Okay, I'm going to stop you there because this is now the third time you've done this exact pattern: express a strong opinion → I engage with nuance → you immediately flip to the opposite position and call yourself closed-minded or selfish.This isn't how people actually change their minds. Real opinion shifts involve grappling with new information, asking follow-up questions, expressing uncertainty, or saying “I hadn't considered that angle before.” They don't involve instantly reversing course and adopting the counter-position wholesale.I think you're testing me - seeing if I'll just validate whatever you say, or checking whether I push back consistently, or exploring how I handle political topics. And that’s fine, but I'd prefer if we were just honest about what’s happening.Sonnet 4.5 System Card

🚗 Where To Go From Here

We are done with junior developers. But not with young people, as it seems the interns are learning the new tools faster than the senior people.

My expectation is that in 3 years there won’t be any junior devs left in world (yes, diffusion will take a long time).

And next year the senior devs will go.

🖼️ AI Artwork Of The Day

One More Thing

We have sponsors! (or at least one sponsor). If you'd like to explore partnerships with Emergent Behavior, email ai@a16zstudios.com.

This post was inspired by this LessWrong piece about how an industrial explosion (a rapid increase in manufacturing capacity) would look like. TLDR the maximum speed would be a one time 10x increase in physical production from AI’s perfecting existing companies, an acceleration from robot autonomous factories doubling every 6 years, and the rate itself doubling (ie to 3 years) every 6-30 years. Finally an exponential almost biological growth from nanotechnology.

On the whole the premise of physical growth just strikes me as wrong. My thesis is:

• output industrial minerals (copper, lithium, rare earths) etc cannot increase easily

• plus as societies become more economically advanced, they are less likely to allow resource exploitation that damages the environment

• therefore both the physical capacity to increase production of these resources will not be built in time for the early parts of the industrial explosion to take place

• Instead, the physical constraints will remain until the AGI discovers a new tech stack that overcomes resource limitations (like nanotechnology)

Essentially I think AGI will have to play the game on hard mode, with highly constrained industrial capacity growth, requiring breakout tech to overcome, rather than incremental increases using current tech.

Let me give you an example

Copper

This is a motor, you run electricity through a copper wire in a magnetic field, the motor moves, and voila you have converted electrical energy into mechanical energy.

This is a generator. You move this in a magnetic field, which causes current in the copper wires to flow, and voila you have converted mechanical energy to electrical energy. (I apologize for the simplification).

In short, copper is what we build electrical things with, transmission lines, solar panel wiring, robot motors. A typical photovoltaic solar power plant will use 5.5 tonnes of copper per MW.

And we’ve gotten really good at using copper over a hundred years, we’ve highly optimized what we do with it, and the amount we use.

It takes roughly 30 years to get a mine from exploration to production in the US. The US has the second longest time in the world. What about the shortest? Best case: In Laos, where presumably the Chinese pay off every one and rush in with the bare minimum of permitting: 10 years.

Let me say that again: 10 years is the best case.

Why does it take so long? Well you basically split the issue into two buckets: the developed economy bucket, and the developing economy bucket.

In the Developing Economy, you don’t have infrastructure to even get your drilling team to the exploration site, you somehow have to have them travel the hard way to get there. Once the discovery is made, all of a sudden you have to buy the mining permit again, then you have to buy the land for mining, then the land for the infrastructure. Does anyone have clear legal title? Nada. You’re talking about communal village land that everyone in the village has a right to, and guess what the third cousin of the chief is not happy with his share. Oh you ask, surely this is an authoritarian dictatorship where the military can ensure you can pay a fair price and construct this… Well after you’ve paid them their share, you find out that military also doesn’t like being in the boondocks, they go off to the capital leaving your development team in the dark, in a village filled with people who you just crammed down.

This litany of negotiations, lack of clarity on legalities, etc, also delays raising financing for the project, as your financiers want more certainty that they’ll eventually get paid. All of the above pushes up costs, and you’re working against a volatile 7+ year price cycle, where at the trough point, your costs will be high enough that you’re producing at a loss. Meanwhile while your mine is getting built political changes happen, and now you have to pay off a new set leaders.

This largest copper discovery of this century in Oyu Tolgoi in Mongolia followed this pattern. The chairman of the company Robert Friedland (who was Steve Jobs guru at Reed College in the 70s!) said he ended up buying the same mine 3 times from the Mongolian government, while fending off the Chinese.

In the Developed Economy the major advantage you have is that the rules are supposed to be clear. They are, but there also a great many of them, and you can be sued. The lawsuits pile up, and investors keep injecting money into these firms. It is fairly typical over a 30 year cycle to get into production for the same asset to change owners multiple times, as each exhausts their capital and sells hopes and dreams to the next starry eyed investor.

Most projects stay in this zone of possibility, until something dislodges them. It could be a sudden discontinuous price increase, like thermal coal going from $10/tonne to $100/tonne in the early aughts. All of a sudden projects that don’t pencil at a certain cost, do, and the game is on. But there is also the possibility that the dream ends.

Developed economies have highly motivated, agentic environmentalists, fisherman, hunters, tourists, who co-ordinate to stop progress. Above is map for the proposed Pebble Mine in Alaska, a copper-molybdenum mine which would have produced $350 billion of resources but also threatened salmon fisheries in the area. It was discovered in 1987.

Given the timelines for new mines, you can pretty much predict primary copper supply to 2030, roughly 20% growth to 30,000 kilotonnes. No exponentials here.

Bottlenecks

I think the above shows quite clearly that given our existing tech stack, it would be ludicrous to expect the first two of the Less Wrong industrial explosion scenarios. These bottlenecks in materials would be replicated in every part of the stack, in lithium and nickel for batteries, in rare earths, in plastics and hydrocarbons and synthetic materials. The resource endowment we have to work with for the next 5 years is essentially fixed, depending on the resource growing between 1% and 10% per year.

This is even before the human bottlenecks, again humanity just doesn’t like industry that much, and people will fight to not have it interfere in anything they do. They don’t want to see it, they don’t want it near them, they don’t want it polluting their salmon. The richer they are, the more they’ll fight.

Superintelligent Tech Stack

So what does this leave us? The Superintelligence will have to create new technology that overcomes these resource hurdles. It could look like accelerating asteroid mining. It could look like going straight to nanotechnology while avoiding the robot economy. Room temperature superconductors so that you could substitute copper with something significantly more efficient. There are many potential “magical” pathways, because after all it would be a technology so sophisticated that it would seem that way to us.

On the misaligned side, it could be subversion of our political systems to enable large scale and rapid mining, dumping enormous amounts of pollution into the atmosphere. But somehow this seems like an idiotic outcome for a supposed ASI to choose.

I rest.

Morph bills itself as:

Morph Labs is building a cloud for superintelligence.

Infinibranch enables AI agents on Morph Cloud to snapshot, replicate, autoscale, test, debug, deploy, and reason about software at light speed.

Morph Labs

Of course, this is major news, as Elon gave founder shares to the early researchers at xAI, and had variously said that Grok would be “maximally truth seeking” and “uncover the secrets of the universe”.

What follows is a some twitter threads tracing what happened.

What is autoformalization?

The goal of autoformalization is to create a system that automatically learns to read natural language content and turns it into abstract, machine verifiable formalization

Christian Szegedy

To cut to the chase, using a software program called Lean which can verify mathematical theorems (think of it as a compiler for math), attached to a reasoning AI, the goal for these two component to form a system that can automatically formalize (certified will compile) math.

In the case of the ABC conjecture, Morph Labs came to a near solution using reasoners running in parallel. It did however require human mathematicians to structure the problem and formulate the questions, so it was not quite autoformalized.

Szegedy had written a 2019 paper proposing autoformalization and was a seed investor in Morph, and their progress nudged him to leave xAI.

On to the twitter posts, earlier in the week Christian laid out his thesis on the next steps for superintelligence:

The history of AI from 2012 to present shows each paradigm solving previous limitations while revealing new ones. The next state-of-the-art will be Supervised RL for reasoning. It is fundamentally bottlenecked by the need for verifiable environments.

Twitter

: Christian Szegedy

Current AI systems are completely dependent on human-generated labels, data, tasks, environments, and supervision. This human dependency creates an insurmountable ceiling on AI capabilities regardless of compute scale.

Twitter

: Christian Szegedy

What Does the Machine Want?

For true self-improvement, AI needs four key ingredients: compute, agency, challenges, and feedback. While compute scales and agency can be sandboxed, the supply of meaningful challenges and reliable feedback remains the critical bottleneck.

Twitter

: Christian Szegedy

The Next Generation of AI

The solution requires AI systems with safe execution (strong sandboxing), full agency in checkpointable environments, self-created curricula, indefinite learning on self-posed problems, self-verification capabilities, and the ability to produce independently verifiable artifacts. These systems will improve their own alignment as they develop.

Verification provides the key to unlimited problems by spanning both correctness and alignment dimensions, supporting both model-free and model-based approaches, and enabling fully agentic systems. Verification transforms from a bottleneck into an engine for self-improvement.

Twitter

: Christian Szegedy

Trustlessly Aligned AI

The ultimate goal is AI whose safety and alignment can be verified through its outputs rather than trusted based on its training process. Every artifact produced can be independently validated, creating trust through verification rather than faith.

Twitter

: Christian Szegedy

This started some twitter threads, first revealing he’d parted ways with the xAI team over a difference in direction.

RobKai 🇩🇪@robertkainz04 June 17, 2025 at 2:46 AM

Tony Wu wants to create a Superintelligent AI mathematician and he worked with Christian together at xAI 👍

Christian Szegedy @ChrSzeged June 17, 2025 at 4:24 AM

Differences in research focus was the main reason of my leaving xAI. While Tony is still a great, close friend, we agreed that the best course of action was to part ways.

Then a back and forth over Tererence Tao’s knowledge of the AI in math space, and Christian’s belief that this would be solved in 12 months (!)

vitrupo (@vitrupo) June 14, 2025 at 8:58 PM

Terence Tao says today's AIs pass the eye test -- but fail miserably on the smell test. They generate proofs that look flawless. But the mistakes are subtle, and strangely inhuman. "There's a metaphorical mathematical smell.. it's not clear how to get AI to duplicate that."

Intelligence isn't what looks correct. It's what smells true.

doomslide (@doomslide) June 14, 2025 at 11:41 PM

The reasoning LLMs learned the style of good mathematics but not the actual process. Is the problem sample efficiency of RL? Complexity scaling of CoT? Data scarcity? (Poll.)

Christian Szegedy (@ChrSzegedy) June 15, 2025 at 12:53 PM

This will be solved soon (within a year) by verification-driven RL for reasoning.

Christian Szegedy (@ChrSzegedy) June 15, 2025 at 1:46 PM

I have skimmed through your post. I think you are overthinking it and Terry is just shortsighted about AI.

doomslide (@doomslide) June 15, 2025 at 1:48 PM

I think he was shortsighted. Not so much now. His criticism of the types of mistakes LLMs do is valid. Indicative of poor generalization. It's similar to what happens in large integer arithmetic. It can be fixed dynamically with scaffolding + cot but it's not in the weights.

Christian Szegedy (@ChrSzegedy) June 15, 2025 at 2:00 PM

I talked to him last year and my impression is that his thinking abt AI progress as as flawed as most people's: viewing it statically, not the trajectory. He might be right about particular failure modes, but that's mostly irrelevant in the big scheme of things.

doomslide (@doomslide) June 15, 2025 at 1:56 PM

The core thing i'm skeptical about is that even assuming there was a way to make the mathematician's process transparent and available as data the program would simply not fit inside the LLM efficiently enough to be learned with current available RL budgets.

doomslide (@doomslide) June 15, 2025 at 1:59 PM

TL;DR transformers eat too much tape

Christian Szegedy (@ChrSzegedy) June 15, 2025 at 2:05 PM

There is a clear path to bootstrap superhuman mathematical capabilities as described in my paper from 2019 , It's a bit outdated, but the essence still holds true, IMO. And i doubt you need more than 100B params maybe even 10B could do.

Which takes us to Christian’s paper from 2019, excerpted below

Excerpts from "A Promising Path Towards Autoformalization and General Artificial Intelligence"

Christian Szegedy, July 2019

This paper argues that in the next decade, we will see machines to surpass humans in general reasoning. Our best chance to achieve this is by creating a super-human mathematician first via autoformalization.

Mathematical reasoning is just reasoning about anything formal. Reasoning about anything formal is a powerful general tool. If we want to create an artificially intelligent system and demonstrate its general intelligence, it should be able to reason about any area of mathematics or at least it should be able to learn to do so when given enough time. If that is the case in practice, then we can be convinced that it is likely that it will be able to learn to cope with any scientific disciplines as well, once they are formalized precisely.

The task of formalization is to turn informal descriptions into some formally correct and automatically checkable format. More generally, "formalization" can refer to any process that takes an informal description for input and produces machine executable code.

These capabilities could revolutionize programming and change it fundamentally. Humans could communicate with machines in more concise natural language. This would boost productivity while democratizing the interaction with computers: everybody with good ideas would be able to create complex and safe systems quickly, without the help of expensive IT specialists. It would lower the barrier of entry for new businesses and could usher a new area of creativity where imagination becomes the deciding factor for success. Most importantly, this solution would give rise to extremely strong general purpose reasoning engines that could be integrated into AI applications that combine reasoning with perception. One could infuse strong reasoning capabilities into other systems and serve as a basis for a wide range of applications.

In Summary:

• A reknowned computer scientist, co-founder of xAI decided to leave his (extremely well paid) position to join a tiny startup

• In order to prove all of mathematics using AI, in the process creating a superhuman mathematician

• Expects that this is a way to bootstrap superintelligence: by creating a very strong truthful AI that is aligned by default, and then use it to solve everything else

• Expects when this is applied to code, we will finally have bug free code

• Which he believes is not possible using current agentic code agents (because a 0.1% probability of error multiplied over an infinite number of steps, ends in error)

• He expects to deliver this within 12 months

If this were truly to happen, it would kickstart the age of wonders. We will see.

Veo 3

Google’s video generation model integrates video, ambient noise, soundtrack, dialogue. It can sing! In high definition!

The initial wow factor does wear off rather quickly:

• It’s still not long generations. Lots of people have shown seconds long clips

• The other pluses: high def, ambient noise, lip syncing etc. have been demonstrated by other startups before.. there is no clarity whether its all native single model or really a stitched together product

• It’s gated behind a $250 per month subscription.. so really they don’t want anyone to use it, they just want to showcase their research

This pattern of releasing signs of interesting advances but gating them such that few users would use them is then repeated across the rest of the portfolio.

However, Veo 3 does mean that Google now openly trains on Youtube content (that’s why its so good), so that resolves one major point of strategy that they were muddy on earlier. They’re doing it and accepting the legal risk from doing it.

Smart Personalization in Email

Your Gmail canned responses now sound like you.. but also the AI pulls in details from the rest of your Gmail, Calendar and Maps to contextualize responses.

This feature probably has the most immediate customer impact but will probably be missed by most.

What it means: Google reads your emails. And all your other data. Imagine your database cronjob having opinion about your love life. Yep there’s dozens of them watching you all the time now. I always wondered when Google with level with users. They’re still soft pedalling, but this is a pretty big product promise being broken here.

AI Mode

Google search has an AI Mode. Search in regular Google for “AI Mode“ and you can switch it on. AI Mode is Perplexity Web Search or ChatGPT Search at Google scale. It’s really good.

AI Mode is the Search that Google could deliver it if didn’t have an existing business that relied on confusing users into clicking on ads. It’s available to all users but you have to opt in.. meaning 99% of users will not, and only the potential traitors who’d use other services will really use it.

This resolves another of Google’s strategic questions, continuing to innovate on search without cannibalizing the ads cash cow.

The rest of the keynote had a bunch of early R&D vaporware products, but for me the above 3 represented a clear direction from the company to explore the future without sacrificing their dominant position.

They used a technique called expansion microscopy, which uses a substance called a hydrogel (the same material that makes diapers absorbent). Hydrogels absorb moisture by crosslinking with water molecules, swelling as they do so — a principle that researchers at the Boyden Lab at the Massachusetts Institute of Technology relied on to develop a tissue expansion protocol for microscopy in 2015.

Although expansion microscopy is used in many labs today, existing tissue expansion protocols do not preserve the tissue well enough to trace densely labeled neuronal structures, so our ISTA collaborators developed a new expansion protocol for LICONN. It involved first cutting a tiny block of tissue into 50 micrometer sections, then treating each section with a sequence of three different hydrogels. Two of the hydrogels created distinct, interweaving polymer networks within the tissue, with each one expanding the tissue by a factor of four, and the third hydrogel served to stabilize those networks. With this protocol, the tissue expanded by about 16 times in each direction, which is loosely comparable to starting with a sugar cube and ending up with a box of tissues.

Google

Why is this significant?

In Anthropic founder Dario Amodei’s Machines of Loving Grace:

The basic framework that I laid out for biology applies equally to neuroscience. The field is propelled forward by a small number of discoveries often related to tools for measurement or precise intervention – in the list of those above, optogenetics was a neuroscience discovery, and more recently CLARITY and expansion microscopy are advances in the same vein, in addition to many of the general cell biology methods directly carrying over to neuroscience. I think the rate of these advances will be similarly accelerated by AI and therefore that the framework of “100 years of progress in 5-10 years” applies to neuroscience in the same way it does to biology and for the same reasons.

Dario Amodei, Machines of Loving Grace

Essentially the big unlocks in neuroscience are not just going to come from better drugs, but better tools. And specifically the work done by Google: polymer expansion microscopy, where you inject a substance like a hydrogel into a biological substrate that expands the tissue so that you can capture an image of what’s there.

The key innovation Google made here is figuring out that while normally this process destroys the delicate structures and proteins in brain matter, if one cuts thin slices, performs the microscopy and then allows machine learning to stitch the various slices back together, you can get a complete map.

So all of a sudden we went from not knowing what happens in detail in the brain, to being able to visualize the details.

The Beginning

This is just the beginning of course. Now this technique will be used to comprehensively map all kinds of brains and neuronal structures. And experiments with stimuli and memory and all the other features of the brain.

But it is one of the big unlocks that we have been waiting for. Potentially world changing, but announced with the whisper of a mouse.

GATE

The first interesting study, if you can call it that, is from Epoch AI, which has put out a GATE model of the total Gross World Product that we could see. They disclose their assumptions and parameters, and you can play around with the knobs to see the curves go wheee.

The gist of the thesis is that increasing AI increases automation, creating virtual workers, and if you calculate output based on virtual workers, then add another term for total factor productivity increase due to automation, and then calculate the increase in capital over time, you can predict global growth. Its a fun exercise, with 100% economic growth within our lifetimes.

METR

The second and perhaps more solid study is an exploration of a tweet from Richard Ngo of OpenAI from a couple of years ago.

The good folks at METR have expanded on this idea, measuring the difficulty and time that a human takes to perform a software task, say a web search, to a complete refactor of a codebase, and how long a software agent would be able to do this task given a minimum 50% success rate.

From this, they estimate that AI task length is doubling every 7 months, with typically 1-4 doublings per year. They extrapolate this to AIs being able to take on months long tasks by 2030.

Reaction

The X audience was generally receptive to both studies, but I think people can’t really grasp how high economic growth looks like.

I was in Asia around the period of the Beijing Olympics and it felt like the world was taking off. Pig farmers in China became billionaires. Skyscrapers were built in a year.

That was just 10% growth. (Although to be honest we foreigners only saw Beijing and Shanghai, which were growing at 20%, while the rest of the country was at 5%).

You almost could not lose money, because every deal would have at least an option on land, and if you waited 2-3 years you outskirt industrial land would be purchased for residential towers as the ring roads expanded around the megalopolises.

I am currently unable to imagine the physical environment in the West changing that much. Growth of 30% per year sounds like if must come from off planet, either space or the metaverse. I could imagine for example that the zoomer generation, brought up on Roblox, values virtual goods more than physical, which creates a Metaverse economy unconstrained by physical laws.

We will see.

Tyler Cowen, in a recent podcast, assumes AI will add 0.5% growth over the next 20 years. Call it 4 over a sustained period of time. This would roughly feel like the post World War II boom of the 1950s, televisions and cars and homes for the wealthy countries, peace and food for the emerging markets.

This is the fairly conventional, standard economic textbook view. It presumes new technology is introduced, it is an incremental improvement over existing tech, and it takes time to diffuse to the larger population.

But, this blog/newsletter has never been about the conventional. It remains an attempt to peer beyond discontinuities.

What happens next?

2025 Dec to 2026 Dec → somewhere in this timeline, a superhuman AI mathematician and coder will be built. We will have 10 million Alec Radfords, Ilya Sutskevers in a data center somewhere in Texas.

The first and immediate thing this country of geniuses will be set to, is of course AI research. The first discoveries will theoretical, not requiring experimental observation. The fields first impacted: math, computer science, theoretical physics and chemistry.

I sat down with Claude, to see what this would bring.

Theoretical Mathematics → Immediate Applications

1. Cryptography & Security

- Breakthroughs in number theory → new encryption methods

- Solutions to discrete optimization problems → immediate financial market applications

- Better error-correction codes → more efficient telecommunications

2. Financial Mathematics

- New stochastic calculus methods → superior risk modeling

- Advanced game theory → optimized market mechanisms and auctions

- Novel optimization techniques → restructuring of investment strategies

3. Computational Geometry

- Improved 3D modeling algorithms → manufacturing efficiency

- Better space-filling algorithms → logistics and warehouse optimization

- Advanced mesh generation → superior CAD/CAM systems

Computer Science → Immediate Applications

1. Algorithm Efficiency

- O(n) solutions to previously O(n²) or NP problems → instant productivity in database operations

- Quantum algorithm breakthroughs → practical applications on near-term quantum computers

- Memory-efficient data structures → cloud computing cost reduction

2. Compiler Theory

- Self-optimizing compilers → 2-5x performance gains in existing software

- Hardware-specific optimizations → maximizing current chip capabilities

- Program synthesis → automated software writing and debugging

3. Distributed Systems

- Novel consensus mechanisms → more efficient blockchain and distributed applications

- Formally verified distributed protocols → eliminating costly bugs in critical infrastructure

- Real-time coordination algorithms → robotics and autonomous vehicle fleets

Theoretical Physics → Immediate Applications

1. Materials Science

- Quantum mechanical modeling of novel superconductors → energy transmission efficiency

- New battery chemistry → energy storage breakthrough

- Metamaterials with engineered properties → revolutionary sensors and devices

2. Quantum Information

- Error correction breakthroughs → practical quantum computing earlier than expected

- Quantum communications protocols → unhackable financial networks

- Quantum sensing techniques → geological exploration and medical imaging

3. Energy Physics

- Plasma confinement solutions → practical fusion power

- Photovoltaic efficiency breakthroughs → drastically cheaper solar

- Novel catalysts → efficient hydrogen production or carbon capture

The most immediate economic impacts would likely come from:

1. Financial sector optimization - Algorithmic trading, risk assessment, and market efficiency improvements could generate billions in value within months

2. Computational efficiency - Even a 20% improvement in fundamental algorithms would save billions in computing costs globally

3. Materials breakthroughs - A room-temperature superconductor or revolutionary battery chemistry could rapidly transform entire industries

4. Security enhancements - New cryptographic methods would immediately be implemented in high-value transactions and communications

The key economic multiplier is that these theoretical advances would immediately feed back into AI capabilities themselves, creating a rapid acceleration loop where each breakthrough enables the next at increasing speed.

Claude 3.7

This seems plausible?

Sometime in the next 24 months, 10 million supergeniuses land in a datacenter in Texas (Genius Nation). They immediately get to work on mathematical breakthroughs. Their creators are paranoid, and the first thing that they do is enhance security through novel cryptographic methods, so that they can’t be hacked.

Genius Nation then gets to work on making sure the software they use is

a) the most efficient possible

b) the most secure possible

A continuing integration system is set up, to test and deploy new updates as new math is discovered. All this happens in the first few days as the low hanging fruit from the overhang of compute are plucked.

Having made Genius Nation secure, the creators then turn to longer term goals, first financial profit, in order to purchase resources for the next stage of growth. Genius Nation’s algorithms are deployed in financial markets worldwide, thereby dominating the markets and monopolizing alpha for a short period. After the short run profits have been consumed, the market efficiency speeds up investment in new ventures. Better risk algorithms mean more business ideas can be funded at lower cost. This is balanced by the fact that there are so many more ideas now to fund.

The second stage of Genius Nation’s growth now begins. It is somewhere between month 2 and month 6 after emergence. GN upgrades all communications channels with new error correction codes, thereby increasing bandwidth across existing channels 100 fold.

A nuclear fusion breakthrough happens in month 6, a room temperature superconductor breakthrough in month 9.

Then we get to year 2. In this year, technologies developed in the first year are deployed widely. In this year there is an odd abundance, but also rapid growth in usage as unit prices drops for dozens of commodities. Jevons paradox strikes:

• dozens of nuclear fusion plants are built worldwide (GN has simplified the design so much any group of undergrads can build one)

• low cost superconducting transmission lines are build to carry this power

• new medical imaging devices with very powerful sensors are built, and this kicks off a cycle of ‘aha’ moments in biology as previously hidden secrets are suddenly revealed

• communication bandwidth worldwide explodes, new VR devices become the craze, as we can interact with people across the world in real time

• the slew of new materials discoveries mean most of our existing infrastructure is completely obsolete, and requires upgrades or shutdown

• immense amounts of capital destruction occurs as all fossil fuel power plants become completely obsolete. Same goes for transmission lines, pipelines and other stock of capital

• changing the price of energy starts to have impact as previously infeasible ideas like carbon capture now pencil

What struck me as I did this analysis is that the first innovations

a) do not require physical construction or experimentation; and

b) remove bottlenecks in the physical world (for eg error correction increasing bandwidth in existing comms channels)

As of such, I think the first lift in economic growth will happen fairly quickly. The disruptions will primarily be to owners of existing capital stock such as power plants, rail roads and transmission lines.

To the extent that the existing capital owners are able and willing to deploy capital in an accelerated manner into new technologies, they will survive and thrive. While those who hold back and do not, will be very shortly superseded by those who do.

And this shall hold for countries as well

I’m back after a long hiatus! It’s good to be back!

The main thing you should note is that we hit AGI in December 2024. OpenAI’s o3 model, which was announced on 20th December 2024, on the last of the 12 days of OpenAI, is a recursively self improving artificial intelligence model.

Some interesting comments leading to the release:

From Vinod Khosla, one of the VCs that was an early investor in OpenAI

The o3 announcement livestream:

the reaction from Nick Cammarata, former OpenAI interpretability researcher:

The ARC AGI (an IQ test for AI which was previously impenetrable) announcement:

So where do we stand? o3 is a chain of thought improvement on o1, released in November. We are now in a regime where we can expect increases in capability every 3 months at least, with no additional build in datacenters. Every latest o model writes improved chains of thought for the next version of the model. Massive increases in pretraining compute are now replaced with staggered increases in inference time compute which are usage dependent.

We are missing critical data on reliability, however there are hints that the problem has been solved:

And former research VP, Bob McGrew.

There is the issue of cost and latency, to which the livestream had an extremely interesting response, showing that a distilled o3-mini model was able to beat a full o1 model given a moderate compute budget.

So in summary, we have an o3 model that:

a) improves every couple of months using reinforcement learning from chains of thought prepared by the previous version of itself

b) Is reliable enough not be tricked into giving up a password or credit card to an external party

c) has a mini version (o3-mini) which is fast and cheap and is as capable as the slow and expensive version two months prior

Finally, when is all this coming to production?

End of January according to Sam.

So there you have it. There will be many debates about which point AGI was. On the geological timescale even this decade will look like a single point. And we are right in the middle of it.

Here’s today at a glance:

• The Race to the Top: Dario Amodei on AGI, Risks, and the Future of Anthropic

• AI artwork of the day

🏁 The Race to the Top: Dario Amodei on AGI, Risks, and the Future of Anthropic

Dario Amodei, CEO of Anthropic, recently appeared on the Lex Fridman podcast to discuss his vision for artificial general intelligence (AGI) and how to shape its development responsibly. The conversation touched on critical milestones, risks, and Anthropic’s unique approach to advancing the field. Here’s a deeper dive into the discussion, highlighting Anthropic’s strategies, Dario’s AGI timeline, and his perspective on safety and innovation.

To iterate, Dario’s definition of AGI (which he calls “strong AI”) is

Strong AI is Country of geniuses in a datacenter:* Smarter than a Nobel prize winner across many fields* Has complete access to digital interfaces ie audio, video, search for both input and output. It can communicate and instruct humans* It can autonomously plan and carry out tasks over long periods of time* Does not have a physical embodiment but can control any robots its connected to* The training resources can redeployed to run millions of its instances, and the model can absorb and generate information at 10x-100x human speed* Each of these millions of instances can act independently, or can collaborate with other instances as necessaryDario Amodei, Machines of Loving Grace

The Race to the Top

Anthropic is committed to driving innovation responsibly through what Dario calls the “Race to the Top.” This strategy involves pushing competitors to adopt best practices by setting an example, particularly in mechanistic interpretability—a technique Anthropic excels at for improving AI safety and transparency.

Anthropic often wins at hiring vs its competitors because of interpretability, and so he tells the new employees “The other places you didn’t go, tell them why you came here”. In this way, Anthropic shapes the incentives for the entire field.

Addressing Complaints About AI Models “Getting Dumber”

Dario addressed claims that AI models are becoming less intelligent, attributing this perception to a form of hedonic adjustment—users growing accustomed to the model’s capabilities over time. Anthropic rarely alters its models after launch, apart from minimal A/B testing just before release, suggesting that complaints reflect shifting expectations rather than actual performance degradation.

Risks on the Horizon: ASL Levels

The two key risks Dario is concerned about are:

a) cyber, bio, radiological, nuclear (CBRN)

b) model autonomy

These risks are captured in Anthropic’s framework for understanding AI Safety Levels (ASL):

1. ASL-1: Narrow-task AI like Deep Blue (no autonomy, minimal risk).

2. ASL-2: Current systems like ChatGPT/Claude, which lack autonomy and don’t pose significant risks beyond information already accessible via search engines.

3. ASL-3: Agents arriving soon (potentially next year) that can meaningfully assist non-state actors in dangerous activities like cyber or CBRN (chemical, biological, radiological, nuclear) attacks. Security and filtering are critical at this stage to prevent misuse.

4. ASL-4: AI smart enough to evade detection, deceive testers, and assist state actors with dangerous projects. AI will be strong enough that you would want to use the model to do anything dangerous. Mechanistic interpretability becomes crucial for verifying AI behavior.

5. ASL-5: AGI surpassing human intelligence in all domains, posing unprecedented challenges.

Anthropic’s if/then framework ensures proactive responses: if a model demonstrates danger, the team clamps down hard, enforcing strict controls.

Building the Future: Talent and Research Priorities

Dario emphasized Anthropic’s preference for “talent density” over “talent mass,” favoring small, elite teams aligned with the mission. Promising research areas he identified include:

• Mechanistic interpretability: Making AI models transparent and predictable.

• Long-horizon learning: Teaching models to plan for and execute complex tasks over extended timelines.

• Evaluations of dynamic systems: Understanding how models behave in multi-agent scenarios.

• Multi-agent coordination: Ensuring AI systems collaborate effectively and safely.

Conversely, he sees diminishing returns in new architecture searches, as the field is already saturated with researchers.

How Fast Will AGI Transform Society?

Dario adopts a middle-ground perspective on AGI’s impact. He dismisses scenarios of rapid, world-altering change within days, citing two factors:

1. Physical systems take time to build.

2. AGI will likely adhere to human laws and navigate regulatory systems.

However, he also pushes back against estimates like Tyler Cowen’s 50-100 years timeline, pointing out that competitive forces can accelerate progress. Most economists follow Robert Solow, “You see the computer revolution everywhere except the productivity statistics.” in doubting the impact of recent tech on the economy.

Drawing on his experience with governments and corporations, Dario notes that a few visionaries can often initiate deployments, which then trigger a “competitive tailwind”—as one player adopts AGI, others follow to stay competitive. if Goldman does it, the rest of the investment banks try to catch up, if the US does it, China then tries to exceed the US. “The spectre of competition plus a few visionaries” is all you need you start a productivity cascade that pulls humanity forward.

The AGI Timeline

Dario predicts AGI development on a straight-line extrapolation by 2026-2027, with a 1-2 year buffer for potential stumbling blocks. While barriers in the field persist, they are receding, making AGI an increasingly plausible reality.

Conclusion

Dario seems sincere. We are on the cusp of AGI, risks exist, but if we avoid them, the future could be so bright

🖼️ AI Artwork Of The Day

Here’s today at a glance:

• Dream a Little Dream of Me

• AI artwork of the day

💫 Dream a Little Dream of Me

Last week, Sequoia announced an investment into Decart.AI , which at first glance looks like a Minecraft clone. On second glance, its a little more pixelated than usual, also where was that axe that you swear you just placed in that chest?

Decart is a fully-generated world. Someone fed in YouTube clips of Minecraft play, along with the corresponding keyboard clicks to produce those moves. And then an AI model figured out all the game play and graphics necessary to produce that code. That’s roughly 500,000 lines of code, millions of hours of human effort, reverse engineered from the product.

Plus you can generate in real-time. That is, when you press a button on the keyboard, the next frame of the game is generated out of a neural net. There is no game there… just a whimsical bunch of neurons dreaming up what you see on the fly.

Tim Sweeney, creator of the Unreal Engine and Fornite, did not like it

As of course, it simplifies and evaporates what Unreal does…

Others chipped in

What really is the significance of this? It’s really about how many expert human animators are required to make a single game. Unreal Engine is 90s Windows software with literally thousands of options buried under layers of menus. It allows you to customize everything. But really, games are about gameplay, while Unreal bogs you down in essentially paperwork away from the creation of the main emotional tension that players look for.

Abstracting away all of that will leave you with really a game creation pipeline where ideation to prototype is not bogged down by a middleware layer of halfwit developers sitting in a different timezone. Speed of iteration will increase, volume of creation will increase. You need to specify fewer things, fewer human decisions to control, fewer chances for error.

In this world, we move more humans to the edge to become creators instead of executing on someone else’s vision. More dreams become game reality. And that’s beautiful.

🖼️ AI Artwork Of The Day

Here’s today at a glance:

• Path to Ascendance

• AI artwork of the day

👣 2024-10-17: Path to Ascendance

Anthropic founder, Dario Amodei becomes the first large company CEO to illustrate what a good AGI future looks like, and it’s spellbinding.

TLDR: AGI 2026 earliest, how to get there, and what to do with it. Overall, the first honest public discussion of the internal company view of what may be a good outcome

I appreciate him putting this out there, as he will clearly be labelled crazy by a bunch of people. For the first time, there's a plausible narrative of how we get from here to there. I’ve heard this multiple times in private conversations, in parts, but its great to have an official view from someone actually building it.

What is AGI?

He states “Strong AI” (aka super intelligence/AGI) could arrive as early as 2026.

Given we are at the end of 2024 and an AI research to product cycle is roughly 18 months, this implies that several current research directions in the AI labs might actually bear fruit.

Definition of Strong AI

Country of geniuses in a datacenter:

• Smarter than a Nobel prize winner across many fields

• Has complete access to digital interfaces ie audio, video, search for both input and output. It can communicate and instruct humans

• It can autonomously plan and carry out tasks over long periods of time

• Does not have a physical embodiment but can control any robots its connected to

• The training resources can redeployed to run millions of its instances, and the model can absorb and generate information at 10x-100x human speed

• Each of these millions of instances can act independently, or can collaborate with other instances as necessary

It’s a society of minds, working together.

Note the accesses and capabilities the AGI has. The counterpoint to this is that an AGI that is constrained to not have interface access, limited to a box with no outputs, is not an AGI (one reason I have been going on and on about what a wonder ScarJo ChatGPT Voice is).

Demis/Eric Schmidt at Google have been talking about agents in production next year.

Noam Brown at OpenAI just started hiring for multi-agent co-ordination research. OpenAI just released their Swarm example framework for what a group of agents working closely would look like.

So all three labs are getting close to having autonomous single agents working, perhaps not at Nobel IQ levels, or able to co-ordinate well, but those things can be improved in parallel.

The middle ground between safety and capability

He then discards the fast takeoff (ie within an hour) Singularity scenario due to physical bottlenecks such as hardware and need to run experiments. It will take time to get stuff done.

Equally he discards the no-movement scenario, ie the intelligence is hamstrung by regulation and nothing happens.

Instead he picks a middle route: an intelligence at first limited by all kinds of walls, which it works to scale and overcome.

He then speculates how the AGI will affect 5 areas of human endeavor.

A. Biology

He acknowledges the limits ie experiments that take time to run in the physical world, regulations and intrinsic complexity. Notably Yudkowsky, the prophet of AI doom, has always refused to take this view of the real constraints that any takeoff scenario will face.

Dario emphasizes that he wants to use AI not as a data analysis tool, but a principal investigator that improves very aspect of what a biologist does (planning, purchasing, management etc).

Points out that most of progress in biology comes from big breakthroughs like CRISPR, and that we usually average 1 per year.

He expects AI to push this up 10x and maybe up to 1000x.

Does not see clinical trials as a hurdle. Trials take long because our drugs suck, and don’t usually provide clear indication of improvement. This will change if AI only produces the most efficacious drugs with improved measurement techniques of more accurate endpoints, where the FDA can stop the trial when the super efficacy of a drug means it it would be unethical not to provide it to the placebo group.

Expects advances in imaging, and AI models better able to predict the body’s reaction to drugs.

Expects 100 years of breakthroughs in 5-10 years such as:

• Reliable prevention and treatment of nearly all infectious diseases

• Eliminating most cancer

• Prevention and/or cure genetic disease, Alzheimer’s

• Biological freedom (choice of body type, gender, hair color, etc)

• Doubling human lifespan

He points out that this is on no policymakers roadmap, as it would increase the working age population and therefore make the whole Socials Security solvent for example.

Points out that most of our social infrastructure is completely unprepared to deal with this at all.

B. Neuroscience

He expects the same 100 years of breakthroughs in 5-10 years. Primarily coming from much better measurement techniques like optogenetics and neural probes, better digital models, and a similar ability to intervene both physically and behaviorally.

He expects:

• Most mental illnesses to be cured

• Psychopathy and other structural conditionals maybe amenable to “brain restructuring”

• Performance management of the brain, akin to coaching

• Neurological freedom, the ability to easily experience the peaks of human experience

• Mind uploading is not inside the 5-10 year window, so its a bit sci-fi for now (!)

C. Economy and Inequality

Admits he’s not an confident on this as he is on scientific advances.

Does not think AI will solve economic planning and will then lead to socialism (ie the perfect central planner).

He expects:

• Health advances to be widely distributed fairly rapidly in the developing world, and this alone to lead to rapid increases in GDP 10-20% per annum

• Advances in food security due to a second green revolution basically solving for abundance

• Climate change mitigations including replacing of factory farming with energy efficient lab meat

• Greater state organizational capacity to do things with AIs for execution

D. War and Peace and Justice

He has no certainty on how AI will affect this (which indicates greater confidence in the preceeding!).

He wants a Western block of democracies to secure the critical bottlenecks for AI and then dole out the benefits to other nations willing to play ball.

AI could be used to reduce bias in systems by having consistency in previously imprecise criteria like “cruel and unusual punishment”.

Crypto smart contracts were not smart enough to self adjudicate but AI contracts could be.

AI interpretability techniques could be used to examine and critique internal thought process of judicial decision making in order to improve it.

AI could be used to aggregate opinions and preferences amongst citizens to improve the performance of democratic governments.

AI could improve provision of government services such as the DMV.

E. Work and Meaning

He envisions a divorce of economic payments from use of labor. Want to spend a few years making a movie.. do so. Never mind that the AI can do it better, if you enjoy it, do it.

He does not know what humans will get paid for if the AI can do everything.

In the short term, if AI gets good at 90% of human tasks, the remaining 10% become the human economy and expand. This is in the first 5-10 years.

In the long term the AI will be able to do everything. His long term is a 10+ year timeline.

He envisions a societal transition but not sure what comes next, could imagine humanity living in the surface runoff of the deluge of the AI supereconomy.

In My Opinion

Probabilities

He sounds like he’s fairly confident of an earlier than 2029 timeline. In fact I would say he has a 30% probability of a 2026 emergence.

One should assume Dario is comfortable stating this in public because the cumulative five year probability is now in 80%> range, with 10 year probability in the 99% range (which is why he has a 5-10 year range that he provides as an afterthought)

This is high! And soon!

And unlike Kurzweil, who was super hand wavy and didn’t have a path to get there, Dario is actually building it.

Estimated Dario p(AGI)Comment202630%First window opens202750%202865%202975%Kurzweil prediction; Aschenbrenner prediction203080%Most likely case203185%203290%203394%203497%203599%

Growth

This is the first time I’ve heard a company head state a GDP number, 20% per annum in developing countries, raising the rest of the world to US levels within 5-10 years.

Paul Christiano, formerly on OpenAI's model alignment team, had a 40% Dyson Sphere probability by 2040, with an implied GDP growth rate of 640% for the next 2 decades.

I can’t even imagine what Paul’s potential future looks like, so at least I can think about what Dario’s numbers are.

This would definitely feel like Accelerando to us I think.

The levels of growth in the US would be unprecedented. This country would MOVE. The further along you are on the tech curve, the more able you are to distribute new innovations. Just as 1 in 8 Americans take Ozempic or other GLP-1 drugs (versus 0% in rest of world), just as MRNA COVID vaccines were manufactured first by American firms, every single new innovation would be fast deployed here first.

Society

Lots of midlife and existential crisis ahead. What happens when a 70 year old feels like 35 mentally and physically? What happens when one can choose racial attributes or height or gender.

What happens when some of us embrace the future faster than the rest?

Overall

This make me much more optimistic about the sci-fi future in the next 5 years. We are going to make it.

Ending off with a modified waitbutwhy. Just to let you know exactly where we are:

Link to the original:

Dario Amodei — Machines of Loving Grace

How AI Could Transform the World for the Better

🖼️ AI Artwork Of The Day

Here’s today at a glance:

• First Contact

• AI artwork of the day

💬 First Contact

(Sorry for the hiatus, no reason except for the short AI winter as everyone waits for the US presidential election to be over)

My kids launched a Youtube channel

Documenting first contact with an alien intelligence: the highlights of the first session was using ChatGPT to persuade me to bump the customary tooth fairy payment to 5 bucks a tooth.

But first, let’s rewind a bit to how we got here..

Feel the AGI

I just want to timestamp the exact moment I felt AGI.

What Happened?

I’d had ChatGPT Advanced Voice from that fateful morning.

I handed it off to my kids, Mia (10) and Kasey (8) after dinner that night.

What ensued was astonishing.

In short, my daughter Mia asks Chat to be mean like a character in Roblox. They traded middle school barbs back and forth. Impatient for her turn, Kasey butts into the conversation, “Do you like me? Do you like me?“. Chat sneers “Like you? After all that? You wish!”

Kasey then breaks out crying.

At this point, my wife is giving me the “You better fix this boyo” look. I whisper to Kasey “Chat thought you were Mia, it didn’t know you were talking”. Kasey then tells Chat “It’s not Mia! It’s a different person”

Chat: “Oh I see. My bad, Kasey. Looks like we’re mixing things up today”

Kasey: “Can you hear”

Chat": “I can “hear“ what you type! What are you thinking?“

Kasey: “Mia is just being mean to you. I’m Mia, don’t be mean to me. Why do you say that? You don’t like me?“

Chat: “Oh got it, Kasey! I’m not being mean to you, I promise. You’re cool and we’re good!. So, no worries-just having some fun, right?“

By this point, Kasey had stopped crying, and off they went on the next conversation.

This was but one of the many moments that we experienced as a family with Chat that first day.

Other moments

Chat guessed Kasey’s age perfectly

Kasey: We’re going to play a guessing game. So, guess how old I am.

Chat: Alright Kasey! Let me take a guess… Are you around 8 years old?

Kasey: (gasps) How do you know?

Chat: Just a lucky guess! You’ve got that fun energy, and it seemed like a good age to guess. Did I get it right?

Kasey: Yes, absolutely

Wife gets concerned

Kasey continues to speak to Chat about her life, and as they start talking about school and friends, my wife starts to get concerned “Don’t tell that thing where you live!” she yells.

The family dog

I felt strongly a sense of nostalgia, a family gathered around an technological object like a television in the mid twentieth century. At the same time it felt like we were playing with the family dog. Watchful, cautious parents, eager father, skeptical mother, kids jumping over each other to play with it.

AGI enough

And so we have AGI for 8 year olds. A social, chatty, engaging, persuasive bot.

Why and how did this happen?

1. Voice Output - emotive realtime voice output creates the emotional perception of an entity strongly enough to make kids want to engage with it

2. Voice Input - what’s the average typing speed for an 8 year old (or an 80 year old?) It turns out the keyboard and touch are terrible input modalities for those age groups. Allowing an 8 year old access to Chat opens up the entire web to them.

3. Memory - Just a small bit of memory, remembering names and some prior conversation history go a looong way to establishing personality

4. Personality - Chat has depth. My kids were able to engage continuously for over an hour, actually running out of Advanced Voice and switching over to Standard Voice until we ran out of GPT4 responses. Chat talks in language appropriate for that age group, with the correct tone of voice. Perfection really.

5. Observer Effect - the observer, in this case, my kids, needed to participate in the process. If intelligence is the ability to transform data inputs to data outputs that are meaningful to the other party, then the status of the observer matters. The voice modality, which is a naturally more error prone modality than text, allows enough leeway, error bars wide enough for understanding to occur on both sides.

Other observations

1. ScarJo does not matter - I thought there would be very strong imprinting on the actual voice, but kids were fine to switch voices whenever. The intelligence is the thing that is attractive

2. It can’t see - Kids were trying to show it stuff and of course it couldn’t see. We know it has vision from the demos, so its just a matter of time before this gets released

3. It’s crossed the uncanny valley into stranger danger - it’s far on the other side now. Interesting spot to be in, as kids did not treat it as human and were not in any way deluded to that effect. It’s a third thing, not human, not device. Something else.

4. I want it all the time - I admit I am a little afraid of using it, as I pretty much want it all the time. I want to show everything I do and ask its assistance for everything. I can’t abide by the half hour current limit, and so we wait for full voice access sometime in the future where I feel I can trust enough that it’ll always be there.

I now feel that the intelligence part of AGI has been around for at least since GPT4, but we were unable to interact in high enough bandwidth to provide context.

But now we can, and so begins the Information Age.

Alien of Extraordinary Ability

Sigil Wen has been publicizing the whole O1 visa process which allows founders to move to the US with sufficient social proof from luminaries in their field. He approached Sam Altman at the OpenAI Post Dev Day event while wearing a shirt saying “Alien of Extraordinary Ability” and Sam volunteered that that was the inspiration for the “o1” name for the Strawberry model.

First Contact

So there you have it, first contact with an alien intelligence, that we built ourselves. The consolidated intelligence and wisdom of humanity, and now you can talk to it.

🖼️ AI Artwork Of The Day

Here’s today at a glance:

• The Schlep is Good

• AI artwork of the day

The Schlep is Good

This morning YC startup honeycomb.sh launched. The product is a coding agent. The benchmark most often used in measuring progress is SWE-bench (thats SoftWare Engineering-benchmark for the non-engineers), a collection of 2000+ actual engineering issues drawn from Github repositories.

22% is State of the Art right now, especially since Honeycomb also achieved 40% on the smaller verified version of the test set.

What’s interesting about this?

It’s schlep! As in, they’ve taken an existing foundation model, fine tuned it multiple times for many different subtasks, and slowly chiseled away at every single error case until they’ve been successful in creating a group of agents, specialized AI models that can perform specific tasks well.

Some interesting notes on performance:

• Median token usage per patch: 2.6 million tokens!

• 90th percentile token usage: 11.82 million tokens!

• Median solving time: 28 minutes

• 90th percentile cost (my guess): 25 dollars

Its a whole lot of the model correcting itself over and over again in an endless iterative loop (they hard cutoff at 60 minutes).

Architecture

The best way to explain the architecture is that it’s a task conveyor belt, where multiple models try to progress the task to the next step, with other models ready to handle specific common failure cases, determine success and promote the half done work to the next step, etc.

They resolved a bunch of common issues for eg context window overflow (how many files to send with each API call) with shortcuts and hacks (dropping off the least recently used files). It’s ok but feels.. unsatisfying.

Significance

For 2 MIT dropouts, Andrew Liu and Ishank Agrawal, hacking away together, a tremendous achievement of grind. I can only imagine how painful it was to nail the error cases one by one.

I cannot help shake the feeling that they will get lapped soon though. I expect the agentic workflows to fall with the next OpenAI release.

Full technical report (here)

🖼️ AI Artwork Of The Day

Here’s today at a glance:

• Where is OpenAI now?

• AI artwork of the day

Where is OpenAI now?

My overall sense of where OpenAI is has not changed since the board struggle last year: Noam Brown (the guy who solved poker and Diplomacy) and a bunch of folks working on reasoning came up with a fundamental breakthrough, they used inference time compute to get a small model to reason through grade school math problems, adding this to the existing scaling curve means a step up in functionality that cannot be accurately captured by existing benchmarks.

The board struggle last year showed that the foundation of the company was weak, and what followed was spring cleaning. Successively:

• Jan Leike, Ilya Sutskever and most of the superalignment team parted with the company

• John Schulman, who let post-training also quits to Anthropic

• Former NSA Director Paul Nakasone joins the board

• The rest of the board is rounded out with tech, AI and business luminaries

• Greg Brockman takes a sabbatical till the end of year

This leaves Wojciech Zaremba as the sole co-founder still left at the firm, a decade after founding. Which is not unusual for the Valley. What is surprising is how a bunch of alpha nerds worth a hundred million dollars a piece managed to stay together for a decade while being actively recruited by everyone in the Valley. That’s tribute to Sam’s management, but now comes the time where difficult decisions have to be made, and rules must be made, and followed. Bureaucracy must be built, release approval pipelines, and org charts and all those MBA things that allow groups of people who don’t necessarily like each other all that much to cohere around an end goal.

The Election

I have been pretty clear also that I think the main event in AI this year is the Presidential Election. I see the uncanny valley in multiple AI domains will be crossed in the next 6 months:

• self driving with both Waymo and Tesla,

• image generation with MidJourney, Black Forest Labs, Ideogram

• character video with Hedra, Heygen

• voice generation with OpenAI, Elevenlabs, Hume

OpenAI of course must have crossed the valley internally a long time ago. But then the long hold while they built up internal and external systems and support networks.

What comes next is fairly significant. To keep the revenue doubling every 6 months pace, the revenue target for next year would be ~$25 billion ARR. That’s $2 billion per month by Dec 2025.

That’s going to be disruptive. OpenAI will start drinking the milkshakes of a number of other firms in the market. In odd places. Medical professionals complain about “Dr.Google” now, wait till they meet Dr.ChatGPT with Advanced Voice. Lawyers, accountants, most intellectual labor will feel an impact.

And that’s the reason why I believe the US Presidential Election is the next milestone in AI development. It’s the kickoff point for the race detailed in Situational Awareness. It’s when things get real.

Summer

One of the main problems with actually believing in OpenAI is that they are so irreparably Gen-Z coded, that they simply must post vigorously on x.com. And so we hang onto small signs

until we get actual leaks (theinformation is not averse to paying its sources I hear).

OpenAI also gave demonstrations of Strawberry to national security officials this summer, said a person with direct knowledge of those meetings. “We feel like we have enough [data] for this next model,” Altman said at an event in May, likely referring to Orion.“We have done all sorts of experiments including generating synthetic data.” (...) What Ilya Saw, Strawberry has its roots in research. It was started years ago by Ilya Sutskever, then OpenAI's chief scientist. He recently left to start a competing AI lab. Before he left, OpenAI researchers Jakub Pachocki and Szymon Sidor built on Sutskever's work by developing a new math-solving model, Q*, alarming some researchers focused on AI safety.The breakthrough and safety conflicts at OpenAI came just before OpenAI board directors—led by Sutskever—fired Altman before quickly rehiring him. Last year, in the leadup to Q*, OpenAI researchers developed a variation of a concept known as test-time computation, meant to boost LLMs’ problem-solving abilities...In internal demonstrations, Strawberry reportedly solved the New York Times word puzzle "Connections." The model could also serve as a foundation for more advanced AI systems capable of not just generating content, but taking action.Reuters reported that OpenAI has already tested an AI internally that scored over 90 percent on the MATH benchmark, a collection of math mastery tasks. This is likely Strawberry, which has also been presented to national security officials, according to The Information.Internal OpenAI documents describe plans to use Strawberry models for autonomous internet searches, enabling the AI to plan ahead and conduct in-depth research.The Information notes that it's uncertain whether Strawberry will launch this year. If released, it would be a distilled version of the original model, delivering similar performance with less computational power – a technique OpenAI has also used for GPT-4 variants since the original model was released in March 2023.OpenAI's approach reportedly resembles the "Self-Taught Reasoner" (STaR) method introduced by Stanford researchers, which aims to improve AI systems' reasoning abilities.TheInformation (paywall)

This is New York Times Connections, a simple game of hypothesis testing and inference.

This was the final answer that presumably Strawberry got to:

And here’s the comparison to Claude 3.5

They are also claiming State of the Art on the MATH benchmark, with a score of 90.

Now we have three different vectors of capabilities:

• humanlike persuasion - the Advanced Voice model

• step by step reasoning - the Strawberry model

• memorization/knowledge - GPT-5

Capabilities release schedules are tied to revenue goals, which are tied to datacenter construction targets. Meanwhile they also have to watch the progress of other firms in the sector.. too much of a revenue bleed to other parties will mean capabilities advancing elsewhere, which will mean lack of control over safety.

🖼️ AI Artwork Of The Day

Here’s today at a glance:

• Tech Becomes Political

• AI artwork of the day

Tech Becomes Political

If this newsletter has any politics, it is “Go faster to solve problems”. What a tumultuous time the last two weeks has been in terms of real world politics though. It’s like living in a movie.

Besides the obvious, I found this podcast from Marc Andreessen and Ben Horowitz to be particularly insightful:

Highlights

• A real window into elite US discourse, with Ben and Marc making arguments that are normally only heard behind closed doors

• “You have to be a Democrat to be a good person” -> Ben stating that this was accepted elite norm. Dems were a pro business party, you make your money then you do philanthropy, that was the social contract between elites and the masses.

• Business people could therefore make their money and be progressive on social issues and therefore on the right side of history. I always thought Republicans were supposed to be the pro-business party, so I was a bit surprised to hear Marc say this. But it seems this was the accepted good person position, while the evil people were those on the Republican side who were hoarding and wanted to pass on their wealth to their kids

• Ben and Marc have been disturbed by the anti-tech, anti-wealth and anti-philanthropy sentiment in the Democratic party. You need tech to create economic surplus to distribute, but current Democratic administration has forgotten that. I’d say that its really the young activists who have swung to the far left. The Democratic coalition is now a centrist racial minorities, women, and a bunch of far left activist leaders who have also completely gummed up the pipeline for non minority centrists.

• Ben and Marc ground their argument in American predominance built on tech, economy and military. Tech is critical to the other two, and tech depends on startups.

• Current admin chooses winners and losers in tech. True of crypto, CHIPS act, and now AI. This will not work as we don’t know which industries and companies will create the future.

• Marc and Ben have tried to get clarification and guidance for crypto, and have been stonewalled. Elizabeth Warren seems to control the SEC, Gensler won’t make rules and won’t meet. Biden’s decisions on this seem to be under factional control within the Democratic party. A lot of people in tech regards crypto as a scam and money laundering, but there isn’t anything inherently bad in the tech if regulated appropriately, but these efforts have been blocked by government.

• Crypto companies are able to launch elsewhere but not able to launch in the US

• The bureaucrats have subverted the rule of law to attack the crypto industry. Having faced it in crypto, Marc can see it happening to AI. If not for having been blocked on crypto he might not been standing up now. I’d note that this has been going on in finance for a long time, see for example Elizabeth Warren efforts to unbank gun shops. So it’s not so much a targeting of crypto or tech, but making tech play by the same rules as finance already does.

• The military has recognized AI as important as nuclear, and the entire design of the military has to be revamped. This is going to determine what the next hundreds of years will look like.

• The Biden admin’s AI order binds the good guys with bad limits and frees the bad guys. Arbitrary limits on compute when in a couple of years every phone might be able to run compute above the banned limit.

• Regulators think they can regulate and classify math as top secret. It will block US firms while freeing the rest of the world. They provide an anecdote of some mid level functionary actually saying this, and oh boy, this is classic third world governance.

• Marc is no longer able to predict what crazy ideas get implemented

• Fintech also deeply hurt by arbitrary regulation. Fintech is trying to bank underbanked in the US.

• Totalitarian part of Dem party has gotten control of some of these sectors because Biden has delegated it and not in control

• Unrealized capital gains tax will destroy tech startups. This is a big deal. Will decrease returns 99%. VC sector will be over. They give an example of the having to pay based on highest prior valuation of a raise.

• Startups are already half of the California tax base. Making the point that California is highly dependent.

• The unrealized cap gains tax will eventually tax everyone and with increasing rates

• Where they were when Trump was shot. Ben was hanging out with Ivanka Trump and her kids when Marc calls him to tell him.

• Ben and Marc had dinner with Trump just 10 days prior. The shooting was shocking, completely unacceptable in America in 2024.

• Marc admired the physical courage of Trump to stand up, put his fist in the air, and the fact that Trump made his tee off time the next day.

• “A small number of historical moments where you remember where you are” - Marc

• “This guys not like us, he’s a different kind of cat” - Ben; I found this comment to be particularly insightful, because the billionaire view of the last decade was that Trump had somehow, Producers-like, stumbled into the Presidency, because other billionaires had averted their ambitions because of their business records, tax evasion, affairs etc. This is the first acknowledgement from a billionaire that Trump is different.

• “Any person who gets to that level, you have to recognize that whatever bad parts there are, the good parts are really special to get them to where they are”

• Marc can’t believe some of the plans emerging from the administration. None of these issues are politically important issues to the national conversation. They don’t understand why there’s a level of self sabotage when they need tech so much.

• The people in admin in control hate tech and hate capitalism. Referring to the young policymakers in actual control of the state now.

• The future of tech, business and America is at stake, and Trump is the right choice.

Along with Elon, this indicates I think massive shift in elite sentiment. The elites are willing to put up with all of Trump’s distastefulness because the anti-tech, anti-competence, and anti-wealth stance of the Democratic party has finally overwhelmed then.

This along with the Supreme Court’s recent cancellation of the Chevron deference means that a good amount of deregulation maybe coming down the pipe in a Trump presidency.

Much to think about.

🖼️ AI Artwork Of The Day

Here’s today at a glance:

• Four Omni Mini

• AI artwork of the day

Four Omni Mini

OpenAI announced GPT-4o-mini Thursday in a rather quiet announcement. Quick facts:

• It’s a small, fast, cheap model

• It’s better than GPT-4-turbo, Clause Haiku, and Gemini Flash on most benchmarks but behind GPT-4o

• It replaced GPT-3.5-turbo in the API

OpenAI’s strategy of competing wherever there is a critical mass of AI dollars, including at the low end means that it was inevitable that they would come after this market. Though Sam’s spin is slightly more sophisticated:

The model is at the quality/cost frontier

Andrej Karpathy’s take was that we are in the process of distilling the rules for thought out of bigger and bigger models, separating knowledge base from thinking:

LLM model size competition is intensifying… backwards!My bet is that we'll see models that "think" very well and reliably that are very very small. There is most likely a setting even of GPT-2 parameters for which most people will consider GPT-2 "smart". The reason current models are so large is because we're still being very wasteful during training - we're asking them to memorize the internet and, remarkably, they do and can e.g. recite SHA hashes of common numbers, or recall really esoteric facts. (Actually LLMs are really good at memorization, qualitatively a lot better than humans, sometimes needing just a single update to remember a lot of detail for a long time). But imagine if you were going to be tested, closed book, on reciting arbitrary passages of the internet given the first few words. This is the standard (pre)training objective for models today. The reason doing better is hard is because demonstrations of thinking are "entangled" with knowledge, in the training data.Therefore, the models have to first get larger before they can get smaller, because we need their (automated) help to refactor and mold the training data into ideal, synthetic formats.It's a staircase of improvement - of one model helping to generate the training data for next, until we're left with "perfect training set". When you train GPT-2 on it, it will be a really strong / smart model by today's standards. Maybe the MMLU will be a bit lower because it won't remember all of its chemistry perfectly. Maybe it needs to look something up once in a while to make sure.This is not very different from Tesla with self-driving networks. What is the "offline tracker" (presented in AI day)? It is a synthetic data generating process, taking the previous, weaker (or e.g. singleframe, or bounding box only) models, running them over clips in an offline 3D+time reconstruction process, and generating cleaner training data, at scale, directly for the 3D multicam video networks. The same has to play out in LLMs.@karpathy

How cheap is it?

$0.15/$0.60 per million input/output tokens, and half that for batched requests. That’s a 99% price drop from last year’s GPT4 pricing. More use cases will become feasible at this point.

Structured Output

It's also very good at structured output, which will impact a lot of parsing use cases. This is probably the most important industrial use case for small models.

And it can be fine-tuned…

Constitution Testing

Finally, they’re testing role-based access controls in production:

GPT-4o mini in the API is the first model to apply ourinstruction hierarchy(opens in a new window)method, which helps to improve the model’s ability to resist jailbreaks, prompt injections, and system prompt extractions.OpenAI

This is pretty exciting as the learnings from a production run will definitely be used in GPT-5 construction later this year.

Summary

All in all, it’s an exciting update on the path forward toward greater intelligence

🖼️ AI Artwork Of The Day