OpenAI released GPT-5.4 benchmarks this week. The scores are impressive: 75% on OSWorld, 94.4% on GPQA Diamond, 50% on FrontierMath.
These benchmark drops happen constantly now. Every few weeks, another lab publishes a table showing their model beating the last one. It’s become a vanity flex, a leaderboard game that most investors and even most builders have learned to tune out.
I don’t care who’s winning the benchmark race. I care about what it means for the ecosystem. The silicon, the memory, the interconnects, and the infrastructure that has to actually serve these models at scale.
And this one is worth paying attention to. Not because of the scores, but because of the fine print.
At the top of the benchmark table, in gray text that most people scrolled past: “Run with maximum available reasoning effort.”
That’s not a disclaimer. That’s a business model. And it validates everything we’ve been building in the Memory Wars series, including a thesis about Groq that I published last week in The Fourth Piece.
Let me explain.
Scaling Laws Are Holding. That’s the Point.
I need to say something I had article in my drafts before the official scores dropped: the scaling laws skeptics are wrong.
Remember the narrative from late 2024? “Scaling has hit a wall.” “Pretraining is over.” “GPT-5 was disappointing.” Ilya Sutskever himself said the “age of pretraining is ending.” Gary Marcus declared vindication every other week.
Look at the FrontierMath progression. This is a benchmark of research-level mathematics, problems so hard that Terence Tao called them “incredibly difficult” and predicted they’d remain unsolved for years. Here’s the trajectory:
o3 (Dec 2024): 25.2% → GPT-5.2 Thinking: 40.7% → GPT-5.4 Thinking: 47.6% → GPT-5.4 Pro: 50.0%
On Tier 4, the hardest problems: GPT-5.2: 18.8% → GPT-5.4 Pro: 38.0%. That’s a doubling in the span of months. On problems designed to be unsolvable.
Scaling didn’t hit a wall. It found a new dimension. The labs shifted from pretraining scale (bigger models, more data) to inference-time compute scale (more reasoning tokens, deeper chains of thought, “extreme” thinking modes). Both are scaling laws. Both consume infrastructure. The second one, inference scaling, consumes infrastructure at the point of deployment, not just the point of training.
And now comes the detail that makes the infrastructure math explode: GPT-5.4 supports a 1 million token context window, more than double GPT-5.2’s 400K tokens. OpenAI is reporting improved performance on tasks that run for several hours, with better retention of details across multi-step processes. The “extreme” thinking mode lets the model burn significantly more compute on hard problems.
A 1 million token context window with an extreme reasoning mode running multi-hour tasks. Read that sentence again from an infrastructure perspective. That’s not a chatbot query. That’s a sustained, memory-intensive compute workload that looks more like a training job than a traditional inference request.
For investors, this is the most important thing in the entire benchmark release: scaling laws are intact, and they’ve shifted to inference. Every dollar that used to go exclusively to training clusters now has a matching dollar going to inference infrastructure. The demand didn’t plateau. It moved.
The Reasoning Tax
Here’s what “maximum available reasoning effort” means at the silicon level.
When GPT-5.4 hits 94.4% on GPQA Diamond, a benchmark of PhD-level scientific reasoning — it doesn’t get there by being more efficient. It gets there by thinking longer. Chain-of-thought reasoning generates 10x to 100x more tokens per query than a simple completion. The model writes out its reasoning steps, evaluates them, backtracks, tries alternative approaches, and synthesizes an answer.
Every single one of those reasoning tokens passes through the KV cache.
In The Memory Wars, I walked through the math: a 70B parameter model with a 128K-token context window consumes 40GB of KV cache per user. Serve 10 users simultaneously and that’s 400GB just for the cache. Before you even store the model weights.
Now multiply the sequence length by the reasoning overhead. A GPT-5.4 Thinking query doesn’t generate 500 tokens of output. It might generate 5,000–50,000 tokens of internal reasoning to produce those 500 tokens of visible response. The KV cache scales with the total sequence, including the hidden reasoning chain.
The KV cache formula I laid out in The Memory Wars makes this concrete:
KV Cache = 2 × 2 × head_dim × n_heads × n_layers × seq_length × batch_size
When seq_length goes from 2K (simple query) to 50K (deep reasoning chain), the KV cache grows 25x. That’s 25x more HBM consumed. 25x more memory bandwidth required for every attention operation. 25x more pressure on the memory wall.
This is the Reasoning Tax. And every model lab is racing to make it bigger, not smaller.
Early testers note that GPT-5.4 Thinking is actually more efficient on routine tasks, using fewer reasoning tokens than previous models for the same results. But that efficiency doesn’t reduce infrastructure demand. It gets reinvested. When reasoning gets cheaper per token, users push harder problems at the model, and the model responds with longer, deeper chains of thought. The ceiling goes up, not the floor.
The Benchmarks That Actually Matter
The benchmark table tells a specific story if you know what to look for. Not all benchmarks are created equal from an infrastructure perspective.
Single-shot reasoning (GPQA Diamond, FrontierMath) is memory-intensive but bounded. The model thinks hard once, then stops. The KV cache spikes and releases.
Agentic workloads are a different beast entirely. Look at the benchmarks where GPT-5.4 Thinking dominates:
SWE-Bench Pro (57.7%): software engineering tasks that require multi-step reasoning. The model reads code, plans changes, implements them, tests, and iterates. That’s a sustained inference session with a KV cache that grows with every step and never releases until the task is complete.
BrowseComp (Thinking: 82.7%, Pro: 89.3%): agentic web browsing. The model navigates pages, reads content, makes decisions, backtracks. Each browsing step adds to the context window. The KV cache accumulates across the entire browsing session.
Toolathlon (54.6%): agentic tool use across sustained multi-step interactions. Same pattern: context accumulates, KV cache grows, memory pressure compounds.
These aren’t benchmarks that test how smart a model is in isolation. They test how well the entire inference infrastructure handles sustained, memory-intensive workloads.
The Two Paths
This is where the benchmarks connect to the Groq thesis.
There are now two architecturally distinct paths for serving reasoning-era models. Both face the same memory wall. They attack it differently.
Path One: Brute Force (GPU + HBM)
This is how GPT-5.4 achieves those benchmark scores today. Conventional GPUs with dynamic scheduling, massive HBM stacks, hardware-managed caches. When the reasoning chain generates 50,000 tokens, the KV cache spills across HBM. The GPU’s tensor cores sit idle waiting for memory, the same arithmetic intensity mismatch I documented in The Memory Wars, now multiplied by the Reasoning Tax.
The brute force path scales by adding more HBM. More stacks, more capacity, more bandwidth. NVIDIA’s Vera Rubin roadmap (288GB HBM4 per GPU at up to 22TB/s, scaling to 1TB per GPU with Rubin Ultra) is the engineering response to this path. It works. It’s expensive. And the Reasoning Tax makes it more expensive with every generation of smarter models.
Path Two: The Compiler Path (Dataflow + SRAM + Optical Sync)
This is what NVIDIA paid $20 billion for when it acquired Groq.
As I wrote in The Fourth Piece, Groq’s architecture eliminates the dynamic scheduling overhead entirely. The compiler pre-schedules every operation: every memory access, every data movement — before the chip runs. The hardware makes zero dynamic decisions. It executes a pre-computed schedule with cycle-accurate precision across 230MB of on-chip SRAM running at 80 TB/s internal bandwidth.
For reasoning workloads, this changes the economics fundamentally. The reasoning chain isn’t a surprise to the hardware. The compiler has already mapped the entire execution graph. There are no cache misses because there is no cache. There are no memory stalls because every access was scheduled at compile time. The KV cache doesn’t “spill” because the compiler placed it exactly where it needs to be.
The limitation was always scaling: Groq’s deterministic execution requires cycle-accurate synchronization across chips, and electrical interconnects introduce jitter that breaks the model. That’s what NVIDIA’s ISSCC optical clock-forwarding paper solves. That’s why I called it “the fourth piece” that connects the Groq compiler, stacked memory, and backside power delivery into a coherent product thesis.
GPT-5.4’s benchmarks just showed the world why this matters. Every reasoning token that GPT-5.4 generates on a GPU, waiting for HBM, stalling on attention, burning power on idle tensor cores — is a token that could run faster, cheaper, and more deterministically on a compiler-scheduled dataflow engine.
OpenAI proved the demand. NVIDIA is building the supply.
The Infrastructure Map
The Reasoning Tax doesn’t create winners in one layer. It creates winners across the entire memory hierarchy, exactly as the Memory Wars framework predicted.
There’s an important nuance here that connects to what I wrote about Nemotron 3 in December. NVIDIA’s open-source inference stack is driving the cost per token toward zero. Nemotron’s MoE architecture, NVFP4 quantization, multi-token prediction: all of it makes each individual token cheaper to serve. But the Reasoning Tax works in the opposite direction. GPT-5.4 doesn’t use fewer tokens because inference got cheaper. It uses orders of magnitude more tokens because reasoning got better. Cost per token goes down. Tokens per query go up. Net infrastructure demand increases. This is why NVIDIA can give away the model and still sell more hardware.
HBM suppliers (SK Hynix, Samsung, Micron): Path One isn’t going away. Training stays GPU + HBM. Prefill stays GPU + HBM. And even Path Two needs massive model weight storage. The 16-Hi HBM4 race I covered in The Memory Wars (30μm wafers, sub-10μm bonding layers, 775μm height limits) becomes more urgent as reasoning models grow larger. Every generation of smarter models demands more memory capacity.
Optical interconnects (Lumentum, Coherent, Astera Labs, Credo): Agentic workloads distribute across nodes. A SWE-Bench Pro task doesn’t fit on one GPU. It requires sustained east-west traffic between compute nodes as the reasoning chain progresses. For Path Two, optical clock-forwarding is the enabling technology that makes rack-scale dataflow inference physically possible. Both paths drive optical interconnect demand.
Advanced packaging (TSMC, Besi, SUSS MicroTec): Path One needs more CoWoS capacity for bigger HBM stacks. Path Two, if NVIDIA builds a 3D-stacked SRAM inference engine on A16 with backside power delivery — needs hybrid bonding equipment and the packaging ecosystem I’ve been tracking. Different products, same equipment winners.
Memory architecture innovation (Rambus, Intel): David Patterson’s January 2026 paper diagnosed exactly this moment. The Turing laureate wrote that the primary challenges for inference are “memory and interconnect rather than compute.” Controller-managed ECC from the REACH thesis, high-bandwidth flash, processing-near-memory. These are all responses to the same Reasoning Tax that GPT-5.4 just made undeniable.
So What?
Six takeaways, plain English:
1. Scaling laws are intact. They just moved. FrontierMath: 25% → 50% in under 18 months. Tier 4 doubled in months. The new dimension is inference-time compute. The demand didn’t plateau. It shifted from training clusters to deployment infrastructure.
2. The 1M-token, multi-hour inference session is here. This isn’t a chatbot query. It’s a sustained compute workload with a KV cache that grows continuously for hours. Infrastructure demand per query is scaling with model intelligence.
3. Agentic workloads compound the problem. Coding, browsing, and tool-use tasks don’t spike and release. They accumulate context across entire sessions. The agentic era needs sustained memory, not burst memory.
4. NVIDIA’s $20B Groq bet just got external validation. OpenAI proved inference-time compute is the growth vector. The compiler-driven dataflow path is NVIDIA’s answer to serving this demand without being perpetually bottlenecked by HBM supply.
5. The memory hierarchy fragments. Training stays HBM. Hot decode moves toward SRAM. Context overflow goes to flash. Patterson’s four directions and the tiered architecture from Raja Was Right are playing out in real time.
6. GTC on March 16 is the next catalyst. If any Groq-derived architecture, stacked SRAM detail, or optical inference interconnect shows up in Jensen’s keynote, the thesis moves from framework to product. I’ll be there live, covering it for subscribers.
BEP Research Is Going Paid — and GTC Is Coming
I know there are a lot of Substacks out there. Some too technical, some too financial, most somewhere in between. What I try to do is different. I look at investing like systems. I try to see the forest through the trees. I won’t bombard you with spreadsheets or model outputs. I’m a long-term investor, not a trader. I don’t chase quarters or try to time earnings. My philosophy is to understand how the pieces connect — silicon to software to business model to investment thesis — and explain it in a way that helps you think, not just react.
That approach has served me well over the past 20 years. By writing my ideas here, it keeps me accountable.
BEP Research is going paid on March 9. Annual subscriptions are $400/year, but through March 9 you can lock in the early-bird rate of $350/year. Once the paywall goes up, the discount goes away. If you’ve been reading for free and finding value, now is the time.
GTC 2026 starts March 16. I’ll be there in person. On-site analysis of NVIDIA’s architecture announcements, networking roadmaps, and co-packaged optics developments will be a cornerstone of paid subscriber coverage.
Related Reading
NVIDIA Nemotron 3: Why This Changes Enterprise AI Economics — How NVIDIA’s open-source inference stack drives cost per token toward zero, and why that accelerates hardware demand
The Fourth Piece: Why Groq Changes the GTC Thesis — The dataflow inference engine thesis connecting Groq, ISSCC optical clock-forwarding, and the co-design framework
The Memory Wars: Why NVIDIA’s 2028 Architecture Ends the AI Chip Competition — The complete memory hierarchy analysis: HBM, SRAM, the 16-Hi race, and why Feynman changes everything
Raja Was Right, Part 2: The New Memory Hierarchy — Patterson’s four directions, the REACH thesis, and the investment map for memory architecture innovation
A Pre-NVIDIA GTC Thesis Update — Where the three-piece convergence stood heading into GTC
About the Author
Ben Pouladian is a Los Angeles-based tech investor and entrepreneur focused on AI infrastructure, semiconductors, and the power systems enabling the next generation of compute. He was co-founder of Deco Lighting (2005–2019), where he helped build one of the leading commercial LED lighting manufacturers in North America. Ben holds an electrical engineering degree from UC San Diego, where he worked in Professor Fainman’s ultrafast nanoscale optics lab on silicon photonics and micro-ring resonators, and interned at Cymer, the company that manufactures the EUV light sources for ASML’s lithography systems.
He currently serves as Chairman of the Leadership Board at Terasaki Institute for Biomedical Innovation and is a YPO member. His investment research focuses on AI datacenter infrastructure, GPU computing, and the semiconductor supply chain. Long-term NVIDIA investor since 2016.
Follow on Twitter/X: @benitoz | More at benpouladian.com
Disclosure: The author holds positions in NVIDIA and related semiconductor investments. This is not investment advice.
So, theoretically, when moving to HBM5, with a hypothetical 1GB of HBM on one Nvidia AI GPU, the context window will be so large that it can solve a highly complex problem?
The constraints now lies in the HBM capacity and bandwidth for LLM companies to expand their processing capability? In exploring “better answer or solution” ?
You've argued the Groq co-design is additive TAM and that better models keep increasing memory bandwidth demand. But does that inflect? Even with multi-chip optical sync extending SRAM’s effective ceiling, there’s a crossover point where reasoning hcains outgrow what that architecture can economically serve - and the HBM’s density advantage reasserts. What share of total inference demand do you think stays below that crossover long term? Or do reasoning models become the default and the LPU-hybrid case shrinks to a latency-sensitive niche?