So, theoretically, when moving to HBM5, with a hypothetical 1GB of HBM on one Nvidia AI GPU, the context window will be so large that it can solve a highly complex problem?
The constraints now lies in the HBM capacity and bandwidth for LLM companies to expand their processing capability? In exploring “better answer or solution” ?
Great question — you’re on the right track. HBM5 will massively raise the ceiling on capacity and bandwidth. But the key insight is that reasoning itself is a tax on that memory. Longer chains of thought, bigger KV caches, iterative token generation — all of it eats into your HBM budget before you even get to the actual problem. So more memory helps, but how efficiently you use it is what really determines performance. That’s the core of the Memory Wars thesis
You've argued the Groq co-design is additive TAM and that better models keep increasing memory bandwidth demand. But does that inflect? Even with multi-chip optical sync extending SRAM’s effective ceiling, there’s a crossover point where reasoning hcains outgrow what that architecture can economically serve - and the HBM’s density advantage reasserts. What share of total inference demand do you think stays below that crossover long term? Or do reasoning models become the default and the LPU-hybrid case shrinks to a latency-sensitive niche?
So, theoretically, when moving to HBM5, with a hypothetical 1GB of HBM on one Nvidia AI GPU, the context window will be so large that it can solve a highly complex problem?
The constraints now lies in the HBM capacity and bandwidth for LLM companies to expand their processing capability? In exploring “better answer or solution” ?
Great question — you’re on the right track. HBM5 will massively raise the ceiling on capacity and bandwidth. But the key insight is that reasoning itself is a tax on that memory. Longer chains of thought, bigger KV caches, iterative token generation — all of it eats into your HBM budget before you even get to the actual problem. So more memory helps, but how efficiently you use it is what really determines performance. That’s the core of the Memory Wars thesis
It’s quite scary
We might end up working on a 8GB laptop for a long time because DRAM becomes so expensive due to supply shortage from the HBM consumption.
One more follow up question:
Scaling law. But no one knows the magnitude of scaling in next gen training? This is the unknown?
You've argued the Groq co-design is additive TAM and that better models keep increasing memory bandwidth demand. But does that inflect? Even with multi-chip optical sync extending SRAM’s effective ceiling, there’s a crossover point where reasoning hcains outgrow what that architecture can economically serve - and the HBM’s density advantage reasserts. What share of total inference demand do you think stays below that crossover long term? Or do reasoning models become the default and the LPU-hybrid case shrinks to a latency-sensitive niche?