What a Transformer Actually Costs

Twelve days apart, the same lab put two language models on the same public price list. The first, called GLM 5.3, costs a dollar forty for a million tokens of input. A token is a piece of a word, so call that three quarters of a million words. The second, a version of the same model called Flash, costs fifteen cents.

Nine times cheaper, from the same building, in under a fortnight.

That is not a sale. Several things changed inside the model at once, and a price is a business decision as much as a cost readout, so nobody outside that lab can say how much each change contributed. But one of the things that changed is the part of the machine that has been both its great strength and its central expense since 2017. It is worth understanding what that part does, because once you can see it, a surprising number of otherwise arbitrary things stop being arbitrary. Why cheap models suddenly got good. Why the size of the context window is a number companies announce on launch day. Why a long conversation gets slow and expensive in a way a short one does not. And why almost nobody runs one of these on their own hardware, even when the hardware would hold it.

This is a briefing on the landscape. It will not make you an engineer. It should let you follow the conversation, and ask the right question when someone is selling you something.

Everything looks at everything

Before 2017, a machine reading a sentence read it roughly the way you would read it aloud. One word, then the next, carrying a running summary forward as it went. A running summary is a bottleneck. By the end of a long paragraph, whatever mattered at the beginning had been compressed, and compressed again, and was mostly gone. Everyone in the field knew this, and none of the fixes really worked.

The paper that changed it was published in June of 2017 by eight researchers at Google, under the title Attention Is All You Need. Its proposal was, on the face of it, extravagant. Stop carrying a summary. For every word in the input, look at every other word in the input, all at once, and work out which of them matter for this one.

Take the sentence the bank will close at five. To place the word bank, the machine looks at close and at five, weighs them, and settles on the building rather than the river. It does that for every word at once, then does it again through dozens of stacked layers. The operation is called attention. The design built out of it is the transformer. Every chatbot you have used is one. If you want the fuller picture of how attention shapes what a model writes, the shelf already has it, in How Chatbots Actually Write. This piece is about what that article leaves alone: what attention costs.

Two things followed from the design, and the second is the one usually left out.

The first is that it could be trained in parallel. A running summary has to be computed in order, which leaves most of a very expensive chip sitting idle. Attention has no order. Every position can be worked on at the same time as every other, so for the first time you could throw an enormous amount of hardware at the problem and have the hardware actually be busy. That is the half of the story everybody tells.

The other half is about distance. In the older design, a word at the start of a passage and a word at the end were separated by every word between them, and whatever connected them had to survive each step. In a transformer, any two words are one step apart. That is not an argument about speed. It is an argument about what the machine can learn at all, and it is why the transformer did not merely train faster but understood more.

So the design was not a trick. It was a better idea, and it won on merit.

It also came with a bill.

The bill

Every word looking at every other word means the work grows faster than the conversation does. Double the length of what the model is reading and you have not doubled the attention work. You have quadrupled it, because every new word has to be compared with every word already there, and every word already there now has one more to look at.

For a single reply, that would be tolerable. Conversations are not single replies. Every time you send a message, the model needs everything that came before it, and rather than recompute all of that from scratch it keeps a running record in memory, one entry per token it has seen, so it can look back quickly. That record has a technical name, the key-value cache, and you do not need the name. You need to know what it does: it grows by one entry with every token, for as long as the conversation runs, and it belongs to that conversation alone.

This is where the cost turns from an idea into a number. In September of 2025, NVIDIA published the figures for one popular open model, Meta’s Llama 3, in its large size. The model itself, the thing everyone shares, takes about a hundred and forty gigabytes of memory, and it never grows. One live conversation that has run to the model’s full length takes about forty gigabytes more. Just for that conversation. A second conversation takes another forty. A third, another forty.

By the fourth conversation, what nobody sees outweighs the model. And the fifth has only just said hello.

That is the whole bill, in one picture. The model is a fixed cost, like a building. The conversations are a running cost that grows with every token and stacks with every person talking at once, and here the running cost can outweigh the building. When a vendor announces a bigger context window, it is announcing a bigger ceiling on that second number. When your long chat gets sluggish toward the end, you are feeling the second number climb. And when a lab quietly gets nine times cheaper, this is the first place to look.

How big it got

For a while this was a problem for people who ran data centres. Then the field decided that the way to make models smarter was to have them think out loud before answering, writing thousands of tokens of working before the first word you see. That made every conversation long by default, and the model’s own reasoning now fills the cache faster than any person could type.

One research group, Epoch AI, modelled what that does at planetary scale, across every current-generation NVIDIA chip on Earth. Serving short requests, the fleet can handle roughly twenty billion tokens a second. Serve it requests that carry the model’s full context, and capacity falls to about five hundred million. The same chips, the same length of answer, forty times less throughput. And their model varies only the input, holding the answer fixed, so the effect of reasoning models is not in that number at all. It is worse.

That is the pressure behind almost everything you have read about AI in the past year. The chip shortage, the price cuts, the parade of new model names, the rise of the Chinese labs. Most of it is the industry paying down this one bill.

What is being done about it

There are three lines of attack, and all three shipped this year.

The first is to stop most of the model from doing the expensive thing at all. Three labs, working separately, arrived at the same answer within weeks of each other: keep full attention in about a quarter of the layers, and give the other three quarters a cheaper form of memory that stays the same size no matter how long the conversation runs. In the Flash model from the price list, thirty-four of forty-five layers work that way. Only eleven still keep a record that grows. The fix is not cleverer attention everywhere. It is most of the building with the lights off.

The second is to make the layers that still pay attention look at less. Instead of every word comparing itself with every word, the model learns which handful of earlier words are worth consulting and skips the rest. One lab claims that at very long lengths this cuts the memory needed to a tenth of what its previous model used. That is the lab’s own figure, unmeasured by anyone outside, but the direction is not in dispute.

The third is to reshape the hardware around the operation. The mathematics of attention has not changed since 2017. What has changed, in four rewrites over four years, is how that mathematics is arranged to fit whatever chip is current. And the end of that road is already visible: OpenAI now reports that its own models helped optimise parts of the arithmetic circuits on a chip it designed with a partner, and wrote some of the code that runs attention faster than the human experts had. Parts of, on selected pieces, starting from working designs. Not a chip designed by a machine. But the loop is closing.

One thing is worth noticing about who says this out loud. The link between long reasoning and the cost of serving it is stated plainly in the technical reports of Chinese labs and in systems papers out of Microsoft Research, and it appears nowhere in the launch communications of the leading American labs. Their announcements talk about capability. The bill is discussed by the people paying it in public.

Not settled

It would be tidy to say the fix is in. It is not, and the most honest evidence comes from a lab that tried it at full scale and turned back.

MiniMax, in Shanghai, shipped a model built on the cheap-memory approach, ran it in production, and then rebuilt its next model with full attention on every layer. They published why. On the standard tests, the efficient version appeared to match. At real scale, on problems that took several steps of reasoning, clear gaps appeared. Their phrase was that they found no variant that reliably matched full attention quality in production. Their next model changed direction again, toward the second line of attack above, looking at less, and its technical report does not mention the reversal.

Meanwhile the lab behind the Flash model published its own study in February finding that the efficient approach carries an inherent accuracy gap on fine-grained tasks, and shipped it in August regardless. And none of the three labs that converged on the one-in-four ratio has published why one in four is the right number.

None of that means the efficient models are bad. It means the field is running the experiment in public and has not agreed on the result. When you are told a cheap model is as good as an expensive one, that is the claim being tested. The honest answer today is: for many things, yes; for the hardest things, not reliably; and nobody can tell you in advance which is which.

So could you just run one yourself?

I asked this question seriously, because I am the kind of person who owns the hardware. The Flash model on the price list is open. You can download it. In the compressed form most people would run, it is a little under a hundred and eighty gigabytes, and a Mac Studio with two hundred and fifty-six gigabytes of memory will hold it with room for a conversation. That machine, at the education price, is a little over nine thousand dollars.

So it fits. The question is whether it pays, and the arithmetic is not close. Against fifteen cents a million tokens on the rental price list, the machine earns its keep only if you push more than thirty-seven million fresh tokens through it every single day for three years. That is not a busy person. That is a small company’s entire text traffic. For nearly everyone, renting wins by a wide margin, and wins again because the machine arrives in six to seven weeks, in a field where the model you are buying it for shipped last week.

That leaves the reason people actually want the box, which is not money. It is that the conversation never leaves the room. And here the honest version is narrower than the scary one. The provider behind that price list states in writing that it never logs prompts or responses, never trains on them, and retains nothing. So the trade is not privacy against price. It is a physical guarantee against a contractual one. Which is worth more to you depends on what you are sending, who could demand it, and whether you would rather trust a lock or a promise.

The labs can see that trade as clearly as you can, and they are moving on it. The open models got good enough that the box became a real alternative to a frontier contract, so at least one frontier lab is now building the box into its cloud: Anthropic’s newest models come with an arrangement under which the model runs in Anthropic’s cloud and the data lives in infrastructure the customer controls. That is nearer a lock than a promise, without buying hardware. Two things to notice. It is announced, not shipped. And it arrives first for the largest enterprise customers, in phases, with a zero-retention promise as the fallback for everyone else who qualifies. As with most things in this story, the strongest guarantee goes first to where the most money is. So the governance question is no longer lock or promise. It is which of three you can actually get, at your size, and that is the right question to bring to whoever runs your systems.

What to do with this

You now hold one idea: the model is a fixed cost, and every conversation is a growing one. That is enough to ask three questions that most buyers never ask.

Ask what part of your work is actually long. Reading a contract, summarising a call, answering a customer, drafting a paragraph: most of it is short, and a short task on a cheap model is now good and nearly free. The expensive models earn their price on the long, many-step problems, and the honest vendors will tell you which of your work is which.

Ask what repeats. If the same document, the same instructions, the same background goes into every request, most providers will now charge a fraction of the price to keep it in memory rather than reread it. On the price list at the top of this piece, cached input is three cents where fresh input is fifteen. That discount exists because of the bill described above, and it is yours if you ask.

And ask where the conversation lives. Not because renting is dangerous, but because there are now three honest answers, and the one you get depends on how big a customer you are. The answer should be written down somewhere, by someone, before it matters.

Close

The transformer won because it let every word see every other word. Nine years on, the whole industry is working out how much of that seeing it can afford to give back, and no one yet agrees on the answer. Some of the labs are betting that most of a model can stop looking and nothing important is lost. One of them tried it, measured it, and went back.

The next model you hear about, whatever its name, will be somewhere on that line. Now you can ask which end.