TLDR: Processing input tokens is done in parallel, generating output tokens is sequential. The GPUs memory is therefore longer occupied when generating tokens and the extra cost you pay for output tokens reflects the additional memory cost it takes to generate them, and not necessarily the compute cost. Batch APIs find the right balance between memory and compute capacity of the GPUs, offering much cheaper prices for the tradeoff of non-instant responses.
Coming up...
Bottlenecks of AI
Compute: FLOPS and FLOPs
Memory: VRAM and memory bandwidth
LLM Inference Theory
Batch API
Introduction
When you look at the pricing tables of APIs from OpenAI or Anthropic you would see that you pay $3 per 1 Million Input Tokens but $15 for the same amount of Output Tokens. Since the amount of computations necessary for processing 1 Million Input Tokens is the same then for processing 1 Million Output Tokens (assuming you use a proper caching mechanism), why is that?
Bottlenecks of AI
The main bottlenecks of AI at the current state one could argue are:
Data (Non-Synthetic High-Quality)
GPUs
Energy
New Ideas
Since the first and the last only apply to moving the field forward and creating more capable models and not to your prompt "What does amortization mean?" at 8:45 in the morning, the second and the third are more relevant in our scenario.
It is estimated that an average prompt for ChatGPT consumes around 2.9 Wh, which is nearly 10 times more than a single Google Search. Summing this with around 200 Million queries per day for a whole year and calculating in spike usages, the energy consumption would be around 227 GWh, enough energy to run the whole country of Finland or Belgium for a day [1]
But with Microsoft reactivating nuclear reactors to solely power their data centers [2], the energy issue is much more solvable at the moment and not the driver of the API pricing.
API pricing is largely driven by GPUs, which are the lifeblood for AI systems. The scarcity, high cost, and specific requirements of GPUs heavily influence how companies like OpenAI and Anthropic operate and scale [3]
Three parameters of a GPU are especially important:
How much computations it can do per second (aka "compute" -> measured in (Tera)FLOPs per second)
How much data it can store (aka "memory" -> measured in (Giga)Bytes)
How much data it can load per second (aka "memory bandwidth" -> measured in (Tera)/Bytes per second)
Compute: FLOPS and FLOPs
GPUs are very very good at doing basic computations (operations) on numbers (floating point numbers). For the purpose of this blog post, lets stick with decimal numbers ~ floating point numbers, which is not 100% true (see here for a very good writeup on floating points) [4]. Floating point numbers can have a precision, which in analogy to decimals can be approximated with how many decimal values i allow before rounding. Common levels of precisions are FP(8,16,32,64) where FP64 is the most precise but also the most memory-intensive way to store your numbers.
Most of the operations in a Neural Network are additions and multiplications. The maximum amount of these operations a GPU can do per second is expressed with FLOPS (Floating Point Operations per Second). For example, the flagship GPU of the moment, the H100 (~40.000€), from Nvidia can do about 30 TFLOPS (30 Trillion Operations per second) of FP64.
Figure 1: H100 from Nvidia [5]There is always a confusion between FLOPS and FLOPs. FLOPS refers to the amount of operations per second, whereby FLOPs is the plural of a FLOP.
Memory: VRAM and Memory Bandwidth
The main architecture of a GPU is shown in Figure 2. The DRAM (Dynamic Random Access Memory) is the external memory of a GPU (also called VRAM). The data (model weights, datasets, activations) on the DRAM gets loaded on to the L2-Cache, which is accessed by multiple parallel Streaming Multiprocessors (SMs) which also have their own L1-cache. The two most important memory parameters of a GPU are the memory capacity of the VRAM and the data transfer capacity (memory bandwidth) per second to load the data from the VRAM to the SMs, where the actual processing happens. Taking the H100 again as an example, it has a VRAM of 94 GBs and a memory bandwidth of 3.9 TB/s.
Figure 2: The main components of a GPU. PC stands for Processing Cores. [6]
Okay, thanks for the informations, but why do i have to pay now 3-5x more?
LLM Inference Theory
Now the basics are set. When you send your prompt, it gets first tokenized (converted into numbers) and loaded as a tensor (optimal format for a GPU) into the VRAM of the GPU together with the model weights (also as tensors).
The two phases of a Transformers are the prefilling (processing of input tokens) and decoding (generation of the next token). Prefilling is a parallel process, whereas decoding is a sequential process. This means that the model weights, prefilled tokens and newly generated tokens will have to be loaded again and again for each token that is generated and thereby occupying the GPU memory. When the decoding phase is finished, the response tokens are detokenized into a human-readable string and shown as the output prompt.
Figure 3: Process of LLM inference. [7]
Balancing compute and memory
To find the ideal balance between compute and memory use, we can divide compute (FLOPS) by memory bandwidth. Lets take the current H100 specs as an example. A realistic floating point precision is FP16. The compute of 30 TFLOPS for FP64 is 120 TFLOPS for FP16. Dividing 120 TFLOPS with the memory bandwidth of 3.9 TB/s results in ~31 FLOPs/Byte. Since FP16 values are 16-bits (2 bytes), we multiply by 2, resulting in ~62 FLOPS per FP16 value read.
This 62 FLOPS/value figure represents the optimal compute-to-memory usage ratio, meaning that to fully utilize the GPU's resources, for each FP16 value read from memory, you should perform around 62 FLOPS of compute. From this, we can draw a couple of key points:
If you perform fewer FLOPS per value read, the GPU will be memory-bound, meaning the compute units (SMs) will idle while waiting on data transfers to and from memory. This underutilizes the GPU’s compute potential.
If you perform more FLOPS per value read, the GPU will be compute-bound, meaning the memory is underutilized while the SMs are busy with excess computation.
For large language models (LLMs), the first case (memory bottleneck) is often the limiting factor, as memory bandwidth can’t keep up with the GPU’s maximum compute capacity.
Now, consider the cost difference between processing input and generating output tokens:
Processing input tokens: This can be done in a single forward pass, allowing the model weights to be loaded once and used across multiple calculations, effectively batching the operation.
Generating output tokens: For each output token, a separate forward pass is required, which means the model weights are reloaded every time with minimal compute occurring each pass. This is inefficient since the high cost of loading weights is not offset by enough compute.
In summary, the extra cost you pay for output tokens reflects the additional memory cost it takes to generate them, and not necessarily the compute cost. [8]
Batch API
OpenAI and Anthropic are therefore offering a Batch API, which is 50% cheaper than the on-demand API. The Batch API works asynchronous via Static Batching where the Client (You) creates a batch of requests and sends it to the server (OpenAI/Anthropic), which responds with the results within 24 hours. This is for example helpful when you want to create a vector store and need to embed a lot of documents. Since it does not matter if you get the embeddings right now or latest tomorrow but it does matter that you can get them for half the API costs, using the Batch API for bulk document embeddings makes a lot of sense.
Figure 4: Current Batching methods [9]
The current state of the art of the LLM inference batching is Continuous Batching, which works at the token level rather than at the request level. The model server loads each layer of the model sequentially and applies it to the next token of each request. In continuous batching, the same model weights could be used to generate the fifth token of one response and the eighty-fifth token of another.
Thanks for reading!
I hope you learned something!
I can highly recommend reading the blog post of Peter Chang [8] and the Scaling GPT Edition of the Programmatic Engineer [3].
