What's the big deal with Inference anyway?

Goal: to generate tokens given a prompt

KV Cache

KV Cache is an additional step in the self-attention process to speed up computations.

16-bit precision (FP16 / BF16): 16 bits / 8=2 bytes per number

8-bit precision (INT8 / FP8): 8 bits / 8=1 byte per number

4-bit precision (INT4 / FP4): 4 bits / 8=0.5 bytes per number (two 4-bit numbers packed into a single byte)

How does this impact GPU VRAM?

• Double (Batch size)

After a model is released and about hundred of people send prompts to the model at the same time, how does the model know when to respond? And how does it respond to so many people simultaneously? Batching. Batching is acts like a ferry. A ferry has timestamps of when it leaves the dock. It does not wait until every single person is on the boat, that would mean the first person has to wait until the last person boards. It goes out in batches. Every 10 minutes the ferry leaves, the same way every every 10 seconds, those hundred queries are being tokenized, embedded, and so on until the next token is predicted.

Ex. FP16 (P = 2) to FP8 (P = 1) halves the size of every token’s state. Let’s say a GPU has 80 GB VRAM, 40 GB is now dedicated to the KV Cache. Moving to FP8 doubles the number of users / context length without having to upgrade the hardware.

• Improves Inter-Token Latency (ITL)

During token generation (decode phase), memory bandwidth is the main bottleneck. Halving P means the GPU only needs to read half the number of bytes from High-Bandwidth Memory (HBM), speeding up token delivery.

---

(2) Phases

Two phases:

(1) Prefill = take prompt → feed all tokens forward → built key val pairs

Steps:

1. ingest prompt

2. Tokenize and embed prompt into vectors

3. Parallel attention calculation = N-to-N self attention: every single token in prompt simultaneously attends to every other token

4. KV Cache filled = as it’s computing, the model stores the computed representations of all past tokens → called stable data once data is locked in, will not change during decode phase

5. Generates the first token (first word to the answer)

Now since the first word to the answer is generated, we can move onto to generating the rest of the answer in the decode phase. Every word in the answer depends on the words before the current token. Which is why saving these KV matrix values makes computing faster.

Output: first token and full KV cache

(2) Decode = tokens generated one at a time (memory bound) → full answer

Steps:

1. Input for next token is the previously generated token

2. Model looks at finished KV cache matrices

3. 1-to-N attention = single input token is compared to previous tokens (stored in cache) to calculate attention (faster than parallel attention)

4. New key/value vectors added to KV cache – expanding KV cache

5. Next token generated

6. Continue loop until <eos> end of sentence token is generated (complete answer)

Useful for cases where:

Prompt(1) What is Bella doing today?

Prompt(2) What is Bella doing tomorrow?

The model has already calculated the attention scores for the first prompt, so it only has to find the attention score of tokenized tomorrow. It makes the efficiency and speed a model can input data and generate answers faster.

Terms:

Throughput – measures the total volume of work a system processes over a given unit of time (units per second)

Latency – measures how fast a single task completes (time per unit)

---

(3) Key Latency

• Time for first token = measures the time between sending a request and receiving the first output token

In the prefill phase: model takes in all prompt tokens and computes initial Key-value states to populate KV cache

Bottleneck: compute-bound (matrix multiplications across all input tokens in parallel)

How to optimize? Prompt chunking + parallel attention kernels (FlashAttention) + prefix caching + tensor parallelism

• Inter token latency (ITL) = time it takes to generate next tokens

In the decode phase: with every new token, the GPU fetches the model weights and current KV cache, then appends the new KV pair and outputs one token

Bottleneck: memory-bandwidth bound (arithmetic operations / byte transferred)

How to optimize? Quantized KV caching (FP8), Grouped-Query Attention (GQA) + speculative decoding + continuous batching

• Tokens Per Second (TPS) & Serving Cost = measures throughput and operational density split into:

(1) User TPS = streaming speed = 1 / ITL

(2) Server TPS = Total tokens generated per second across all active requests on a GPU cluster SystemTPS=TotalOutputTokensGeneratedacrossBatches / TimeinSeconds

(3) Serving Cost Impact: Increase batch size → increase System TPS → increases KV cache memory consumption – GPU has KV cache pages, requests queue or be evicted

---

(4) Bottlenecks

• Memory Bandwidth

Where it hits: The Decode phase (token generation)

Root cause: one token per sequence at a time, meaning GPU must stream the model’s weights and current KV cache for the sequence

Impact: Modern tensor cores can execute trillions of operations/second → con: memory buses transfer data at a slower rate → compute engines sit idle waiting for memory transfers → ITL becomes memory-bandwidth bound

• GPU VRAM Capacity & KV Cache Bloat

Where it hits: Maximum context length, batch concurrency, and system throughput

Root Cause: Static model weights take a fixed amount of VRAM (e.g., ~16 GB for an 8B model in FP16), but the KV cache grows dynamically with:

Impact: without allocators, pre-allocating memory for max sequence lengths waste 60-80% of VRAM → VRAM fills up meaning the server must reject incoming queries/swap KV blocks → increases Time to First Token (TTFT)

• Compute (FLOPs)

Where it hits: Prefill phase + high batch sizes

Root Cause: model ingests N prompt tokens simultaneously → self-attention calculations fully saturate the Tensor Cores

Impact: prompt lengths scale into thousands of tokens → increasing TTFT

• Inter-GPU Communication Overhead (Distributed Inference)

Where it hits: Multi-GPU serving pipelines (Tensor Parallelism & Pipeline Parallelism)

Root Cause: (1) Tensor Parallelism (TP) splits individual matrix multiplications across GPUs → @ every attention/MLP layer GPUs synchronize activations (2) Pipeline Parallelism (PP) splits layers sequentially across GPUs/nodes → pipeline bubbles: idle time while waiting for previous stages to finish → inter-node network latency

Impact: inter-node networks lack sufficient bandwidth → add latency to GPU computation time

• CPU Scheduling & Serving Engine Overhead

Where it hits: when requesting dispatching, tokenization, and dynamic batches

Root Cause: Host CPUs manage tokenization/detokenization, priority queues, continuous batching schedulers

Impact: If CPU runtime cannot prepare micro-batches/dispatch GPU kernels fast enough → GPU execution queues cap at maximum server throughput

---

(5) Parameters

Parameters:

• Temperature (float): model’s confidence and randomization ( x < 1 = more confident & less random) (x > 1 = less confident and more random)

• top_k (int): how many tokens to consider (instead of considering all 50,000 tokens, just look at top 2 or 3 tokens with highest attention scores) (can turn off by setting to 0)

• top_p (float): consider tokens from top x% of confidences (can turn off by setting to 0)

• beams (int): how many tokens out to consider at a time (set a beam of 3, GPT predicts 3 tokens and decides which one is the best next token)

• do_sample (bool): if true, introduce randomness and not pick the best token, but look at the correlation of the probability distribution (more random text generated)

Normal search with current token: The

Predicts: The dog, The nice, The car

Temperature search:

Top k: readjust probabilities to be sharper, making them less random and stand out more

• top_k = 6; a normal dist would mean probabilities of top 6 tokens add up to .68

• top_k = 6; top 6 tokens add up to .99

Top p: readjust probabilities of the min tokens that exceed a parameter

• top_p = .92; a normal dist would mean top 9 tokens add up to .92 (like mean avg.)

• top p = .92; top 3 tokens add up to .92 (like top tokens to the left of the median)

Beam search – (predicts next 2 tokens to figure out which token comes next)

• Predicts: The dog has, The nice woman, token has too low attention scores

do_sample

By readjusting these parameters, you can train a model to speak more naturally and make general patterns with enough randomization in them to not memorize, but understand the why this phrasing? behind a sentence.

Narrow Generation Ability is when pre-trained model is tuned too finely with these parameters that they memorize information, rather than understand it.