Choosing a GPU for LLM inference: Memory, latency, and capacity
GPU capacity is not decided by whether a model fits. Calculate whether the service can keep its first-token and streaming-latency promises while serving the same work concurrently, then validate it with representative request patterns.

Design scenario: an 8B RAG server for B2B customer support
This scenario covers a service in which agents at 120 customer organizations search Korean and English policy documents and review drafted replies. Scale and timing are design inputs and validation targets.
- Documents and permissions
- 120 customer organizations / 60,000 policy documents / search scope isolated by organization
- Model
- 8B dense, BF16 weights
- Request
- 6,000 input tokens, 450 output tokens
- Peak
- 24 active streams, 12 queued requests
- Target
- TTFT p95 at or below 2.5 seconds; ITL p95 at or below 80ms
Set the SLO and admission rules before sizing capacity
Success for this service does not mean high GPU utilization. A tenant must see an answer begin even after submitting a long policy document, while another tenant’s short question must not wait indefinitely. TTFT is the time from a request reaching the server to delivery of its first output token. ITL is the interval between adjacent output tokens while streaming. TTFT p95 is the 95th percentile of first-token latency by request; ITL p95 is the 95th percentile of observed output-token intervals. Keeping those sampling units distinct prevents long requests from being hidden.
Admission control is part of capacity design. Once 24 streams are active, new requests enter a queue; once 12 are waiting, the service returns a retryable response. Product policy determines whether long-context requests use a separate queue or lower priority. Unlimited concurrency may sound accommodating, but it can break the SLO of requests already in progress.
vLLM metrics provides the signals needed to observe TTFT, inter-token latency, request stages, and KV cache.
- Admission
Check tenant limits and the queue
- Retrieval
Set document chunks and the input-token budget
- Prefill
Observe TTFT and long-input pressure
- Decode
Control ITL and active streams
- Completion
Record metric, cost, and quality samples
Weight calculation is the starting line
A simple lower bound for storing 8B parameters at 2 bytes each in BF16 is 8,000,000,000 × 2 = 16,000,000,000 bytes, or about 14.9 GiB. That calculation covers weights only. It excludes execution-engine workspace, communication buffers, graphs, and fragmentation headroom.
| Area | How to calculate or verify it | Design implication |
|---|---|---|
| Weights | Parameters × storage bytes | Minimum deployable baseline |
| KV cache | Layers, tokens, active sequences | Cost of concurrency and long inputs |
| Runtime headroom | Observe per engine | Prevents OOM and abrupt latency |
For that reason, “a 16GB GPU is enough” does not answer this workload. Even if the same weights load, insufficient cache and headroom prevent the active-stream target from being met.
Treat KV cache as the product of request length and concurrency
For this capacity calculation, assume a dense-GQA model with 32 layers, 8 KV heads, a head dimension of 128, and BF16 K and V at 2 bytes each. KV per token is 32 × 8 × 128 × 2(K/V) × 2 bytes = 131,072 bytes, or about 128 KiB. The 6,450 tokens formed by 6,000 input tokens plus a 450-token output cap are a simple estimate of about 806 MiB per sequence. With 24 simultaneously active sequences, that is about 18.9 GiB.
| Input | Value | Result |
|---|---|---|
| KV per token | 32×8×128×2×2 bytes | 128 KiB |
| Sequence | 6,450 tokens | About 806 MiB |
| 24 active streams | 806 MiB×24 | About 18.9 GiB |
Confirm the actual model’s layer count, KV heads, precision, and cache-management behavior with the relevant checkpoint and runtime. That is also why cache metrics belong in operations: this estimate is not a fixed fact.
Memory capacity and performance capacity are different
Memory that can hold 24 streams does not mean prefill and decode will meet the TTFT and ITL targets. Long inputs increase prefill work, and overlapping outputs disturb the decode schedule. Batch policy, token budgets, tensor parallelism, and the network all affect performance capacity.
| Decision | Pass criterion | First action on failure |
|---|---|---|
| Memory | Leave cache and runtime headroom | Limit context or active count |
| TTFT | p95 at or below 2.5 seconds | Separate prefill; control queueing |
| ITL | p95 at or below 80ms | Adjust decode budget or replicas |
Assigning 4 GiB of runtime headroom yields weights at 14.9 GiB + KV at 18.9 GiB + headroom at 4 GiB = about 37.8 GiB. The table below compares available memory budgets for one replica. Read it separately from a particular GPU’s marketed capacity or measured usage.
| Available budget | Estimated need | Remaining space | Design decision |
|---|---|---|---|
| 24GiB | 37.8GiB | 13.8GiB short | Exclude the current BF16, context, and concurrency combination |
| 48GiB | 37.8GiB | 10.2GiB | Test actual engine memory and latency |
| 80GiB | 37.8GiB | 42.2GiB | Verify with cost whether extra memory produces throughput |
Four GiB is an initial budget, so measure again under warmup and maximum-length load. Quantization may reduce weights while leaving BF16 KV cache unchanged. Compare prefill and decode performance only after choosing candidates with memory to spare.
Run benchmarks as a representative request matrix
Do not use a card specification or one prompt result as deployment evidence. Fix the model version, quantization, max tokens, batch policy, warmup, region, and replica count, then repeat the matrix below. Because supported paths and quality effects vary by combination, confirm the applicable backend in the official quantization documentation and pass domain evaluation first.
| Group | Input/output | Concurrency | Decision |
|---|---|---|---|
| Short query | 800/250 | 8 | TTFT and ITL p95 |
| Standard RAG | 6,000/450 | 24 | Target SLO and queue |
| Long document | 12,000/600 | 8 | Isolation policy and error rate |
Report throughput with valid completed requests and generated tokens; do not hide cancellations, timeouts, or retries.
After five minutes of warmup, run each group three times for 20 minutes and store group results separately. First use fixed-concurrency tests to observe behavior while requests generate; next increase external arrival rates to 0.25, 0.5, and 1 request per second to observe backlog. A fixed-concurrency test alone does not promise completed work per minute.
Record product-API arrival, queue admission, engine invocation, first token, and completion under the same request ID. Engine TTFT does not stand in for user latency that includes retrieval and API waiting. Start with a queue of at most 12 and a one-second residence limit; requests beyond it receive 429 with a retry time. Alongside p95 for successful requests, record the rejection and timeout rate among all arrivals. Under normal load, use less than 1% as the validation target; in overload tests, observe latency for existing streams and recovery time.
| Run conditions | User latency | Processing outcome | Resource and quality |
|---|---|---|---|
| Model hash, engine, GPU, driver, input group, arrival rate | API TTFT p50/p95/p99, ITL p95, queue-residence p95 | Arrival, completion, 429, cancellation, error counts, and valid completions/second | Peak memory, KV utilization, citation fitness, output-token count |
The dashboard should show request fate, not just GPUs
Alongside GPU utilization, operations need request-length distribution, active sequences, KV cache use, queue residence time, cancellation rate, and error causes. TTFT can worsen when long prefills arrive despite low utilization; user experience can differ even at high utilization if only short decodes are running.
Tenant-level metrics matter too. You need to know whether one tenant’s bulk-document requests raise overall p95, or whether a particular retrieval result inflates input length, in order to adjust fairness policy. Treat personal data and raw prompts under the applicable access policy and keep only the aggregates operations need.
| Signal | Interpretation | First check |
|---|---|---|
| Sustained TTFT p95 breach | Prefill or queue pressure | Input length and queue |
| Sustained ITL p95 breach | Decode contention | Active streams |
| Cache saturation | Approaching memory capacity | Isolate long requests |
Connect alert thresholds to the acceptance targets in the scenario. Measurements must use the same definitions as pre-deployment tests to make trends comparable.
Tune retrieval quality and capacity policy together
In RAG, input tokens are determined before the model. Adding more retrieved results may appear to provide stronger evidence, but it increases prefill time and KV cache while delaying first answers for other requests. Cutting everything short can instead remove context. Retrieval top-k, chunk length, deduplication, reranking, and citation method must therefore change in one test sheet shared by quality and infrastructure teams.
In this scenario, set a token budget before adding document chunks, and state which chunks remain when it is exceeded. The question itself, freshness, authorization, retrieval score, and duplication can set priority. Data isolation between tenants must be guaranteed at retrieval; inference-queue fairness cannot replace access control.
Use two evaluation sets. One is a product-quality set that checks whether answers use supporting documents correctly. The other is a capacity set that mixes short, standard, and long inputs to measure the SLO. Both must pass when the model or retrieval policy changes. Recording whether improvement in one worsens the other tests the intuition that more context or a larger GPU is always better.
Scaling is not only adding replicas; it is failure response
Initially, send only part of traffic to one replica and observe p95 and cache pressure using the same matrix. When TTFT exceeds target, first inspect the queue, input cap, and prefill path. When only ITL worsens, inspect decode contention and active streams. OOM or cache saturation signals that new requests should be limited and the long-document path isolated.
Set scaling conditions from the duration of SLO breaches and queue length, not “GPU at 90%.” A new replica joins only after health checks, model warmup, and rollback criteria pass. For failure, decide in advance the concurrency allowed on remaining replicas, the request-rejection message, and whether a non-streaming alternative exists.
Keep numbers and user cases together when making changes
Capacity planning is a change-management record, not a document for deployment day. When the model version, tokenizer, quantization, maximum context, retrieval policy, or replica count changes, rerun the same matrix. Retain execution environment driver and runtime, prompt version, and measurement start and end time. That prevents unexplained latency from being attributed to a card generation or model name by guesswork.
Review meetings do not look at p50 alone. Use the length, tenant, queue state, and error code of requests with worse p95 to order responses. In the short term, adjust input caps and admission control; in the medium term, change model routing, replicas, or cache policy; in the long term, redesign document-processing experience. Recording the same SLO and measurement definitions makes direct comparison with prior deployments possible even after changing models.