Technical guides

Choosing a GPU for LLM inference: Memory, latency, and capacity

GPU capacity is not decided by whether a model fits. Calculate whether the service can keep its first-token and streaming-latency promises while serving the same work concurrently, then validate it with representative request patterns.

A charcoal metal structure resembling a precision server core and an aircraft wing

Design scenario: an 8B RAG server for B2B customer support

This scenario covers a service in which agents at 120 customer organizations search Korean and English policy documents and review drafted replies. Scale and timing are design inputs and validation targets.

Documents and permissions
120 customer organizations / 60,000 policy documents / search scope isolated by organization
Model
8B dense, BF16 weights
Request
6,000 input tokens, 450 output tokens
Peak
24 active streams, 12 queued requests
Target
TTFT p95 at or below 2.5 seconds; ITL p95 at or below 80ms

Set the SLO and admission rules before sizing capacity

Success for this service does not mean high GPU utilization. A tenant must see an answer begin even after submitting a long policy document, while another tenant’s short question must not wait indefinitely. TTFT is the time from a request reaching the server to delivery of its first output token. ITL is the interval between adjacent output tokens while streaming. TTFT p95 is the 95th percentile of first-token latency by request; ITL p95 is the 95th percentile of observed output-token intervals. Keeping those sampling units distinct prevents long requests from being hidden.

Admission control is part of capacity design. Once 24 streams are active, new requests enter a queue; once 12 are waiting, the service returns a retryable response. Product policy determines whether long-context requests use a separate queue or lower priority. Unlimited concurrency may sound accommodating, but it can break the SLO of requests already in progress.

vLLM metrics provides the signals needed to observe TTFT, inter-token latency, request stages, and KV cache.

Capacity and protection flow for a RAG request
  1. Admission

    Check tenant limits and the queue

  2. Retrieval

    Set document chunks and the input-token budget

  3. Prefill

    Observe TTFT and long-input pressure

  4. Decode

    Control ITL and active streams

  5. Completion

    Record metric, cost, and quality samples

Weight calculation is the starting line

A simple lower bound for storing 8B parameters at 2 bytes each in BF16 is 8,000,000,000 × 2 = 16,000,000,000 bytes, or about 14.9 GiB. That calculation covers weights only. It excludes execution-engine workspace, communication buffers, graphs, and fragmentation headroom.

Separating the memory budget
AreaHow to calculate or verify itDesign implication
WeightsParameters × storage bytesMinimum deployable baseline
KV cacheLayers, tokens, active sequencesCost of concurrency and long inputs
Runtime headroomObserve per enginePrevents OOM and abrupt latency

For that reason, “a 16GB GPU is enough” does not answer this workload. Even if the same weights load, insufficient cache and headroom prevent the active-stream target from being met.

Treat KV cache as the product of request length and concurrency

For this capacity calculation, assume a dense-GQA model with 32 layers, 8 KV heads, a head dimension of 128, and BF16 K and V at 2 bytes each. KV per token is 32 × 8 × 128 × 2(K/V) × 2 bytes = 131,072 bytes, or about 128 KiB. The 6,450 tokens formed by 6,000 input tokens plus a 450-token output cap are a simple estimate of about 806 MiB per sequence. With 24 simultaneously active sequences, that is about 18.9 GiB.

KV sizing formula for review
InputValueResult
KV per token32×8×128×2×2 bytes128 KiB
Sequence6,450 tokensAbout 806 MiB
24 active streams806 MiB×24About 18.9 GiB

Confirm the actual model’s layer count, KV heads, precision, and cache-management behavior with the relevant checkpoint and runtime. That is also why cache metrics belong in operations: this estimate is not a fixed fact.

Memory capacity and performance capacity are different

Memory that can hold 24 streams does not mean prefill and decode will meet the TTFT and ITL targets. Long inputs increase prefill work, and overlapping outputs disturb the decode schedule. Batch policy, token budgets, tensor parallelism, and the network all affect performance capacity.

Decisions that separate two kinds of capacity
DecisionPass criterionFirst action on failure
MemoryLeave cache and runtime headroomLimit context or active count
TTFTp95 at or below 2.5 secondsSeparate prefill; control queueing
ITLp95 at or below 80msAdjust decode budget or replicas

Assigning 4 GiB of runtime headroom yields weights at 14.9 GiB + KV at 18.9 GiB + headroom at 4 GiB = about 37.8 GiB. The table below compares available memory budgets for one replica. Read it separately from a particular GPU’s marketed capacity or measured usage.

Review of 24 streams by one-replica memory budget
Available budgetEstimated needRemaining spaceDesign decision
24GiB37.8GiB13.8GiB shortExclude the current BF16, context, and concurrency combination
48GiB37.8GiB10.2GiBTest actual engine memory and latency
80GiB37.8GiB42.2GiBVerify with cost whether extra memory produces throughput

Four GiB is an initial budget, so measure again under warmup and maximum-length load. Quantization may reduce weights while leaving BF16 KV cache unchanged. Compare prefill and decode performance only after choosing candidates with memory to spare.

Run benchmarks as a representative request matrix

Do not use a card specification or one prompt result as deployment evidence. Fix the model version, quantization, max tokens, batch policy, warmup, region, and replica count, then repeat the matrix below. Because supported paths and quality effects vary by combination, confirm the applicable backend in the official quantization documentation and pass domain evaluation first.

Measurement matrix and acceptance targets
GroupInput/outputConcurrencyDecision
Short query800/2508TTFT and ITL p95
Standard RAG6,000/45024Target SLO and queue
Long document12,000/6008Isolation policy and error rate

Report throughput with valid completed requests and generated tokens; do not hide cancellations, timeouts, or retries.

After five minutes of warmup, run each group three times for 20 minutes and store group results separately. First use fixed-concurrency tests to observe behavior while requests generate; next increase external arrival rates to 0.25, 0.5, and 1 request per second to observe backlog. A fixed-concurrency test alone does not promise completed work per minute.

Record product-API arrival, queue admission, engine invocation, first token, and completion under the same request ID. Engine TTFT does not stand in for user latency that includes retrieval and API waiting. Start with a queue of at most 12 and a one-second residence limit; requests beyond it receive 429 with a retry time. Alongside p95 for successful requests, record the rejection and timeout rate among all arrivals. Under normal load, use less than 1% as the validation target; in overload tests, observe latency for existing streams and recovery time.

Result columns to retain for every test run
Run conditionsUser latencyProcessing outcomeResource and quality
Model hash, engine, GPU, driver, input group, arrival rateAPI TTFT p50/p95/p99, ITL p95, queue-residence p95Arrival, completion, 429, cancellation, error counts, and valid completions/secondPeak memory, KV utilization, citation fitness, output-token count

The dashboard should show request fate, not just GPUs

Alongside GPU utilization, operations need request-length distribution, active sequences, KV cache use, queue residence time, cancellation rate, and error causes. TTFT can worsen when long prefills arrive despite low utilization; user experience can differ even at high utilization if only short decodes are running.

Tenant-level metrics matter too. You need to know whether one tenant’s bulk-document requests raise overall p95, or whether a particular retrieval result inflates input length, in order to adjust fairness policy. Treat personal data and raw prompts under the applicable access policy and keep only the aggregates operations need.

Examples of alert conditions
SignalInterpretationFirst check
Sustained TTFT p95 breachPrefill or queue pressureInput length and queue
Sustained ITL p95 breachDecode contentionActive streams
Cache saturationApproaching memory capacityIsolate long requests

Connect alert thresholds to the acceptance targets in the scenario. Measurements must use the same definitions as pre-deployment tests to make trends comparable.

Tune retrieval quality and capacity policy together

In RAG, input tokens are determined before the model. Adding more retrieved results may appear to provide stronger evidence, but it increases prefill time and KV cache while delaying first answers for other requests. Cutting everything short can instead remove context. Retrieval top-k, chunk length, deduplication, reranking, and citation method must therefore change in one test sheet shared by quality and infrastructure teams.

In this scenario, set a token budget before adding document chunks, and state which chunks remain when it is exceeded. The question itself, freshness, authorization, retrieval score, and duplication can set priority. Data isolation between tenants must be guaranteed at retrieval; inference-queue fairness cannot replace access control.

Use two evaluation sets. One is a product-quality set that checks whether answers use supporting documents correctly. The other is a capacity set that mixes short, standard, and long inputs to measure the SLO. Both must pass when the model or retrieval policy changes. Recording whether improvement in one worsens the other tests the intuition that more context or a larger GPU is always better.

Scaling is not only adding replicas; it is failure response

Initially, send only part of traffic to one replica and observe p95 and cache pressure using the same matrix. When TTFT exceeds target, first inspect the queue, input cap, and prefill path. When only ITL worsens, inspect decode contention and active streams. OOM or cache saturation signals that new requests should be limited and the long-document path isolated.

Set scaling conditions from the duration of SLO breaches and queue length, not “GPU at 90%.” A new replica joins only after health checks, model warmup, and rollback criteria pass. For failure, decide in advance the concurrency allowed on remaining replicas, the request-rejection message, and whether a non-streaming alternative exists.

Keep numbers and user cases together when making changes

Capacity planning is a change-management record, not a document for deployment day. When the model version, tokenizer, quantization, maximum context, retrieval policy, or replica count changes, rerun the same matrix. Retain execution environment driver and runtime, prompt version, and measurement start and end time. That prevents unexplained latency from being attributed to a card generation or model name by guesswork.

Review meetings do not look at p50 alone. Use the length, tenant, queue state, and error code of requests with worse p95 to order responses. In the short term, adjust input caps and admission control; in the medium term, change model routing, replicas, or cache policy; in the long term, redesign document-processing experience. Recording the same SLO and measurement definitions makes direct comparison with prior deployments possible even after changing models.

References