LLM Inference Server Architecture and GPU Serving
A B2B support tool must protect permissions before it produces a fast sentence. We design document access, retrieval scope, queues, and human handoff as one request path.

Design scenario: a B2B support assistant for 240 organizations
Support staff at each organization search contract, product, and incident-response documents to draft and review customer replies. The system holds when no citable document exists and excludes unauthorized documents from both search results and prompts. Scale and timing are design inputs and verification targets.
- Organizations
- 240, with document and user ACLs isolated by organization
- Peak requests
- 180 per minute, with 30 concurrent generation requests as a design input
- Input budget
- 800-token question + up to 6,000 tokens of retrieved context
- Output budget
- Drafts up to 700 tokens; streaming can be cancelled
- Evaluation sample
- 200 each for citation grounding, access blocking, and abstention decisions; 600 total
The problem is an authorized answer, not a model call
If one company’s contract document appears in another company’s support workspace, the service has failed regardless of response speed. Before retrieval, it establishes organization, user role, and document state, then applies the result identically to retrieval filters and generation context. The model receives only permitted document passages, the question, and the response format.
The second problem is uncertainty. When retrieval scores are low or documents conflict, plausible prose increases the support agent’s verification burden. Responses have three states: grounded draft, needs verification, and handoff to an owner. Defining these states as product requirements keeps the behavior intact when models change.
| State | Input condition | What the user sees |
|---|---|---|
| draft | Permitted documents exist and supporting passages are sufficient | Draft, cited documents, and a send-for-review control |
| review | Evidence is insufficient or policy documents conflict | Documents to check and reason for holding |
| handoff | Access error, risk keyword, or deadline exceeded | Owner queue, original question, and retrieval record |
Requests take a short path and a long path
Conversation requests remain on a synchronous path. Authentication, organization checks, retrieval, reranking, generation, and audit records share one request ID. Source-document intake and embedding generation take an asynchronous path. Uploading a document does not immediately make it answer evidence; only a version that has completed extraction, chunking, ACL assignment, and indexing is promoted to retrieval.
| Step | Processing | Action on failure |
|---|---|---|
| 1 | Confirm organization_id and role from the session | 401 or 403; do not begin retrieval |
| 2 | Normalize the question and classify risk | Hand off to policy queue; skip the model call |
| 3 | Retrieve candidates with organization, role, and document-state filters | Return review if no candidate exists |
| 4 | Rerank permitted candidates and fix cited passages | Return review on conflict |
| 5 | Stream from the LLM after queue admission | Handoff on deadline; discard partial answer |
| 6 | Audit the response, evidence, and policy decision | Do not deliver without an initial audit record; retry completion records through a durable queue |
- Session check
Establish organization_id and role.
- Policy branch
Hand off risk and access errors before the model call.
- Filtered retrieval
Limit candidates by organization, role, and document state.
- Reranking
Select cited passages only from permitted candidates.
- Generation and cancellation
Stream within queue and deadline limits.
- Audit record
Attach decision, evidence, and versions to the request ID.
Retrieval filters come before reranking
Vector retrieval finds semantically close documents; it is not an authorization engine. Index records carry document ID, organization_id, permitted roles, document version, and active state. The query narrows on these attributes first. A reranker operates only within that narrowed candidate set. The generation prompt carries the selected passage’s document ID and version as citation metadata.
The index is for finding candidates quickly. The system rechecks the current ACL in the source authorization store immediately before building a prompt and immediately before delivering a cached answer or stream. A document receiving a revoke event is invalidated immediately in the index and response cache, and blocked from delivery before asynchronous deletion completes. Retrieval-cache keys include organization, role, ACL version, and index version.
| State | Retrievable | Included in generation context | Operational action |
|---|---|---|---|
| uploaded | No | No | Await extraction |
| indexed | Yes, after ACL filtering | Yes, after ACL recheck before delivery | Record version and hash |
| revoking | No | No | Invalidate index and cache; confirm deletion |
| failed | No | No | Request administrator reprocessing |
Permission changes apply to responses already in generation. Active requests carry referenced document IDs and ACL versions; revocation cancels the affected stream and generation. The delivery gate checks the current permission version before each chunk and stops delivery if it cannot verify it. It processes the confirmed change and chunk-delivery authorization sequentially through the same gate to prevent a race immediately after a check. It records the result as revoked and discards the incomplete draft in the interface. Tokens already delivered cannot be recalled; this contract blocks subsequent delivery.
GPU serving is designed around peak queues, not averages
The design input is 180 requests per minute. Even at 3 per second, not every request generates concurrently. Long context and 700-token output can still grow KV cache use and queueing. Fix model size, quantization method, context length, and concurrent sequences, then measure with representative questions. vLLM exposes TTFT, inter-token latency, and KV-cache metrics; use names from its metrics documentation in the observability contract.
The approval target is not a GPU name. It is the distribution of queueing, generation, and errors with 30 concurrent requests and the input/output budget. Quantization is one way to reduce memory and requires renewed evaluation of model/backend compatibility and task quality. Confirm supported backends in vLLM quantization documentation.
| Axis | Values | Record |
|---|---|---|
| Concurrent generation | 10, 20, 30 | TTFT p50/p95, queue time |
| Context length | 2,000 / 4,000 / 6,800 tokens | KV-cache use, errors |
| Output limit | 250 / 700 tokens | Inter-token p50/p95, cancellation rate |
| Batch policy | Dynamic batching on/off | Throughput and first-response change |
Queueing, deadlines, and cancellation are API contracts
180 requests per minute is an offered load of 3 per second. To sustain that arrival rate with 30 active slots, average active time must be under 10 seconds to retain headroom. Ten seconds is the saturation boundary calculated as 30÷3. A queue absorbs short bursts; it does not increase sustained throughput. Benchmark feasible active time for 6,800-token context and 700-token output. Thirty is therefore an admission ceiling, not a GPU capacity claim.
The initial policy allows 3 active requests per organization, 30 overall, and 36 waiting overall. If the queue is full or an organization exceeds its limit, return 429 with Retry-After. An agent may hand work to a separate review queue rather than wait; this queue is separate from GPU execution. The pre-generation queue deadline is 12 seconds and the complete generation-path deadline is 45 seconds. Cancel streams when the browser disconnects, do not reuse generated tokens as the next request’s context, and use retry keys without sending duplicate answers after worker failure.
Evaluation separates model scores from operating decisions
The evaluation set begins with 600 cases: 200 questions where permitted documents alone should ground the answer, 200 that induce documents with different permissions, and 200 that should abstain because evidence is absent. Rather than force one correct sentence, record citation suitability, abstention decisions, format compliance, and the human reviewer’s edit effort. Use approved samples or anonymized material rather than real customer information.
| Stage | Acceptance check | Condition for next stage |
|---|---|---|
| Closed evaluation | Confirm 0 access violations in the sample | Retrieval and generation logs reviewed |
| Internal pilot | Review and handoff are distinct in the UI | Agent feedback can be classified |
| Limited organization release | Limit, deadline, and cancellation events are observable | Rollback procedure rehearsed |
| Expansion review | Measure metrics again on representative traffic | Capacity and model-change proposal approved |
Deliverables
We leave request state transitions, ACL attributes, document lifecycle, API error formats, prompt and citation formats, and load-test results as one design package. Operators must be able to see which model and index version answered for which organization. A model change is treated as a deployment candidate that must pass the evaluation set and traffic conditions again.
- Retrieval-filter and role-mapping specification
- Ingestion jobs and reprocessing/deletion procedures
- Queue, deadline, cancellation, and error-response contract
- Measurement dashboard fields and load-test records
- Model, prompt, and index-version change record
Phased rollout and operational intake
Begin with a closed evaluation using internal documents and limited roles. Then show drafts only to selected organizations while retaining human review before sending. Increase organization count and concurrency limits one stage at a time as observations accumulate. Suspected access violations, missing citations, or rising error rates turn off generation immediately while leaving retrieval results and handoff available.
For an inquiry, provide organization structure, role matrix, document stores and deletion rules, representative questions, good and bad answers, and expected peak requests. These inputs determine the evaluation matrix and request limits before GPU specifications do.
In the initial operations session, trace one request end to end: user role, source documents, applied filters, selected candidate, and model cancellation through one request ID. Without this path, quality, access, and capacity problems cannot be separated. Audit records are a ledger of identifiers, versions, and policy outcomes needed for a decision, not an indiscriminate copy of source text.
Test abnormal paths before expanding traffic: newly indexed documents delayed in ingestion, documents in deletion, sessions with changed roles, disconnected streams, and organizations beyond queue limits. The intended result is that unauthorized documents never enter generation context and both users and operators end in a state that states the next action.
Observe more than request count: organization queue length, cancellation reason, candidate count, candidates remaining after filtering, citation-bearing response rate, and review/handoff rate. Together these separate slow models, empty retrieval, and an organization consuming shared capacity. Record identifiers and aggregates rather than personal data or document bodies.
Change management is more than changing model files. Changing an embedding model, chunking rule, reranking criterion, prompt format, or safety policy can change retrieval and generation together. Each candidate carries its change list, evaluation result, limited-organization observation period, and reversion condition so an operator can reproduce and correct an incident.