Services

LLM Inference Server Architecture and GPU Serving

A B2B support tool must protect permissions before it produces a fast sentence. We design document access, retrieval scope, queues, and human handoff as one request path.

A conceptual illustration of a processor, memory, and cooling elements arranged inside a graphite housing.

Design scenario: a B2B support assistant for 240 organizations

Support staff at each organization search contract, product, and incident-response documents to draft and review customer replies. The system holds when no citable document exists and excludes unauthorized documents from both search results and prompts. Scale and timing are design inputs and verification targets.

Organizations
240, with document and user ACLs isolated by organization
Peak requests
180 per minute, with 30 concurrent generation requests as a design input
Input budget
800-token question + up to 6,000 tokens of retrieved context
Output budget
Drafts up to 700 tokens; streaming can be cancelled
Evaluation sample
200 each for citation grounding, access blocking, and abstention decisions; 600 total

The problem is an authorized answer, not a model call

If one company’s contract document appears in another company’s support workspace, the service has failed regardless of response speed. Before retrieval, it establishes organization, user role, and document state, then applies the result identically to retrieval filters and generation context. The model receives only permitted document passages, the question, and the response format.

The second problem is uncertainty. When retrieval scores are low or documents conflict, plausible prose increases the support agent’s verification burden. Responses have three states: grounded draft, needs verification, and handoff to an owner. Defining these states as product requirements keeps the behavior intact when models change.

Request outcome contract
StateInput conditionWhat the user sees
draftPermitted documents exist and supporting passages are sufficientDraft, cited documents, and a send-for-review control
reviewEvidence is insufficient or policy documents conflictDocuments to check and reason for holding
handoffAccess error, risk keyword, or deadline exceededOwner queue, original question, and retrieval record

Requests take a short path and a long path

Conversation requests remain on a synchronous path. Authentication, organization checks, retrieval, reranking, generation, and audit records share one request ID. Source-document intake and embedding generation take an asynchronous path. Uploading a document does not immediately make it answer evidence; only a version that has completed extraction, chunking, ACL assignment, and indexing is promoted to retrieval.

Request sequence and failure handling
StepProcessingAction on failure
1Confirm organization_id and role from the session401 or 403; do not begin retrieval
2Normalize the question and classify riskHand off to policy queue; skip the model call
3Retrieve candidates with organization, role, and document-state filtersReturn review if no candidate exists
4Rerank permitted candidates and fix cited passagesReturn review on conflict
5Stream from the LLM after queue admissionHandoff on deadline; discard partial answer
6Audit the response, evidence, and policy decisionDo not deliver without an initial audit record; retry completion records through a durable queue
Request path from access check to audit record
  1. Session check

    Establish organization_id and role.

  2. Policy branch

    Hand off risk and access errors before the model call.

  3. Filtered retrieval

    Limit candidates by organization, role, and document state.

  4. Reranking

    Select cited passages only from permitted candidates.

  5. Generation and cancellation

    Stream within queue and deadline limits.

  6. Audit record

    Attach decision, evidence, and versions to the request ID.

Retrieval filters come before reranking

Vector retrieval finds semantically close documents; it is not an authorization engine. Index records carry document ID, organization_id, permitted roles, document version, and active state. The query narrows on these attributes first. A reranker operates only within that narrowed candidate set. The generation prompt carries the selected passage’s document ID and version as citation metadata.

The index is for finding candidates quickly. The system rechecks the current ACL in the source authorization store immediately before building a prompt and immediately before delivering a cached answer or stream. A document receiving a revoke event is invalidated immediately in the index and response cache, and blocked from delivery before asynchronous deletion completes. Retrieval-cache keys include organization, role, ACL version, and index version.

Document lifecycle rules
StateRetrievableIncluded in generation contextOperational action
uploadedNoNoAwait extraction
indexedYes, after ACL filteringYes, after ACL recheck before deliveryRecord version and hash
revokingNoNoInvalidate index and cache; confirm deletion
failedNoNoRequest administrator reprocessing

Permission changes apply to responses already in generation. Active requests carry referenced document IDs and ACL versions; revocation cancels the affected stream and generation. The delivery gate checks the current permission version before each chunk and stops delivery if it cannot verify it. It processes the confirmed change and chunk-delivery authorization sequentially through the same gate to prevent a race immediately after a check. It records the result as revoked and discards the incomplete draft in the interface. Tokens already delivered cannot be recalled; this contract blocks subsequent delivery.

GPU serving is designed around peak queues, not averages

The design input is 180 requests per minute. Even at 3 per second, not every request generates concurrently. Long context and 700-token output can still grow KV cache use and queueing. Fix model size, quantization method, context length, and concurrent sequences, then measure with representative questions. vLLM exposes TTFT, inter-token latency, and KV-cache metrics; use names from its metrics documentation in the observability contract.

The approval target is not a GPU name. It is the distribution of queueing, generation, and errors with 30 concurrent requests and the input/output budget. Quantization is one way to reduce memory and requires renewed evaluation of model/backend compatibility and task quality. Confirm supported backends in vLLM quantization documentation.

Initial load-test matrix
AxisValuesRecord
Concurrent generation10, 20, 30TTFT p50/p95, queue time
Context length2,000 / 4,000 / 6,800 tokensKV-cache use, errors
Output limit250 / 700 tokensInter-token p50/p95, cancellation rate
Batch policyDynamic batching on/offThroughput and first-response change

Queueing, deadlines, and cancellation are API contracts

180 requests per minute is an offered load of 3 per second. To sustain that arrival rate with 30 active slots, average active time must be under 10 seconds to retain headroom. Ten seconds is the saturation boundary calculated as 30÷3. A queue absorbs short bursts; it does not increase sustained throughput. Benchmark feasible active time for 6,800-token context and 700-token output. Thirty is therefore an admission ceiling, not a GPU capacity claim.

The initial policy allows 3 active requests per organization, 30 overall, and 36 waiting overall. If the queue is full or an organization exceeds its limit, return 429 with Retry-After. An agent may hand work to a separate review queue rather than wait; this queue is separate from GPU execution. The pre-generation queue deadline is 12 seconds and the complete generation-path deadline is 45 seconds. Cancel streams when the browser disconnects, do not reuse generated tokens as the next request’s context, and use retry keys without sending duplicate answers after worker failure.

Evaluation separates model scores from operating decisions

The evaluation set begins with 600 cases: 200 questions where permitted documents alone should ground the answer, 200 that induce documents with different permissions, and 200 that should abstain because evidence is absent. Rather than force one correct sentence, record citation suitability, abstention decisions, format compliance, and the human reviewer’s edit effort. Use approved samples or anonymized material rather than real customer information.

Stage acceptance criteria
StageAcceptance checkCondition for next stage
Closed evaluationConfirm 0 access violations in the sampleRetrieval and generation logs reviewed
Internal pilotReview and handoff are distinct in the UIAgent feedback can be classified
Limited organization releaseLimit, deadline, and cancellation events are observableRollback procedure rehearsed
Expansion reviewMeasure metrics again on representative trafficCapacity and model-change proposal approved

Deliverables

We leave request state transitions, ACL attributes, document lifecycle, API error formats, prompt and citation formats, and load-test results as one design package. Operators must be able to see which model and index version answered for which organization. A model change is treated as a deployment candidate that must pass the evaluation set and traffic conditions again.

  • Retrieval-filter and role-mapping specification
  • Ingestion jobs and reprocessing/deletion procedures
  • Queue, deadline, cancellation, and error-response contract
  • Measurement dashboard fields and load-test records
  • Model, prompt, and index-version change record

Phased rollout and operational intake

Begin with a closed evaluation using internal documents and limited roles. Then show drafts only to selected organizations while retaining human review before sending. Increase organization count and concurrency limits one stage at a time as observations accumulate. Suspected access violations, missing citations, or rising error rates turn off generation immediately while leaving retrieval results and handoff available.

For an inquiry, provide organization structure, role matrix, document stores and deletion rules, representative questions, good and bad answers, and expected peak requests. These inputs determine the evaluation matrix and request limits before GPU specifications do.

In the initial operations session, trace one request end to end: user role, source documents, applied filters, selected candidate, and model cancellation through one request ID. Without this path, quality, access, and capacity problems cannot be separated. Audit records are a ledger of identifiers, versions, and policy outcomes needed for a decision, not an indiscriminate copy of source text.

Test abnormal paths before expanding traffic: newly indexed documents delayed in ingestion, documents in deletion, sessions with changed roles, disconnected streams, and organizations beyond queue limits. The intended result is that unauthorized documents never enter generation context and both users and operators end in a state that states the next action.

Observe more than request count: organization queue length, cancellation reason, candidate count, candidates remaining after filtering, citation-bearing response rate, and review/handoff rate. Together these separate slow models, empty retrieval, and an organization consuming shared capacity. Record identifiers and aggregates rather than personal data or document bodies.

Change management is more than changing model files. Changing an embedding model, chunking rule, reranking criterion, prompt format, or safety policy can change retrieval and generation together. Each candidate carries its change list, evaluation result, limited-organization observation period, and reversion condition so an operator can reproduce and correct an incident.

References