vLLM Serving Concepts

1. The real problem: a responding endpoint is not proven capacity

vLLM can expose a model through a convenient API, but one successful request proves only that the request completed in that moment. An engineering decision needs the exact model revision, vLLM version, hardware profile, parallelism and scheduling-related configuration, request shape, and concurrency that were measured. This lesson builds a vLLM Serving Configuration & Capacity Evidence Record v1 instead of relying on claims such as “vLLM is fast.” Its decision is valid only for the measured slice.

Keep four layers separate: current tool capability, chosen configuration, measured evidence, and product decision. If they collapse into one another, a later audit cannot reconstruct why a configuration was allowed.

2. Serving-record identity must be immutable and reconstructable

The minimum identity includes model_revision, vllm_version, server_configuration_hash, and hardware_profile. A served_model_name or alias is convenient but does not replace revision or configuration identity. Parallelism, max_model_len, memory budget, batching/sequence limits, and warmup state belong to the evidence context as well.

A materially different configuration gets a new evidence record. Do not mutate the previous qualification in place. Months later, the team should still be able to tell exactly which runtime and configuration produced a reported p95.

3. Treat PagedAttention as a KV-cache management concept

PagedAttention is a core vLLM concept for managing attention/KV-cache memory. Its instructional value here is not a fixed speed promise; it shows that serving behavior depends on cache management and scheduling. Changing context length, batch composition, or concurrency can create a different memory and queue regime, so evidence from one workload does not automatically transfer to another.

A mechanism name is not a performance result. “PagedAttention is part of this runtime” is a tool fact. “This configuration meets our SLO at concurrency 12” requires measurement of that slice.

4. Continuous batching, chunked prefill, and prefix caching are capabilities, not proof

Current vLLM exposes continuous batching and capabilities such as chunked prefill and prefix caching. They may affect queueing, prefill behavior, cache reuse, and throughput, but none of them alone proves product capacity. If configuration or workload changes the way these mechanisms behave, the old benchmark is historical evidence, not current qualification.

Record capability separately from measurement. For example, runtime support for prefix caching is not evidence that the current workload has a useful prefix-reuse pattern.

5. Parallelism is not one number

Serving can use tensor, pipeline, data, or other supported parallel strategies. GPU count alone is not enough provenance. Record tensor_parallel_size and pipeline_parallel_size, and include other decisive topology choices when applicable.

Config A with TP=1 and Config B with TP=2 are different serving configurations even when the model revision is unchanged. Parallelism can alter memory layout, communication, and queue behavior, so qualification does not implicitly transfer.

6. Hash capacity-shaping configuration instead of describing it loosely

The bounded configuration surface in this lesson includes max_model_len, gpu_memory_utilization, max_num_batched_tokens, and max_num_seqs alongside parallelism. Environments can differ in supported knobs and defaults, so server_configuration_hash should represent the effective configuration rather than a casual command snippet.

When a decisive knob changes, raise a retest_trigger. A server that still starts and serves the same model has not automatically preserved the old capacity evidence.

7. Bind capacity to request shape and concurrency

Every measurement needs an input range, output range, and concurrency range. Example: input up to 2048 tokens, output up to 512, concurrency up to 8. A new target of 4096/1024 at concurrency 12 is outside that measured slice.

This lesson forbids unsupported extrapolation. The bounded decision may be PASS, CONDITIONAL, or DEFER, but a PASS always means “for this configuration and this measured slice,” never “for every workload.”

8. Warmup state and measurement protocol are evidence

A benchmark with unknown warmup state is hard to compare or qualify. The record should say whether warmup was completed, intentionally omitted, or unknown. Unknown is insufficient for a bounded capacity PASS.

The lab uses deterministic synthetic fixtures so the concept can be learned without a GPU or external API. In production the same evidence discipline applies to real measurements: exact versions, a declared workload, a repeatable protocol, and explicit sample evidence.

9. p50, p95, queue time, and throughput answer different questions

p50 describes the median experience; p95 exposes more of the tail. Queue time separates waiting from execution, while throughput captures work completed over time. Errors and timeouts must remain visible instead of disappearing inside attractive averages.

Define a product target, for example p95 ≤ 1800 ms, throughput ≥ 120 tokens/s, and zero timeouts for the declared slice. These thresholds are product design choices; vLLM documentation does not choose them for you.

10. Worked example: do not compare Config A and Config B by model name

Assume both configurations serve the same atlas-7b revision. Config A uses TP=1, max model length 8192, and max sequences 8. Config B uses TP=2, max model length 16384, and a different batching limit. A successful single request on B does not make A's capacity record evidence for B.

If the fixture shows that A meets its SLO for a small request shape at concurrency 8, only that slice passes. For B or higher concurrency, DEFER pending fresh measurement is more correct than a fabricated PASS.

11. OpenAI-compatible means a familiar interface, not complete server security

vLLM provides OpenAI-style APIs, but the security boundary must be recorded separately. Current documentation warns that --api-key does not authenticate every endpoint on the same HTTP server. A successful /v1/... request or the presence of an API key therefore does not prove that the entire server surface is protected.

The evidence record stores a security_boundary_note: what the API key covers and what external reverse-proxy/network boundary is expected for exposure. This lesson does not own full production security architecture; it only prevents the false claim that API-key configuration secures every endpoint.

12. Lab: eight deterministic fixtures

The lab needs no GPU or provider call. Eight cases cover a valid baseline, a target beyond measured concurrency, changed config hash, incomplete provenance, unknown warmup, an SLO failure, a missing security-boundary note, and a transfer case with larger context/concurrency.

The learner function returns eight fields: provenance_state, measurement_state, slice_state, service_state, security_boundary_state, retest_state, final_decision, and reason_class. The evaluator checks the state machine, not prose wording.

13. Decision order: provenance → measurement → slice → SLO → boundary

Check provenance first. Missing model revision, vLLM version, configuration hash, or hardware profile makes the capacity evidence unreliable. Then check measurement completeness and warmup. Next compare the target with the measured request/concurrency slice. Only then evaluate SLO and errors/timeouts. Keep the security-boundary note independent.

A configuration can satisfy performance evidence while its boundary note is incomplete; that can produce CONDITIONAL rather than pretending the issue is a benchmark failure. Keep the reason typed.

14. Define retest triggers before you need them

A changed model revision, vLLM version, hardware profile, configuration hash, parallelism, max model length, batching/sequence limit, or target request/concurrency slice can require retesting. Triggers are not meant to block every deployment; they prevent evidence reuse when decisive dimensions changed.

The new record should say what changed and which evidence must be rebuilt. “It is the same model” is insufficient because serving qualification is not bound to model identity alone.

15. Exit checklist and transfer

Before PASS, ask: exact revision? vLLM version? hardware and config hash? parallelism and capacity limits? measured input/output range and concurrency? known warmup state? p50/p95/queue/throughput plus errors/timeouts? security-boundary note? explicit retest triggers?

Transfer task: keep the model revision unchanged but increase maximum context and target concurrency. Rebuild the record and decide PASS, CONDITIONAL, or DEFER from available evidence without extrapolating the old benchmark. A well-supported DEFER is a valid engineering result.