4.4 Load test¶
Before autoscaling anything, see what one replica does under more load than it can batch. The load generator is k6, running as a Job on the CPU nodes — it has no GPU toleration, so it can never land on the node it's testing.
apiVersion: v1
kind: ConfigMap
metadata:
name: k6-load
namespace: vllm
data:
load.js: |
import http from 'k6/http';
import { check } from 'k6';
// 24 concurrent "users", each firing chat completions back to back.
// With --max-num-seqs=16 on the server, one replica admits 16 and
// queues the rest, which is exactly what we want the autoscaler to see.
// VUS and DURATION come from the Job's env (baseline: 60s; autoscale: 15m).
export const options = {
vus: Number(__ENV.VUS || 24),
duration: __ENV.DURATION || '5m',
thresholds: {
http_req_failed: ['rate<0.05'],
},
};
const url = 'http://vllm.vllm.svc/v1/chat/completions';
const prompts = [
'Explain what a Kubernetes taint is in two sentences.',
'Give me three names for a coffee shop on a space station.',
'Summarize why GPUs are hard to schedule in one paragraph.',
'Write a haiku about spot instances.',
];
export default function () {
const body = JSON.stringify({
model: 'qwen2.5-1.5b',
messages: [{ role: 'user', content: prompts[__ITER % prompts.length] }],
max_tokens: 64,
temperature: 0.7,
});
const res = http.post(url, body, {
headers: { 'Content-Type': 'application/json' },
timeout: '120s',
});
check(res, { 'status is 200': (r) => r.status === 200 });
}
24 virtual users, each sending a chat completion and immediately sending the next one when it returns. The server admits 16 at a time (--max-num-seqs), so 8 requests are always waiting.
make baseline applies k6-baseline-job.yaml (60 seconds) and follows its log. While it runs, in another terminal, watch vLLM's own view of the queue:
(No watch on macOS? brew install watch, or while true; do clear; curl -s localhost:8000/metrics | grep -E '^vllm:num_requests_(running|waiting)'; sleep 2; done.)
16 running, 8 waiting, for the whole minute. The replica is saturated by design. When k6 finishes:
█ TOTAL RESULTS
checks_total.......: 418 6.9/s
checks_succeeded...: 100.00% 418 out of 418
checks_failed......: 0.00% 0 out of 418
✓ status is 200
HTTP
http_req_duration..............: avg=3.41s min=0.71s med=3.36s max=5.88s p(90)=4.72s p(95)=5.11s
http_req_failed................: 0.00% 0 out of 418
http_reqs......................: 418 6.9/s
EXECUTION
iterations.....................: 418 6.9/s
vus............................: 24 min=24 max=24
Write down two numbers from your run: http_reqs per second (throughput) and p(95) (tail latency). Yours will differ from the above — L4 vs A10G, Spot vs On-Demand, prompt mix — and that's fine. These are the baseline for one replica. Everything in 4.5 is about moving them.
Why is p95 five seconds for a tiny model?
Because 8 of 24 requests spend their time in the queue before they even start generating. Time-to-first-token is what the queued users feel. Run the same test with --max-num-seqs 64 and p95 drops — until KV cache fills and it climbs again. Batching depth is a latency/throughput dial, not a free lunch; the autoscaler is how you add capacity instead of turning the dial.
What the GPU saw¶
Peek at the DCGM metric for node A during the test — this is the utilisation number Module 5 is built around:
curl -sG 'localhost:9090/api/v1/query' --data-urlencode 'query=max_over_time(DCGM_FI_DEV_GPU_UTIL[2m])' | jq -r '.data.result[] | "\(.metric.Hostname) gpu_util_max=\(.value[1])%"'
Node A pegged; node B (the notebook) at zero. Hold that thought.