NUMA-Aware GPU Capacity Planning: From Batch Scheduling to p99 Latency on Multi-Tenant Servers
Discover effective strategies for optimizing GPU capacity planning and enhancing performance in multi-tenant environments to minimize latency and avoid…
GPU infrastructure failures rarely look like
Frequently Asked Questions
Why does NUMA-aware GPU capacity planning matter for p99 latency on multi-tenant servers?
NUMA-aware planning reduces cross-socket memory traffic, which otherwise increases scheduling variance and tail latency. In multi-tenant setups, additional contention amplifies those effects, making p99 much more sensitive than average latency. By aligning CPU affinity, memory placement, and GPU access patterns to the same NUMA domain, you stabilize performance under load and improve repeatability.
How do batch scheduling and queuing policies affect tail latency (p99), not just throughput?
Batch scheduling improves GPU utilization but can worsen p99 if batches wait too long in queues or if batch sizes vary by tenant. Queueing discipline (e.g., fairness vs priority) changes how quickly requests are served during bursts. For tail control, you typically need workload-aware batch sizing, bounded queue times, and tenant-aware admission control, not just max throughput.
What are common multi-tenant interference sources beyond GPU compute that impact p99 latency?
Beyond compute, p99 can be driven by PCIe/NVLink contention, host memory bandwidth limits, NUMA cross-traffic, CPU scheduling delays, cache and memory allocator effects, and driver-level synchronization. Even when GPUs are not saturated, shared host resources can create sporadic stalls. Capacity planning should include these shared bottlenecks and treat noisy-neighbor effects as first-class risks.
How should capacity planning translate target p99 latency into concrete resource headroom?
Instead of relying on average GPU utilization, derive headroom from observed or modeled queueing delay distributions. A practical approach is to map latency SLOs to maximum allowable wait time plus compute time, then choose the concurrency/arrival limits that keep the p99 within budget. Repeat this per tenant class, because burstiness and request size distributions change the feasible capacity.
When planning for failures, why can “GPU is alive” be misleading, and what should be monitored?
A GPU can appear healthy while the data path degrades, such as stalled DMA, NUMA memory pressure, PCIe retries, or driver/firmware hiccups. Failures often manifest as rising queueing delay, jitter, or sudden tail-latency spikes. Monitoring should include end-to-end request latency percentiles, GPU copy/compute overlap, PCIe/NVLink error counters, and host-side CPU/memory saturation signals.
What NUMA-aware actions typically provide the biggest gains for GPU-serving workloads?
Big wins usually come from binding inference/training worker threads to CPU cores within the GPU’s local NUMA node, using NUMA-local memory allocation for input staging, and ensuring predictable placement for pinned buffers. You may also tune data transfer paths (CPU->GPU copies) to avoid cross-socket traffic. The goal is to reduce variance first, then maximize utilization.