Riverfront AI Labs

Inference performance audit

Where the GPU time actually goes.

An audit is five to ten days of profiling your production inference stack and reporting, with evidence, what is costing you latency and money. Below is the catalogue we work through and the numbers we report against, so you know what you are buying before the call.

Book an audit

What we look for

Four layers, worked in this order. Findings from the earlier ones usually make the later ones unnecessary.

The serving layer

Where most of the recoverable money sits, and the last place anyone looks. Every serving framework ships defaults tuned for a benchmark, and almost nobody revisits them against the traffic they actually get.

The question to ask yourselfWhat is your GPU utilization during an ordinary business hour, not at peak?

  1. Batching sized for the install, not the traffic

    What it looks likeRequests queue while the card sits half idle

    Concurrency and batch limits get set once and never tuned against real arrival patterns. This is routinely the largest single recoverable number in the stack, and it costs nothing but configuration.

  2. Prefill blocking decode

    What it looks likeTail latency spikes whenever a long prompt lands

    One long prompt stalls every decode in flight behind it. Chunked prefill and a scheduler that keeps the two phases apart fix it without buying hardware.

  3. No prefix caching on a shared preamble

    What it looks likeIdentical leading tokens recomputed on every request

    A long system prompt or a RAG preamble means you are paying to compute the same tokens on every call. The fix is a configuration flag and a cache budget.

  4. Speculative decoding left on the table

    What it looks likeDecode-bound workload with compute to spare

    Memory-bound decode has idle FLOPs. Whether a draft model pays for itself depends on the acceptance rate for your traffic, which is a day of measurement rather than a matter of opinion.

KV cache and memory

The cache decides how many requests fit on the card, and therefore what each one costs. Most stacks size it by accident and then buy more GPUs.

The question to ask yourselfHow much of your GPU memory holds live cache, and how much is reserved for a context length nobody ever sends?

  1. Context length set far above real usage

    What it looks likeLarge reserved cache the workload never fills

    The declared maximum context is a memory reservation, not a promise. Sizing it to the 99th percentile of your real prompts frees space for concurrency you are already paying for.

  2. Cache held at full precision

    What it looks likeKV in FP16 while the weights are quantized

    For long-context decode the cache, not the weights, dominates memory. Quantizing it is usually the cheapest way to raise batch size, but it needs an accuracy check against your own evals rather than a blanket recommendation.

  3. Fragmentation and reservation waste

    What it looks likeAllocator failures well below nominal capacity

    Contiguous per-request allocation wastes whatever the request does not use. Paged attention exists for this; not every deployment is actually using it.

  4. Precision chosen once, at the start

    What it looks likeFP16 weights on hardware with FP8 units

    Newer cards have paths the deployment predates. We benchmark the candidates against your own evaluation set and report the trade-off with numbers on both sides.

Quantization is the recommendation most often made carelessly. We will not propose a precision change without measuring it against your evaluation set. If the accuracy cost is real, the report says so rather than burying it.

Kernels

Once configuration is exhausted, the profile stops being a story about settings and becomes one about memory bandwidth. This is where the work gets specialised, and where most consultancies stop.

The question to ask yourselfDo you know whether your decode step is compute-bound or memory-bound? If not, that is the first thing an audit tells you.

  1. Wrong attention backend for the workload

    What it looks likeKernel time dominated by attention at your sequence lengths

    Backends are tuned for different regimes: short prompts and heavy batching behave nothing like long-context decode. Picking against your own length distribution is free performance.

  2. Unfused pointwise chains

    What it looks likeMemory traffic far above the arithmetic requires

    Every unfused elementwise op is another full round trip to HBM. A Triton kernel that fuses the chain often takes an afternoon and shows up immediately in the profile.

  3. Uncoalesced access in custom operators

    What it looks likeAchieved bandwidth well under the roofline

    Hand-written ops carried over from research code are the usual source. The fix is layout, not cleverness.

  4. Occupancy capped by register pressure

    What it looks likeFew active warps per SM despite a large grid

    The compiler makes a conservative choice and the kernel never gets enough warps in flight to hide latency. Sometimes this needs PTX; usually it needs a launch-bound and a restructure.

Serving infrastructure

Throughput on a benchmark rig is not throughput in production. The gap between the two is almost always in how the fleet scales, routes and starts.

The question to ask yourselfWhen traffic doubles, how long before you are serving it at your normal latency?

  1. Autoscaling on CPU metrics

    What it looks likeReplicas added late, removed early, or both

    CPU utilization tells you nothing about token load. Scaling on queue depth and running-sequence count tracks the thing that actually saturates.

  2. Cold starts pulling weights over the network

    What it looks likeMinutes between a scale-up decision and a served request

    New capacity that arrives after the spike has passed is capacity you paid for and did not use. Cached layers and warm pools close most of it.

  3. No length-aware routing

    What it looks likeLong and short requests contending on the same replica

    Mixing a 60k-token summarisation into a chat fleet ruins latency for everyone on that card. Splitting the traffic is often the entire fix.

  4. Parallelism mismatched to the hardware

    What it looks likeTensor parallelism crossing a slow interconnect

    Sharding across a link the model was not sized for turns compute into communication. The right degree depends on your topology, not on the model card.

What we measure

Every claim in the report is attached to one of these, before and after.

TTFT
Time to first token, which is what a user perceives as responsiveness. Dominated by prefill and by queueing, which are two very different fixes.
TPOT
Time per output token once streaming starts. Memory-bandwidth bound in almost every case, and the number batching trades against.
Throughput
Output tokens per second per GPU. The number your bill is actually a function of.
Goodput
Requests per second served inside your latency target. Throughput without this is a number you cannot sell to anyone.
Cost / M tokens
Instance cost divided by delivered tokens. The translation layer between an engineering change and a finance conversation.
MFU
How much of the hardware's arithmetic capability you are actually using, against how much you are renting.

What we need from you

Four things. Without the first two there is no audit, only opinions.

Contact

Send us the profile you already have.

riverfront.aic@gmail.com

Even a rough one: model, GPUs, concurrency, current latency. If we can see from that alone that there is little headroom left, we will tell you on the first call rather than sell you an audit.