dk

Four pitfalls when benchmarking real-time GPU inference on Jetson

Driving inference on a fixed-rate loop and measuring deadline misses is the right way to characterize how a model behaves in deployment rather than back-to-back. But a few things about real-time GPU benchmarking on Jetson are non-obvious, and each one silently produces a plausible-but-wrong number rather than an error. Here are four, with the correct approach and why it matters.

1. Period and deadline are separate knobs

A periodic real-time task has two independent parameters: the period (how often you release work) and the deadline (the time bound you score against). Conflating them — running the loop with its period set equal to the deadline you want to test — breaks the measurement at tight deadlines.

When the deadline under test is near the workload’s compute floor, period ≈ compute time, so utilization → 1. The first cycle that runs slightly long finishes after the next release is already due; that release queues behind it, the queue grows without bound, and response time doesn’t degrade gracefully — it explodes into seconds. That is not a deadline miss; it is a saturated queue.

Drive the loop at a comfortable period and score misses against a tighter deadline:

# period (rate) and deadline are decoupled
infer_bench --hz 100 --deadline-ms 5.0     # 10 ms period, 5 ms deadline

A miss is then P(response > deadline) with deadline < period. The deadline is a threshold you score against, not the rate you drive at.

2. mlockall and CUDA contend for the same RAM on an integrated GPU

The textbook real-time loop uses mlockall(MCL_CURRENT | MCL_FUTURE) so no page fault stalls a cycle. MCL_FUTURE locks every page the process maps from then on. That is fine for CPU work, but it does not coexist with CUDA on Jetson.

An integrated-GPU Jetson has no separate VRAM — CUDA’s buffers come from the same system RAM. So mlockall followed by CUDA initialization makes gigabytes of CUDA allocations permanently unpageable; under root, where the memlock limit is effectively unbounded, it keeps locking until the machine OOMs or hangs.

mlockall, SCHED_FIFO, and affinity all need privilege, so the temptation is to run the whole thing under sudo. Instead, split privilege by concern — clock control and telemetry as root, the inference process unprivileged so CUDA allocates normally:

sudo ./set_domain.sh all        # clock pinning -> root
sudo tegrastats ... &           # telemetry     -> root
# inference itself runs UNPRIVILEGED: no mlockall, no FIFO, CUDA allocates freely
python3 -m harness.infer_bench ...

For a deployment measurement the GPU compute dominates response time, so dropping SCHED_FIFO on the inference thread doesn’t move the numbers — but keeping mlockall away from CUDA is the difference between a run and a reboot.

3. Stop background processes by PID or exact name, never by command-line match

A benchmark cell typically spawns a background adversary and a tegrastats logger, then tears them down. pkill -f tegrastats is the obvious teardown and the wrong one: -f matches against the entire command line of every process, so the script that contains the string tegrastats (in a variable, a comment, or the pkill line itself) matches its own pattern and can kill its own shell.

Use the exact process name, or — better — track the PID you started and kill that:

pkill -x tegrastats              # exact process NAME, not the command line
# or:
tegrastats ... & TEGRA_PID=$!
...
kill "$TEGRA_PID" 2>/dev/null || true

If you started the process, you have its PID. pkill -f with a pattern that appears anywhere in your own invocation is a self-inflicted failure.

4. The workload has to respond to the axis you’re studying

To show that a frequency-aware governor mispicks a clock, the workload’s latency has to actually move with that clock. A memory-bandwidth-bound kernel (a GEMV-shaped decode proxy, for example) saturates above a low frequency: its latency is roughly flat across GPU clocks, so every governor picks the same clock and there is nothing to measure.

A flat curve there is not a null result — it means the instrument can’t see the effect, because for that kernel there is none on this axis. Use a frequency-responsive workload (a CNN, a ViT, an LLM token decoder all scale with GPU clock) and the governor’s mispick becomes visible immediately. Before trusting a benchmark to reveal an effect, confirm the benchmark responds to the variable you’re changing.

Takeaways

← All posts