dk

jetson_clocks was hiding the bug: bwmgr_halt and why EMC locking isn't portable

I have a small measurement rig that pins the external memory controller (EMC) clock on Jetson Orin so I can study how memory speed alone moves inference latency. On my first board — an Orin Nano Super — it had worked for weeks. Lock EMC to 3199 MHz, run the workload, read the numbers; lock it to 204 MHz, run again. Clean.

Then I brought up a second board, an Orin NX 16 GB on a third-party carrier, and the exact same scripts fell apart. Locking the memory clock did nothing. This is the story of how I almost reflashed the board to fix a problem that didn’t exist, and the one-line write that actually fixed it.

The symptom

The lock goes through the BPMP debug interface:

EMC=/sys/kernel/debug/bpmp/debug/clk/emc
echo 1 > $EMC/mrq_rate_locked
echo 3199000000 > $EMC/rate     # ask for 3199 MHz

On the new board, reading rate back gave 665600000 — not what I wrote — and tegrastats showed the clock sitting at the 204 MHz floor regardless of what I requested. Under memory load it stayed pinned low, which is the worst possible failure mode for a memory-bandwidth study: every workload looks slow and nothing responds to the knob I think I’m turning.

I went through the usual suspects. Thermals were fine (~47 °C). The power model was MAXN_SUPER, max_rate read 3199 MHz, EMC wasn’t capped by nvpmodel. The DVFS table was fully populated. So the silicon knew about the high frequencies — it just refused to hold them.

The wrong hypothesis

The new board had one obvious difference from the first: it was a generic-flashed module sitting on a third-party carrier with its own vendor BSP. dmesg had a line that looked like a smoking gun:

bpmp: Fixing up cyclic dependency with external-memory-controller

The story wrote itself: wrong device tree for this carrier, EMC node mis-wired, BPMP can’t honor the clock. The fix would be to reflash the proper vendor BSP with the correct carrier device tree. I unpacked the BSP, found the right image, and had the flashing procedure open in another window.

That reflash would have cost an afternoon and a full environment rebuild — and, as it turns out, it would have fixed nothing.

The two checks that stopped me

Before doing something destructive, I made myself do two things.

First: stop trusting the rate file and measure the clock directly. The debugfs rate value is not the actual EMC frequency. tegrastats is:

$ tegrastats --interval 1000 | grep -o 'EMC_FREQ [0-9]*%@[0-9]*'
EMC_FREQ 0%@204

The @204 is ground truth. Everything I concluded had to agree with that number, not with the file I’d written.

Second: try to refute the device-tree theory before acting on it. My evidence was “empty possible_rates” and “that cyclic-dependency line in dmesg.” So I went back to the working board and checked the same things.

Both fell apart immediately:

Neither “symptom” distinguished the broken board from the working one. My whole case for a device-tree fault was built on two things that are true everywhere. If I’d reflashed, I’d have “fixed” it by coincidence at best.

The actual cause

The real difference between the two boards had nothing to do with the device tree, and everything to do with what each board had run since boot.

EMC frequency on Tegra234 is arbitrated inside the BPMP from two inputs: the static DVFS table, and live bandwidth votes from the BPMP bandwidth manager (bwmgr). mrq_rate_locked adds your requested rate as one more input — but the bandwidth manager keeps arbitrating, and it wins. At idle, demand is low, so EMC drops to the floor and your lock is quietly ignored.

To actually pin the clock, you have to stop the bandwidth manager from arbitrating:

echo 1 > /sys/kernel/debug/bpmp/debug/bwmgr/bwmgr_halt

With that one write in place, locking works exactly. Same board, idle, only bwmgr_halt changes:

steprate filetegrastats
mrq_rate_locked=1; rate=3199000000 (no halt)665600000@204
add bwmgr_halt=13199000000@3199
then rate=21330000002133000000@2133
then rate=665600000665600000@665

The lock goes from “completely ignored” to “exact.”

Why it looked like a board problem

Here’s the part that had me chasing the device tree: jetson_clocks sets bwmgr_halt=1 as part of pinning all the clocks. My first board had run jetson_clocks early on and never rebooted, so the bandwidth manager was already halted — every EMC lock after that “just worked.” The second board was freshly booted, bwmgr_halt was 0, and the same scripts did nothing.

So the failure wasn’t board-specific in any deep sense. It was state-specific. If you ever have one Jetson where EMC locking works and another where it doesn’t, this is the first thing to check:

cat /sys/kernel/debug/bpmp/debug/bwmgr/bwmgr_halt

It’ll almost certainly be 1 on the machine that works and 0 on the one that doesn’t.

The fix, and checking the old data

The fix is to make the lock self-contained instead of depending on whether something else happened to run jetson_clocks first:

lock() {   # lock <rate_hz>, run as root
  [ -e "$BWMGR" ] && echo 1 > "$BWMGR"      # halt the bandwidth manager FIRST
  echo 1   > "$EMC/mrq_rate_locked"
  echo "$1" > "$EMC/rate"
  sleep 1
  echo "requested=$1 actual=$(cat "$EMC/rate")"
}
unlock() {
  echo 0 > "$EMC/mrq_rate_locked"
  [ -e "$BWMGR" ] && echo 0 > "$BWMGR"       # restore dynamic EMC management
}

The first thing I did after this was go back and check the data I’d already collected on the first board, because if EMC locking is this easy to get silently wrong, “it worked for weeks” is not reassuring on its own. Fortunately I’d logged tegrastats through every measurement. The EMC_FREQ field in those logs sat exactly at the intended frequency for every run — 204, 665, 2133, 3199 — because that board’s bandwidth manager really had been halted the whole time. The data held up. But the only reason I could say that with confidence is that the ground-truth clock was in the logs. If you’re pinning EMC for a measurement, log the actual frequency alongside your latencies; the request is not evidence.

Takeaways

← All posts