← Back to Blog

Teaching the Lab a New Trick: Bringing the Strix Halo NPU Online

We’ve been building a dedicated NPU lane in the lab on Strix Halo — not as a gimmick, but as a serious path for efficient always-on inference, lighter agent jobs, and future local AI workflows. Here’s what’s already working, what took actual effort, and what’s still blocking first real NPU inference.

There’s a difference between having an NPU and using an NPU.

A lot of modern hardware ships with AI silicon now, which is great for spec sheets and mildly suspicious keynote slides. But turning that into a real, usable local inference lane on Linux is a more interesting exercise.

That’s what we’ve been working on in the lab recently.

The goal was simple enough on paper: take the Strix Halo machine, expose its NPU properly into a dedicated container, wire up the runtime stack we actually want, and build a clean path for efficient local inference that doesn’t need to burn the bigger GPU lanes on lightweight jobs.

As usual, the paper version was simpler than the real version.

Why we care about the NPU lane

The NPU lane is not meant to replace every other model backend in the lab.

That would be silly.

We already have other lanes that are better suited to bigger or heavier jobs. The point of the NPU path is different:

  • a cleaner lane for lighter always-on inference
  • an efficient path for smaller local agent tasks
  • a home for experiments before promoting workloads elsewhere
  • a way to reserve the heavier GPU lanes for work that actually needs them

In other words: not “run everything on the NPU.”

More like:

Use the right silicon for the right class of work.

That’s much less exciting than claiming the NPU is going to replace a rack of GPUs, but it is also much more true.

What we set up

The core idea was to create a dedicated NPU-focused runtime path on the Strix Halo system using a separate container for experimentation.

Here’s what is now in place:

  • the host already exposes the NPU device as /dev/accel/accel0
  • the amdxdna driver is loading on the host
  • IOMMU passthrough settings were already in place
  • a dedicated container was created for the NPU lane
  • the NPU device is now openable from inside that container
  • AMD XRT NPU userspace is installed inside the container
  • the NPU is visible and identified correctly by tooling

That last point matters. “The device exists” is not the same as “the stack is real.”

We wanted the setup to move beyond theoretical readiness and into actual evidence. And we got that.

Inside the container, the tooling can now see the NPU as a proper device rather than a rumour.

The annoying but important fix

One of the most practical lessons from this work was that the NPU passthrough path only became usable after changing the container from unprivileged to privileged.

That was the unlock.

Before that, the device was present in a technical sense but not usefully accessible. After the change, the accel device could actually be opened from inside the container, which turned the project from “we are staring at a hardware listing” into “we are finally moving.”

That kind of thing is very lab-coded: the difference between blocked and working is not always glamorous. Sometimes it’s just one uncomfortable systems decision and a lot less idealism.

What’s already working

This is the good part: a surprising amount of the stack is already in place.

The container now has:

  • Ubuntu 24.04 base packages
  • AMD XRT NPU userspace
  • the core plumbing needed to talk to the NPU properly

And more importantly, the validation tooling can already see the device details we care about, including:

  • the NPU identity
  • the XRT version
  • the currently loaded firmware version

That means the device discovery path is not the blocker anymore.

The container exists. The device is visible. The userspace tooling is installed. The runtime container can start.

Those are real milestones, and they matter.

Where it still breaks

The remaining blocker is not “we haven’t set anything up yet.”

It’s narrower and much more annoying:

the host-side kernel / driver / firmware combination is not yet on the validated lane required for the FastFlowLM-style Linux NPU runtime path we want.

That’s the whole problem now.

The current runtime validation fails because:

  • the loaded firmware is too old
  • the effective host driver/kernel lane does not match what the runtime expects

That’s actually a good kind of failure.

Not fun, obviously. But good in the sense that we are now past the vague part. We’re no longer guessing whether the device exists, whether passthrough is viable, or whether the userspace path is fundamentally broken.

We now know the remaining issue lives squarely in host compatibility.

That’s progress.

What the runtime expects

The NPU runtime path we’re targeting expects a more specific host environment than “Linux, probably.”

In particular, it wants:

  • a validated kernel lane
  • a compatible amdxdna path
  • newer NPU firmware than the host is currently using

This is host-level work, not container-level work.

That distinction matters, because once you reach this stage, no amount of fiddling inside the container will magically solve a mismatched host stack. At that point you have to stop decorating the guest environment and go fix the actual foundation.

Which is rude, but fair.

What we want to use it for

Assuming the host compatibility piece lands cleanly, the NPU lane should become a very useful part of the lab.

The intended uses are:

1. Lighter always-on inference

Not every local task needs the full drama of a bigger GPU lane. The NPU path is a much better fit for smaller jobs that should just stay available quietly in the background.

2. Agent helper work

Some agent tasks are more about responsiveness and efficiency than brute-force reasoning. That’s exactly where an NPU-backed path gets interesting.

Think:

  • lightweight summarisation
  • small operational helpers
  • bounded background tasks
  • lower-cost local support jobs

3. Experimental model serving

The NPU lane is also a lab lane. That means trying new local serving patterns, checking what actually works in practice, and figuring out which workloads deserve promotion into the more serious parts of the fleet.

4. Better workload separation

This is the broader architectural point.

A healthy lab doesn’t just accumulate compute. It separates workloads intentionally.

The NPU lane gives us a better place to put the work that shouldn’t be contending with bigger local inference jobs elsewhere.

That’s not flashy. It is, however, how you stop your infrastructure from becoming a confused pile of expensive enthusiasm.

The current state, honestly

So where are we right now?

In honest terms:

  • device access: working
  • container path: working
  • userspace tooling: working
  • runtime container startup: working
  • real NPU inference: not yet
  • remaining blocker: host kernel/driver/firmware alignment

That’s not finished, but it’s very far from nothing.

The hardest part of projects like this is often getting from “concept” to “we can prove the stack is real.” We’ve already crossed that line. The rest is now a focused systems problem rather than a vague experiment.

And focused systems problems are, frankly, much easier to respect.

Final thought

The NPU story in local AI is still early.

That’s part of why it’s interesting.

Right now, a lot of this work sits in the awkward middle ground between official promise and actual operational reality. Which means if you want a real NPU lane in a Linux lab, you still have to do some digging, some verification, and a little bit of ritual combat with the host stack.

But that’s also what makes it fun.

The endgame here isn’t a demo. It’s a useful new lane in the lab: efficient, local, and purpose-built for the class of work that doesn’t need to wake the bigger machines every time it wants to be helpful.

That’s the kind of infrastructure improvement we like best.

Not louder.

Just smarter.