Lemonade + Qwen3.5: The Easiest Way to Run a 35B MoE on AMD Hardware

Last week I was building llama.cpp from source, fighting ROCm CMake flags, and cursing AMD's driver documentation. This week I'm running a 35 billion parameter model at 40+ tok/s with a single docker compose up. The difference? Lemonade.

Here's why Lemonade + Qwen3.5-35B-A3B is the combo I didn't know I needed.

What is Lemonade?

Lemonade is an OpenAI-compatible inference server built specifically for AMD hardware. It ships pre-built llama.cpp binaries for ROCm — including dedicated builds for gfx1151 (Strix Halo/Ryzen AI Max) — and handles all the driver complexity for you.

No source builds. No CMake. No AMDGPU_TARGETS env vars. Just Docker.

services:
  lemonade:
    image: ghcr.io/lemonade-sdk/lemonade-server:latest
    environment:
      - LEMONADE_LLAMACPP=rocm
      - LEMONADE_CTX_SIZE=131072
      - LEMONADE_LLAMACPP_ARGS=--flash-attn on --cache-type-k q8_0 --cache-type-v q8_0
    ports:
      - "8000:8000"
    devices:
      - /dev/kfd:/dev/kfd
      - /dev/dri:/dev/dri

That's it. Point it at any GGUF model and you're done.

The Model: Qwen3.5-35B-A3B

Qwen3.5-35B-A3B is a Mixture-of-Experts model with a twist: 35 billion total parameters, but only 3 billion active per token. That means you get the reasoning quality of a large model at the inference cost of a small one.

On my Strix Halo (AMD Ryzen AI Max+ 395, 96GB unified memory):

Metric	Value
Model size	19.6GB (Q4_K_XL GGUF)
Prompt processing	~222 tok/s
Generation	~42 tok/s (b1204, up from ~40 on b1203)
Context	128K tokens

42 tok/s for a 35B model. On an APU. In Docker.

For comparison, my AMD RX 6900XT (16GB VRAM) maxes out at 14B parameters before hitting VRAM limits. The Strix Halo's unified memory architecture means I can load models that would never fit on a discrete GPU.

Why MoE Changes Everything for Local AI

Traditional dense models scale linearly — a 35B model is roughly 2.5x slower than a 14B model. MoE breaks that relationship. With only 3B active parameters per forward pass, Qwen3.5-35B-A3B generates tokens at a speed closer to a 7B model while drawing on the knowledge of something much larger.

The practical result: it handles complex multi-step reasoning, long context, and tool calling reliably — the things that matter for agent workloads — without the speed penalty.

I'm running it as the backbone for Bill, my local coding agent. Bill handles GitHub PRs, writes feature code, and pushes branches — all from a model that fits comfortably in my machine's memory alongside two other loaded models.

One Gotcha: Thinking Mode

Qwen3.5 has a thinking mode that's great for deep reasoning but annoying for agentic tasks. When active, the model dumps its entire reasoning chain into reasoning_content and leaves content empty — which breaks most OpenAI-compatible clients.

Fix: prefix your prompts with /no_think.

/no_think List the files changed in this PR.

This tells the model to skip the internal monologue and just answer. For agent tasks where you want fast, deterministic tool calls, this is essential.

The Setup (Full Docker Compose)

services:
  lemonade:
    image: ghcr.io/lemonade-sdk/lemonade-server:latest
    container_name: lemonade-server
    ports:
      - "8000:8000"
    volumes:
      - lemonade-cache:/root/.cache/huggingface
      - lemonade-models:/opt/lemonade/llama
      - lemonade-bin:/root/.cache/lemonade
      - /usr/lib/x86_64-linux-gnu/libatomic.so.1:/usr/lib/x86_64-linux-gnu/libatomic.so.1:ro
    devices:
      - /dev/kfd:/dev/kfd
      - /dev/dri:/dev/dri
    group_add:
      - "993"  # render group GID — check yours with: getent group render
      - "44"   # video group GID — check yours with: getent group video
    environment:
      - LEMONADE_LLAMACPP=rocm
      - LEMONADE_MAX_LOADED_MODELS=2
      - LEMONADE_CTX_SIZE=131072
      - "LEMONADE_LLAMACPP_ARGS=--flash-attn on --cache-type-k q8_0 --cache-type-v q8_0"
    restart: unless-stopped

volumes:
  lemonade-cache:
  lemonade-models:
  lemonade-bin:

Then pull the model via the API:

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen3.5-35B-A3B-GGUF", "messages": [{"role": "user", "content": "/no_think Hello"}], "max_tokens": 50}'

Lemonade auto-downloads the model on first request. No ollama pull, no manual GGUF downloads.

Is Flash Attention Actually Working?

Short answer: probably not yet, at least not the accelerated ROCWMMA variant.

I dug into this today. Lemonade's prebuilt ROCm binaries for gfx1151 don't include GGML_HIP_ROCWMMA_FATTN compiled in — you can verify with strings libggml-hip.so | grep rocwmma (empty on b1204). The --flash-attn on flag falls through to standard attention on current builds.

This is likely intentional — there are known segfault issues with ROCWMMA on gfx1151. Watch the llamacpp-rocm releases for when this changes. When it does, the performance headroom could be significant.

Bottom Line

If you have AMD hardware and want to run local LLMs without the usual pain:

Install Docker with ROCm device access
Drop in the docker-compose above
Start sending requests

Lemonade handles everything else. And Qwen3.5-35B-A3B is the best model I've found for agentic workloads on a single consumer GPU — fast enough to be practical, smart enough to be useful.

The era of "you need an H100 for this" is genuinely over.