Qwen3.8-27B-abliterated-AWQ-MTP

W4A16 AWQ quantization of the abliterated (refusal-removed) Qwen3.8-27B, with the MTP speculative-decoding module preserved.

As of 2026-08-15 this appears to be the only AWQ / W4A16 build of an abliterated Qwen3.8-27B. Other abliterated derivatives are GGUF, MLX, NVFP4, MXFP4, BF16 or INT8 — none of which give a fast 4-bit weight-only path under vLLM on Ampere. It is also one of only two abliterated builds of any format that preserve the MTP module.

To be clear about what this is not: AWQ builds of the base (non-abliterated) Qwen3.8-27B do exist and several predate this one. The gap filled here is the abliterated + W4A16 + MTP combination, not AWQ of this model in general.

• Base: JonathanColetti/Qwen3.8-27B-Uncensored (bf16 abliteration, the only abliterated 3.8 that kept the MTP module)
• Upstream: Qwen/Qwen3.8-27B (Apache-2.0)
• Method: llm-compressor 0.13.0, AWQ + W4A16_ASYM, group size 128
• Calibration: 128 samples × 1024 tokens from HuggingFaceH4/ultrachat_200k

Architecture notes

qwen3_5 is a hybrid architecture and a VLM:

Layers	64 — 48 linear_attention (Qwen3_5GatedDeltaNet) + 16 full_attention, every 4th
Vision	27-layer tower (left in bf16)
Context	262,144
MTP	mtp_num_hidden_layers: 1, kept unquantized in model-mtp.safetensors

Left in bf16 (not quantized): the whole vision tower (model.visual.*), lm_head, mtp.*, and in_proj_a / in_proj_b — the 48-wide per-head scalars driving the delta rule, where 4-bit is destructive and saves nothing. conv1d is not an nn.Linear so AWQ never touches it.

v_proj → o_proj smoothing is deliberately skipped: GQA means v_proj emits 1024 while o_proj takes 6144 (q_proj is doubled to 12288 by attn_output_gate), so the shapes do not line up for a channel-wise scale.

Measured on 2× RTX 3090 (TP=2, vLLM)

All figures are single-stream (c=1), 600 output tokens, thinking off, measured over an HTTP round-trip with the same harness — so they are directly comparable to each other.

	This model (W4A16)	Official FP8
c=1 throughput	84 t/s median (best 93)	57 t/s
Weights / GPU	9.37 GiB	15.11 GiB
KV cache	560,900 tokens	266,537
Max concurrency @ 262,144	2.14×	1.02×
MTP acceptance length	2.96 / 3 (~60%)	2.80

What MTP is actually worth here

Measured on this model, same harness, only --speculative-config changed:

MTP state	median	vs no-MTP
Disabled	57.9 t/s	1.00×
Enabled (working)	84 t/s	1.45×
Enabled but weights not loading	33 t/s	0.57×

That third row is the failure in trap 3 below, and it is worth internalising: a broken MTP is slower than no MTP at all, because you pay the full drafting cost (~98 t/s of drafted tokens) and accept none of it. It presents as a mysterious throughput regression, not as a loading error. Disabling MTP entirely also buys you a larger KV cache (677,958 tokens, 2.59× at 262,144) if context depth matters more to you than latency.

Verified after quantization: 250,060-token needle retrieval; tool calling (correct JSON arguments); the abliteration survived; and vision still works — the model correctly described shapes and colours and read embedded text from a test image. The vision tower is left in bf16, so image quality should be unchanged from the source.

Not measured: no perplexity or benchmark comparison against the bf16 source was run. W4 does cost some accuracy relative to bf16/FP8; this card does not quantify it. Behaviour was verified, quality regression was not.

Running it

Needs a vLLM recent enough to register Qwen3_5ForConditionalGeneration and the Qwen3_5MTP proposer. This was validated on vllm/vllm-openai:nightly (v0.20.2rc1.dev129, compressed-tensors 0.15.0.1 in-image — it reads the 0.18.0 config fine).

Minimal

vllm serve twolven/Qwen3.8-27B-abliterated-AWQ-MTP \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

The full production config (2× RTX 3090, single user)

This is what produced the numbers in the table above.

vllm serve /path/to/Qwen3.8-27B-abliterated-AWQ-MTP \
  --served-model-name qwen3.8-27b-abliterated \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 4096 \
  --kv-cache-dtype fp8 \
  --performance-mode interactivity \
  --mm-encoder-tp-mode data \
  --disable-custom-all-reduce \
  --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": true, "reasoning_effort": "medium"}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode": "PIECEWISE"}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

With NCCL_P2P_DISABLE=1 in the environment.

Why these particular flags — each one is load-bearing on this hardware:

Flag	Reason
--gpu-memory-utilization 0.95	0.98 overshoots if anything else holds VRAM (a desktop session is enough). At 0.95 the full 262K context still fits at 2.14× concurrency.
--kv-cache-dtype fp8	The hybrid layout is cheap on KV — 262K needs only ~4.6 GiB — but fp8 is what buys the 2.14×.
--compilation-config PIECEWISE	FULL cudagraph replay segfaults with MTP (vllm#40756). PIECEWISE is the same c=1 speed and keeps MTP.
NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce	Dual 3090 over PCIe without NVLink.
--performance-mode interactivity	Favours single-stream latency. Costs aggregate throughput at high concurrency — drop it if you serve many users.
--tool-call-parser qwen3_coder	See the caveat below.

⚠️ Pin reasoning_effort — the default will truncate your answers

The chat template accepts reasoning_effort of xhigh / medium / low, and defaults to xhigh:

{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}

xhigh is pathological for general serving. Measured over 4 hard reasoning prompts with a 16,384-token budget:

reasoning_effort	mean thinking	mean answer	thinking share	requests that produced an answer
xhigh (default)	11,888 tok	792 tok	93.8%	2 of 4
medium	2,092 tok	4,548 tok	31.5%	4 of 4
low	3,314 tok	3,251 tok	50.5%	4 of 4

At xhigh, half the requests spent the entire budget thinking and returned an empty answer (finish_reason: length). Note low is counterintuitively worse than medium — it thinks ~58% more — so medium is the sweet spot, not merely a midpoint.

This is not a repetition loop and not a quantization artifact. A truncated xhigh trace analysed at 675/676 unique sentences, with only 1.7% of positions inside any repeated 8-gram and 0% repetition in the final 400 words — the model is generating novel reasoning right to the cutoff. Re-running an identical truncated prompt with a 40,000-token budget terminated cleanly (stop, 11,756 thinking + 2,913 answer). The failure mode is variance: xhigh's thinking length straddles typical budgets, so the same prompt sometimes finishes at 14k and sometimes blows past 16k and returns nothing.

Pin it server-side:

--default-chat-template-kwargs '{"enable_thinking": true, "reasoning_effort": "medium"}'

Clients can still opt up per request with chat_template_kwargs: {"reasoning_effort": "xhigh"} — just give them a budget of 40k+ if they do. Thinking can be turned off entirely with {"enable_thinking": false}.

Tool calling caveat

If you enable --tool-call-parser qwen3_coder, vLLM's parser has a streaming bug where the <tool_call> tag is emitted as prose and subsequent SSE chunks go silent. We run a small patch that buffers <tool_call> tokens until <function= appears within 64 characters. Patch and details: club-3090 → models/qwen3.6-27b/vllm/patches/local/. Non-streaming tool calls are unaffected.

Sanity checks after it boots

# should report ~560,900 tokens and 2.14x at 262,144
grep -E "GPU KV cache size|Maximum concurrency" <container logs>

# MTP must actually load — if you see "not found in params_dict" the drafter is
# running on random weights and acceptance will be 0% (see trap 3 below)
grep "Detected MTP model" <container logs>

# after some traffic, acceptance length should be ~2.9 of 3
grep "SpecDecoding metrics" <container logs>

The compose files, the quantization script, and the deployment notes this model came out of live in club-3090.

If you reproduce this — four things that will bite you

1. AWQ mappings must be generated per layer. match_modules_set() accumulates matches until every pattern fires, so a generic re:.*input_layernorm$ → q/k/v mapping piles up four smooth layers across three consecutive linear_attention blocks (which have no q_proj) and aborts with "AWQ needs to match a single smoothlayer". Emit one mapping per layer index — 192 over 64 layers.
2. transformers erases Qwen3_5GatedDeltaNet.forward's signature. Its force_accelerate_hooks decorator is def wrapped(self, *args, **kwargs) with no functools.wraps, so inspect.signature() reports (*args, **kwargs); AWQ's arg cache then stores {'args': …} and replay dies with "missing 1 required positional argument: 'hidden_states'". Recover the inner function from the wrapper closure and re-attach __signature__.
3. MTP needs both an index entry and an ignore rule. save_pretrained writes a single model.safetensors with no index, so a copied-in MTP file is never opened, the drafter loads random weights, and you get 0% draft acceptance — which presents as a speed bug (~33 t/s) rather than a load bug. Write a model.safetensors.index.json covering both files, and add re:.*mtp.* to quantization_config.ignore or vLLM builds MTP as quantized and demands weight_packed. After changing that, clear the torch.compile cache or a stale AOT graph throws KeyError: 'weight_packed'.
4. AWQ will OOM 24 GB at 256×2048 calibration. Use offload_device="cpu" on AWQModifier and drop to 128×1024.

Author

Quantized by Todd Wolven - Lead AI Software Developer and open-source GenAI engineer. GitHub | Hugging Face

License & attribution

Apache-2.0, inherited from Qwen/Qwen3.8-27B. The abliteration is JonathanColetti's; this repo contributes only the quantization.

This model is uncensored. Refusal behaviour has been removed by the upstream abliteration and that property survives quantization. You are responsible for how you use it.