Qwen3.6-27B-i1-IQ4_KS-GGUF

This repository contains GGUF format weights for the Qwen3.6-27B model, quantized using the ik_llama.cpp project.

This model was specifically created to run on consumer GPUs with 16GB VRAM. By utilizing q4_0 KV cache quantization, it allows pushing the context length up to 105k tokens.

⚠️ Note: This model is designed exclusively for nVidia GPUs and is based on the advanced KS quants developed by ikawrakow from the ik_llama.cpp repository.

---

Quantization Details & Imatrix File

• Quantization Base: KS Quants (ik_llama.cpp).
• Imatrix File Used: Provided by mradermacher.
• Other Tested Imatrix Files:
  • bartowski – yielded significantly worse results.
  • ubergarm – yielded comparable results.
  • If you find or generate a better Imatrix file, please let me know in the Community tab!

I have also included the script I used for the quantization process in this repository. If you have any concerns regarding malware or security, you can download the official BF16 weights from Unsloth and quantize the model yourself using the provided script.

secret recipe

# 1. Base Tensors & Embeddings
^output\.weight$ -> iq5_k
^output_norm\.weight$ -> f32
^token_embd\.weight$ -> iq4_ks

# 2. Exceptions (High precision for the first block's attention gate)
^blk\.0\.attn_gate\.weight$ -> iq6_k

# 3. Feed-Forward Networks (FFN)
# Aggressive compression (iq4_kss) for middle blocks 5-51
^blk\.([5-9]|[1-4][0-9]|50|51)\.ffn_(down|up)\.weight$ -> iq4_kss
.*ffn_(down|up)\.weight$ -> iq4_ks
.*ffn_gate\.weight$ -> iq4_ks

# 4. Attention Layers
.*attn_v\.weight$ -> iq5_ks
.*attn_(qkv|q|k|output)\.weight$ -> iq4_kss
.*attn_gate\.weight$ -> iq4_ks

# 5. Mamba / SSM Layers
.*ssm_out\.weight$ -> iq4_kss
.*ssm_(alpha|beta)\.weight$ -> iq4_ks
.*ssm_conv1d\.weight$ -> f32
.*ssm_a$ -> f32
.*ssm_dt\.bias$ -> f32

# 6. All Normalization Layers (Kept in f32 for stability)
.*norm\.weight$ -> f32

Benchmark Results & Real-World Impressions

The model was heavily tested in daily production workflows for several days. It runs much faster (1.5x-1.75x) and more reliably than the previous iteration—completely eliminating the issue of "blank outputs", while the search-replace functionality works flawlessly.

• Qwen Benchmark: Successfully passed the performance evaluations on qwen3-6-27b-benchmark.vercel.app.
• Needle In A Haystack: Successfully evaluated with satisfying results across the full 100k context window.
• Comparison: In direct testing, this model performs slightly better than my previous variant: Qwen3.6-27B-i1-IQ4_XS-GGUF.

Perplexity (PPL) Testing

Perplexity evaluations were conducted focusing exclusively on the KV Cache quantization setup (q4_0), as this is the primary target use case:

wget [https://www.gutenberg.org/files/2600/2600-0.txt](https://www.gutenberg.org/files/2600/2600-0.txt) -O pg19.txt \

./llama-perplexity -m Qwen3.6-27B.i1-IQ4_KS-attn_qkv-IQ4_KSS.gguf -f pg19.txt -c 65536 --chunks 32 -ngl 99 -khad -vhad -ctk q4_0 -ctv q4_0 -fa 1 -b 512 -ub 512

Test Log Output:

perplexity: calculating perplexity over 12 chunks, n_ctx=65536, batch_size=512, n_seq=1
perplexity: 71.10 seconds per pass - ETA 14.22 minutes
[1]6.6897,[2]7.0032,[3]7.1989,[4]7.3327,[5]7.4816,[6]7.3770,[7]7.4325,[8]7.4378,[9]7.4754,[10]7.5192,[11]7.5669,[12]7.4040,

Final estimate: PPL over 12 chunks for n_ctx=65536 = 7.4040 +/- 0.02773

---

Production Deployment Configuration (ik_llama.cpp)

Below is the verified and stable execution configuration used for llama-server. It utilizes the corrected chat template (chat.template) by ex-arman68, sourced from froggeric/Qwen-Fixed-Chat-Templates.

llama-server \
        -m "$MODEL_PATH" \
        -a Qwen3.6-27B \
        --ctx-size 105000 \
        --chat-template-file chat_template.jinja \
        --n-gpu-layers 99 \
        --cache-type-k q4_0 \
        --cache-type-v q4_0 \
        --batch-size 512 \
        --ubatch-size 256 \
        --flash-attn on \
        --no-mmap \
        --host 0.0.0.0 \
        --port 8081 \
        --reasoning on \
        --reasoning-format deepseek \
        -t 8 \
        --parallel 1 \
        -khad \
        -vhad \
        --chat-template-kwargs '{"preserve_thinking": true}' \
        --defrag-thold 0.3 \
        --jinja \
        --cont-batching \
        --temp 0.15 \
        --top-k 1 \
        --min-p 0.1 \
        --keep -1 \
        --repeat-last-n 512 \
        --repeat-penalty 1.05