Strata

CUDA 13.2.0有bug,必须升级到13.2.2。

升级cmake:
sudo apt remove cmake
sudo snap install cmake --classic
hash -r
cmake --version
cmake version 4.4.4

https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF
https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF
https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF

export HF_HUB_OFFLINE=1
source /data/D/python312_venv/bin/activate
cd /data/E/Strata-0.1.40.1
IQ3_S:
./setup.sh --gguf-dir /data/H/LLM/Qwen3.8-Flash-Next-GSQ-RCO-GGUF_ISTA-DASLab \
  --model IQ3_S\
  --family qwen \
  --data-dir /data/E/Strata-data \
  --context 262144 \
  --kv int8 \
  --experimental-speed-projection off \
  --vision yes \
  --port 8099
UD-Q4_K_XL:
./setup.sh --gguf-dir /data/E/Qwen3.8-Flash-Next-GGUF-UD-Q4_K_XL_unsloth \
  --model UD-Q4_K_XL \
  --family unsloth \
  --data-dir /data/E/Strata-data \
  --context 262144 \
  --kv int8 \
  --experimental-speed-projection off \
  --vision yes \
  --port 8099

第一次启动时需要calibrate,添加vision支持:
nano strata-unsloth-ud-q4_k_xl.json
 "args": [
  ...
  "--resident-budget-gib",
  "72",
  ...
 ]
 "host": "0.0.0.0",
 "fit_max_tokens": true,
 "reasoning_effort": "medium",
 "reasoning_budget_tokens": 16000,
 "vision": {
  ...
  "gpu": false,
  ...
 }
上面的 "reasoning_budget_tokens": 16000 用于限制思考长度,防止超长思考,很重要
后续使用这条命令启动:
.venv/bin/python -m serve.server --engine strata --config strata-swift-iq3_xxs.json --port 8099
.venv/bin/python -m serve.server --engine strata --config strata-iq3_xxs.json --port 8099
.venv/bin/python -m serve.server --engine strata --config strata-iq3_s.json --port 8099
.venv/bin/python -m serve.server --engine strata --config strata-unsloth-ud-iq4_xs.json --port 8099
.venv/bin/python -m serve.server --engine strata --config strata-unsloth-ud-q4_k_xl.json --port 8099

cc switch参数:
Body覆盖:
{
  "max_tokens": 32000,
  "output_config": {
    "effort": "medium"
  }
}
~/.claude/settings.json
  "model": "deepseek-v4-pro-2[1M]",
  "modelSettings": {
    "deepseek-v4-pro-2": {
      "effortLevel": "medium"
    }
  }

.venv/bin/python -m serve.server --engine strata --config strata-unsloth-ud-q4_k_xl.json  --port 8099 --api-monitor
打开网页:
http://127.0.0.1:8099/api-monitor
在agent中发消息检查log中的参数已否以更新,或者手动测试:
curl http://127.0.0.1:8099/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer xxx" \
-d '{
"model":"Qwen3.8-Flash-Next",
"messages":[
 {
  "role":"user",
  "content":"介绍一下中国"
 }
],
"max_tokens": 32000,
"output_config": {
  "effort": "medium"
}
}'

swift-iq3_xxs      prefill:5400 tok/s, decode:170 tok/s,中间会话会自己中断
iq3_xxs            prefill:5400 tok/s, decode:160 tok/s,中间会话会自己中断
iq3_s              prefill:5400 tok/s, decode:150 tok/s
unsloth-ud-iq4_xs  prefill:4000 tok/s, decode:100 tok/s,中间会话会自己中断
unsloth-ud-q4_k_xl prefill:2000 tok/s, decode:90 tok/s,使用strata-nvfp4项目时中间会话会自己中断