CUDA 13.2.0有bug,必须升级到13.2.2。
升级cmake:
sudo apt remove cmake
sudo snap install cmake --classic
hash -r
cmake --version
cmake version 4.4.4
https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF
https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF
https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF
export HF_HUB_OFFLINE=1
source /data/D/python312_venv/bin/activate
cd /data/E/Strata-0.1.40.1
IQ3_S:
./setup.sh --gguf-dir /data/H/LLM/Qwen3.8-Flash-Next-GSQ-RCO-GGUF_ISTA-DASLab \
--model IQ3_S\
--family qwen \
--data-dir /data/E/Strata-data \
--context 262144 \
--kv int8 \
--experimental-speed-projection off \
--vision yes \
--port 8099
UD-Q4_K_XL:
./setup.sh --gguf-dir /data/E/Qwen3.8-Flash-Next-GGUF-UD-Q4_K_XL_unsloth \
--model UD-Q4_K_XL \
--family unsloth \
--data-dir /data/E/Strata-data \
--context 262144 \
--kv int8 \
--experimental-speed-projection off \
--vision yes \
--port 8099
第一次启动时需要calibrate,添加vision支持:
nano strata-unsloth-ud-q4_k_xl.json
"args": [
...
"--resident-budget-gib",
"72",
...
]
"host": "0.0.0.0",
"fit_max_tokens": true,
"reasoning_effort": "medium",
"reasoning_budget_tokens": 16000,
"vision": {
...
"gpu": false,
...
}
上面的 "reasoning_budget_tokens": 16000 用于限制思考长度,防止超长思考,很重要
后续使用这条命令启动:
.venv/bin/python -m serve.server --engine strata --config strata-swift-iq3_xxs.json --port 8099
.venv/bin/python -m serve.server --engine strata --config strata-iq3_xxs.json --port 8099
.venv/bin/python -m serve.server --engine strata --config strata-iq3_s.json --port 8099
.venv/bin/python -m serve.server --engine strata --config strata-unsloth-ud-iq4_xs.json --port 8099
.venv/bin/python -m serve.server --engine strata --config strata-unsloth-ud-q4_k_xl.json --port 8099
cc switch参数:
Body覆盖:
{
"max_tokens": 32000,
"output_config": {
"effort": "medium"
}
}
~/.claude/settings.json
"model": "deepseek-v4-pro-2[1M]",
"modelSettings": {
"deepseek-v4-pro-2": {
"effortLevel": "medium"
}
}
.venv/bin/python -m serve.server --engine strata --config strata-unsloth-ud-q4_k_xl.json --port 8099 --api-monitor
打开网页:
http://127.0.0.1:8099/api-monitor
在agent中发消息检查log中的参数已否以更新,或者手动测试:
curl http://127.0.0.1:8099/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer xxx" \
-d '{
"model":"Qwen3.8-Flash-Next",
"messages":[
{
"role":"user",
"content":"介绍一下中国"
}
],
"max_tokens": 32000,
"output_config": {
"effort": "medium"
}
}'
swift-iq3_xxs prefill:5400 tok/s, decode:170 tok/s,中间会话会自己中断
iq3_xxs prefill:5400 tok/s, decode:160 tok/s,中间会话会自己中断
iq3_s prefill:5400 tok/s, decode:150 tok/s
unsloth-ud-iq4_xs prefill:4000 tok/s, decode:100 tok/s,中间会话会自己中断
unsloth-ud-q4_k_xl prefill:2000 tok/s, decode:90 tok/s,使用strata-nvfp4项目时中间会话会自己中断