安装依赖环境，pytorch、cuda-toolkit等等
test:
nvidia-smi
/usr/local/cuda/bin/nvcc -V
pip list
安装FlashAttention:
https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3.post1/flash_attn-2.8.3.post1+cu12torch2.8cxx11abiFALSE-cp312-cp312-linux_x86_64.whl
必须安装2.8版的pytorch:
https://download-r2.pytorch.org/whl/cu128/torch-2.8.0%2Bcu128-cp312-cp312-manylinux_2_28_x86_64.whl
pip install nvidia-cutlass-dsl==4.2.1
或者从源码编译：
https://github.com/Dao-AILab/flash-attention
https://github.com/NVIDIA/cutlass/tree/7127592069c2fe01b041e174ba4345ef9b279671
#export MAX_JOBS=2
export NVCC_THREADS=4
export FLASH_ATTENTION_FORCE_BUILD="TRUE"
python -c "import torch; print(torch.cuda.get_device_capability())"
#export FLASH_ATTN_CUDA_ARCHS="120"
python setup.py clean
rm -rf build/ dist/ *.egg-info
#python setup.py install
#编译耗时8小时
pip install . --no-build-isolation
如果报错：
NotImplementedError: Support for egg-based install has been removed.
解决方法：
pip uninstall setuptools
pip install setuptools
git clone https://github.com/karpathy/nanochat
cd nanochat
快速运行手册：
https://github.com/karpathy/nanochat/blob/master/runs/speedrun.sh
export HF_ENDPOINT=https://hf-mirror.com
export NANOCHAT_BASE_DIR=/data/D/nanochat_base
mkdir -p $NANOCHAT_BASE_DIR
下载初始20亿字符数据集(约800MB):
pip install pyarrow
python -m nanochat.dataset -n 8
或者手动下载：
https://huggingface.co/datasets/karpathy/climbmix-400b-shuffle
ClimbMix是一个由NVIDIA发布的高质量、紧凑且强大的4000亿token预训练数据集，通过高效预训练在等量的token预算下实现优越性能。
训练tokenizer:
pip install rustbpe tiktoken
#默认20亿字符
python -m scripts.tok_train --max-chars=2_000_000_000
评估tokenizer:
python -m scripts.tok_eval
下载更多训练数据(约22GB):
python -m nanochat.dataset -n 240
耗时较长。
下载评估数据(约24.9MB):
curl -L -o eval_bundle.zip https://karpathy-public.s3.us-west-2.amazonaws.com/eval_bundle.zip
unzip -q eval_bundle.zip
mv eval_bundle $NANOCHAT_BASE_DIR/
rm eval_bundle.zip
至此环境和数据均已准备完成，开始训练模型。
训练模型:
#depth=12
python -m scripts.base_train --depth=12 \
--model-tag="d12" \
--core-metric-every=999999 \
--sample-every=-1 \
--save-every=-1 \
--device-batch-size=32
Vocab size: 32,768
Model config:
{
"sequence_len": 2048,
"vocab_size": 32768,
"n_layer": 12,
"n_head": 6,
"n_kv_head": 6,
"n_embd": 768,
"window_pattern": "SSSL"
}
Parameter counts:
wte : 25,165,824
value_embeds : 150,994,944
lm_head : 25,165,824
transformer_matrices : 84,935,088
scalars : 50
total : 286,261,730
Estimated FLOPs per token: 7.596959e+08
Auto-computed optimal batch size: 524,288 tokens
Scaling the LR for the AdamW parameters ∝1/√(768/768) = 1.000000
Calculated number of iterations from target data:param ratio: 2,520
Total number of training tokens: 1,321,205,760
Tokens : Scaling params ratio: 12.00
Total training FLOPs estimate: 1.003715e+18
Tokens / micro-batch / rank: 32 x 2048 = 65,536
Tokens / micro-batch: 65,536
Total batch size 524,288 => gradient accumulation steps: 8
Step 00000 | Validation bpb: 3.169591
#depth=24
python -m scripts.base_train --depth=24 \
--target-param-data-ratio=8 \
--model-tag="d24" \
--core-metric-every=999999 \
--sample-every=-1 \
--save-every=200 \
--device-batch-size=4 \
--fp8
Vocab size: 32,768
Model config:
{
"sequence_len": 2048,
"vocab_size": 32768,
"n_layer": 24,
"n_head": 12,
"n_kv_head": 12,
"n_embd": 1536,
"window_pattern": "SSSL"
}
✓ FP8 training enabled (tensorwise scaling) - converted 145/158 linear layers, skipped 13 (too small)
Parameter counts:
wte : 50,331,648
value_embeds : 603,979,776
lm_head : 50,331,648
transformer_matrices : 679,478,976
scalars : 74
total : 1,384,122,122
Estimated FLOPs per token: 4.775225e+09
Auto-computed optimal batch size: 1,048,576 tokens
Scaling LRs by 1.4142 for batch size 1,048,576 (reference: 524,288)
Scaling weight decay from 0.280000 to 0.059738 for depth 24
Scaling the LR for the AdamW parameters ∝1/√(1536/768) = 0.707107
Calculated number of iterations from target data:param ratio: 5,568
Total number of training tokens: 5,838,471,168
Tokens : Scaling params ratio: 8.00
Total training FLOPs estimate: 2.788002e+19
Tokens / micro-batch / rank: 4 x 2048 = 8,192
Tokens / micro-batch: 8,192
Total batch size 1,048,576 => gradient accumulation steps: 128
Step 00000 | Validation bpb: 3.169850
在5090D 32G显存+96GB DDR5内存机器上，depth=12，训练耗时185分钟，开启FlashAttention耗时缩短为102分钟。
模型权重文件model_002520.pt大小为792.8MB。
depth=24，开启FlashAttention耗时为1831分钟。
If display error:
/tmp/tmp4oxwapmp/cuda_utils.c:7:10: fatal error: Python.h: 没有那个文件或目录
7 | #include
Fix:
export C_INCLUDE_PATH=$C_INCLUDE_PATH:/home/li/Python-3.12.12:/home/li/Python-3.12.12/Include
export CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:/home/li/Python-3.12.12:/home/li/Python-3.12.12/Include
或者:
ln -s '/home/li/Python-3.12.12/pyconfig.h' '/home/li/Python-3.12.12/Include/pyconfig.h'
sudo ln -s '/home/li/Python-3.12.12/Include' /usr/local/include/python3.12
评估模型(CORE metric, BPB on train/val):
python -m scripts.base_eval --device-batch-size=16
耗时20分钟。
输出:
BPB Evaluation:
train bpb: 0.848951
val bpb: 0.848383
...
CORE metric: 0.1579
BPB(Bits Per Byte)的字面意思是每个字节消耗的比特数。
在信息论中，它衡量的是大模型对文本数据的压缩效率。根据香农的信息论，一个完美的语言模型如果能100%预测未来的文本，那么它压缩该文本所需的比特数就会极低。
在LLM训练中，BPB的数学本质是以2为底的交叉熵损失（Cross Entropy Loss），但它是针对字节（Byte）级别进行归一化的，而不是针对Token。
为什么不用Loss，而要看BPB？为了解决Token长度的“作弊”问题。
大模型在训练时，交叉熵损失（Cross-Entropy Loss）的计算是基于Token的平均值，分子是总Loss，分母是Token的总数量。
不同的模型使用的Tokenizer（分词器）是不同的。比如同样一句话，LLaMA的分词器可能会切成10个Token，而GPT-4的分词器可能只切成6个Token。
这样一来，两者的平均Token Loss是无法直接横向对比的，因为分母不同。
无论模型怎么分词，底层的原始文本文件大小（以Byte为单位）是绝对固定的。把Loss转换成BPB，相当于把评价标准统一拉到了“模型每看到原始文本的一个字节，能提供多少比特的信息量”。
只有通过BPB，你才能公平地把你的nanochat模型去和市面上其他架构、不同词表大小的模型去全方位对比。
永远不要跨越不同的Tokenizer去对比Token Loss。如果要对比，要么看它们在相同测试集上的BPB，要么直接看它们在实际对话时的回答质量。
下载合成身份对话数据，让模型更像人类：
curl -L -o $NANOCHAT_BASE_DIR/identity_conversations.jsonl https://karpathy-public.s3.us-west-2.amazonaws.com/identity_conversations.jsonl
需梯子:
pip install huggingface_hub
huggingface-cli login
下载smol-smoltalk数据集:
https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk
hf download HuggingFaceTB/smol-smoltalk --repo-type dataset
mkdir -p nanochat/HuggingFaceTB/smol-smoltalk
cp -rL拷贝
下载cais/mmlu数据集:
https://huggingface.co/datasets/cais/mmlu
hf download cais/mmlu --repo-type dataset
mkdir -p nanochat/cais/mmlu
cp -rL拷贝
下载openai/gsm8k数据集:
https://huggingface.co/datasets/openai/gsm8k
hf download openai/gsm8k --repo-type dataset
mkdir -p nanochat/openai/gsm8k
cp -rL拷贝
下载allenai/ai2_arc数据集:
https://huggingface.co/datasets/allenai/ai2_arc
hf download allenai/ai2_arc --repo-type dataset
mkdir -p nanochat/allenai/ai2_arc
cp -rL拷贝
下载openai/openai_humaneval数据集:
https://huggingface.co/datasets/openai/openai_humaneval
hf download openai/openai_humaneval --repo-type dataset
mkdir -p nanochat/openai/openai_humaneval
cp -rL拷贝
下载https://raw.githubusercontent.com/dwyl/english-words/refs/heads/master/words_alpha.txt
放入$NANOCHAT_BASE_DIR/
SFT监督微调(conversation special tokens, tool use, multiple choice):
pip install datasets
python -m scripts.chat_sft --device-batch-size=16
耗时70分钟，开启FlashAttention耗时缩短为41分钟。
Minimum validation bpb: 0.3557
评估模型:
python -m scripts.chat_eval -i sft
ARC-Easy accuracy: 36.15%
ARC-Challenge accuracy: 32.25%
MMLU accuracy: 31.52%
GSM8K accuracy: 3.34%
HumanEval accuracy: 9.76%
SpellingBee accuracy: 99.22%
与模型对话:
python -m scripts.chat_cli -p "Why is the sky blue?"
python -m scripts.chat_web
生成报告:
python -m nanochat.report generate