跳转至

my pretrain history

pretrain

.py:2134] 2024-10-02 22:46:14,378 >> ***** Running training *****
[INFO|trainer.py:2135] 2024-10-02 22:46:14,378 >>   Num examples = 16,527,707
[INFO|trainer.py:2136] 2024-10-02 22:46:14,378 >>   Num Epochs = 3
[INFO|trainer.py:2137] 2024-10-02 22:46:14,378 >>   Instantaneous batch size per device = 8
[INFO|trainer.py:2140] 2024-10-02 22:46:14,378 >>   Total train batch size (w. parallel, distributed & accumulation) = 8
[INFO|trainer.py:2141] 2024-10-02 22:46:14,378 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:2142] 2024-10-02 22:46:14,378 >>   Total optimization steps = 6,197,892
[INFO|trainer.py:2143] 2024-10-02 22:46:14,380 >>   Number of trainable parameters = 1,235,814,400

per_device_train_batch_size: 1  
gradient_accumulation_steps: 8  
learning_rate: 1.0e-4  
num_train_epochs: 3.0  
lr_scheduler_type: cosine  
warmup_ratio: 0.1  
bf16: true

16,527,707 * 3 /8 = 6,197,892

25gb

py:2134] 2024-10-04 08:58:31,915 >> ***** Running training ***** [INFO|trainer.py:2135] 2024-10-04 08:58:31,915 >> Num examples = 826,279 [INFO|trainer.py:2136] 2024-10-04 08:58:31,915 >> Num Epochs = 3 [INFO|trainer.py:2137] 2024-10-04 08:58:31,915 >> Instantaneous batch size per device = 1 [INFO|trainer.py:2140] 2024-10-04 08:58:31,915 >> Total train batch size (w. parallel, distributed & accumulation) = 8 [INFO|trainer.py:2141] 2024-10-04 08:58:31,915 >> Gradient Accumulation steps = 8 [INFO|trainer.py:2142] 2024-10-04 08:58:31,915 >> Total optimization steps = 309,852 [INFO|trainer.py:2143] 2024-10-04 08:58:31,916 >> Number of trainable parameters = 1,235,814,400

1 输出格式

2 训练资源占用

2.1 100M 模型/ V100

V100是15 TFLOPS FP32算力。 1. 当cutoff_len为1时，算力没有跑满，内存瓶颈。 2. 当cutoff_len为8192时，算力应该差不多跑满了。此时算力为6（参数量） token数 * 2.34(speed) = 11T。接近15T.(没有用到Tensor Core，seq_len太多，在attention引入了更多计算) 3. 当cutoff_len为1024，gradient_accumulation_steps=8时，理论上速度与上一条相同，但实际慢了一些，可能是因为多了7轮内存访问。 4. 当cutoff_len为1024, batch_size为8时，因为seq_length小，计算量小了些的缘故。按6 * 100 * 1024 * 8 * 3.3 = 16T，比15T大一些。

task fp16 speed num_gpu batch size per gpu gradient_accumulation_steps cutoff_len gpu memory usage(GB) memory bandwidth
pretrain true 12.9 it/s 1 1 1 1 2.579
pretrain true 12.7 it/s 1 1 1 64 2.617
pretrain true 12.4 it/s 1 1 1 128 2.729
pretrain true 12.4 it/s 1 1 1 256 2.951
pretrain true 12.4 it/s 1 1 1 512 3.183
pretrain true 12.3it/s 1 1 1 1024 4.5
pretrain true 9.03 it/s 1 1 1 2048 6.583
pretrain true 5 it/s 1 1 1 4096 9.647
pretrain true 2.34 it/s 1 1 1 8192 18.067
pretrain true 12.3it/s 1 1 1 1024 4.5
pretrain true 1.82it/s 1 1 8 1024 4.529
pretrain true 1.2s/it 1 1 16 1024 4.529
pretrain true 2.26 it/s 1 12 1 1024 26.353
pretrain true 3.3it/s 1 8 1 1024 18.8 70%
pretrain true 2.5s/it 1 8 8 1024 18.0
pretrain true 4.8s/it 1 8 16 1024 18.0

A100:

  1. Training with DataParallel so batch size has been adjusted to: 8, 所以没法把batch size设成16或32，通过修改gradient_accumulation_steps来提高并行度，没有走作用。
  2. 计算算力是：6 * 100 * 1024 * 8 * 6.99 = 34 TFLOPS，A100的fp engine是19.5 FLOPS，是不够的，主要用的是tensor core。
  3. 看起来是内存瓶颈了，50%的内存带宽是1TB/s，1s计算的token数是1024 * 8 * 6.99 = 57,000个， 平均一个token访存17.5GB。这和memory usage是对得上的。但这个内存占用是1024 * 8才用到的，并不应该每个token都用到所有其他的token相关的内存。反向传播的内存访存是非常大的？怎么算？
  4. 对于较小的模型来讲，gradient_accumulation_steps应该设置为1 ，设大的主要作的是减少权重更新次数？但小模型权重更新写内存本来就不多。
task fp16 speed num_gpu batch size per gpu gradient_accumulation_steps cutoff_len gpu memory usage(GB) memory bandwidth tensor core fp engine
pretrain true 6.99 it/s 1 8 1 1024 18.8 50% 12.9% 3%
pretrain true 1.21 s/it 1 8 8 1024 18.8 50% 12.9% 3%

2.2 1B 模型/ V100 *6

num_gpu batch size per gpu gradient_accumulation_steps cutoff_len gpu memory usage(GB) 算力
6 16 1 1024

2.3 1B 模型/ A100

task speed(s/it) num_gpu batch size per gpu gradient_accumulation_steps cutoff_len gpu memory usage(GB) memory bandwidth fp32 fp32算力(TFLOPS)
pretrain 8.9 1 16 1 1024 42.73 10%(200GB/s) 90% 17.6
forward算力：2.4GFLOPS * 16 * 1024 = 38.4 TFLOPS(约需2s)，如果gradient_accumulation_steps设成8,那就是19s左右。