image->vit->data_parallel->cifar10 loss=Nan #1116

zhangqianYY · 2022-06-15T03:25:18Z

zhangqianYY
Jun 15, 2022

When I run vit data_parallel experiment by the following command:

colossalai run --nproc_per_node 1 train_with_cifar10.py --config config.py
I set : lr=1.8e-10 in train_with_cifar10.py. BATCH_SIZE = 8000, gradient_accumulation = 32 in config.py following Details, but got: Loss = nan.

log:

/usr/local/lib/python3.6/dist-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
  from cryptography.hazmat.backends import default_backend
[06/15/22 11:08:56] INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/context/parallel_context.py:521 set_device                                                 
                    INFO     colossalai - colossalai - INFO: process rank 0 is bound to device 0                                                                                                          
[06/15/22 11:09:03] INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/context/parallel_context.py:557 set_seed                                                   
                    INFO     colossalai - colossalai - INFO: initialized seed on rank 0, numpy: 1024, python random: 1024, ParallelMode.DATA: 1024, ParallelMode.TENSOR: 1024,the default parallel seed is
                             ParallelMode.DATA.                                                                                                                                                           
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/initialize.py:121 launch                                                                   
                    INFO     colossalai - colossalai - INFO: Distributed environment is initialized, data parallel size: 1, pipeline parallel size: 1, tensor parallel size: 1                            
{'BATCH_SIZE': 8000, 'DROP_RATE': 0.1, 'NUM_EPOCHS': 20, 'fp16': {'mode': <AMP_TYPE.TORCH: 'torch'>}, 'gradient_accumulation': 32, 'clip_grad_norm': 1.0, 'dali': {'gpu_aug': True, 'mixup_alpha': 0.2}}
                    INFO     colossalai - colossalai - INFO: train_with_cifar10.py:43 main                                                                                                                
                    INFO     colossalai - colossalai - INFO: initialized distributed environment                                                                                                          
Files already downloaded and verified
[06/15/22 11:09:18] INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/initialize.py:270 initialize                                                               
                    INFO     colossalai - colossalai - INFO:                                                                                                                                              
                             ========== Your Config ========                                                                                                                                              
                             {'BATCH_SIZE': 8000,                                                                                                                                                         
                              'DROP_RATE': 0.1,                                                                                                                                                           
                              'NUM_EPOCHS': 20,                                                                                                                                                           
                              'clip_grad_norm': 1.0,                                                                                                                                                      
                              'dali': {'gpu_aug': True, 'mixup_alpha': 0.2},                                                                                                                              
                              'fp16': {'mode': <AMP_TYPE.TORCH: 'torch'>},                                                                                                                                
                              'gradient_accumulation': 32}                                                                                                                                                
                             ================================                                                                                                                                             
                                                                                                                                                                                                          
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/initialize.py:278 initialize                                                               
                    INFO     colossalai - colossalai - INFO: cuDNN benchmark = True, deterministic = False                                                                                                
                    WARNING  colossalai - colossalai - WARNING: /usr/local/lib/python3.6/dist-packages/colossalai/initialize.py:304 initialize                                                            
                    WARNING  colossalai - colossalai - WARNING: Initializing an non ZeRO model with optimizer class                                                                                       
                    WARNING  colossalai - colossalai - WARNING: /usr/local/lib/python3.6/dist-packages/colossalai/initialize.py:439 initialize                                                            
                    WARNING  colossalai - colossalai - WARNING: No PyTorch DDP or gradient handler is set up, please make sure you do not need to all-reduce the gradients after a training step.         
                    INFO     colossalai - colossalai - INFO: train_with_cifar10.py:77 main                                                                                                                
                    INFO     colossalai - colossalai - INFO: initialized colossalai components                                                                                                            
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/_trainer.py:306 fit                                                                
                    INFO     colossalai - colossalai - INFO: Using LossHook for training, priority = 0                                                                                                    
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/_trainer.py:306 fit                                                                
                    INFO     colossalai - colossalai - INFO: Using AccuracyHook for training, priority = 0                                                                                                
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/_trainer.py:306 fit                                                                
                    INFO     colossalai - colossalai - INFO: Using LRSchedulerHook for training, priority = 1                                                                                             
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/_trainer.py:306 fit                                                                
                    INFO     colossalai - colossalai - INFO: Using LogMetricByEpochHook for training, priority = 10                                                                                       
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/_trainer.py:308 fit                                                                
                    INFO     colossalai - colossalai - INFO: Lower value means higher priority for calling hook function                                                                                  
[Epoch 0 / Train]: 0it [00:00, ?it/s]
/usr/local/lib/python3.6/dist-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 0 / Train]: Loss = nan | LR = 7.0588e-12                                                                                              
[Epoch 1 / Train]: 0it [00:00, ?it/s]
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 1 / Train]: Loss = nan | LR = 1.0588e-11                                                                                              
[Epoch 2 / Train]: 0it [00:00, ?it/s]
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 2 / Train]: Loss = nan | LR = 1.4118e-11                                                                                              
[Epoch 3 / Train]: 0it [00:00, ?it/s]
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 3 / Train]: Loss = nan | LR = 1.7647e-11                                                                                              
[Epoch 4 / Train]: 0it [00:00, ?it/s]
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 4 / Train]: Loss = nan | LR = 2.1176e-11                                                                                              
[Epoch 5 / Train]: 0it [00:00, ?it/s]
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 5 / Train]: Loss = nan | LR = 2.4706e-11                                                                                              
[Epoch 6 / Train]: 0it [00:00, ?it/s]
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 6 / Train]: Loss = nan | LR = 2.8235e-11                                                                                              
[Epoch 7 / Train]: 0it [00:00, ?it/s]
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 7 / Train]: Loss = nan | LR = 3.1765e-11                                                                                              
[Epoch 8 / Train]: 0it [00:00, ?it/s]
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 8 / Train]: Loss = nan | LR = 3.5294e-11                                                                                              
[Epoch 9 / Train]: 0it [00:00, ?it/s]
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 9 / Train]: Loss = nan | LR = 3.8824e-11                                                                                              
[Epoch 10 / Train]: 0it [00:00, ?it/s]
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 10 / Train]: Loss = nan | LR = 4.2353e-11                                                                                             
[Epoch 11 / Train]: 0it [00:00, ?it/s]
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 11 / Train]: Loss = nan | LR = 4.5882e-11                                                                                             
[Epoch 12 / Train]: 0it [00:00, ?it/s]
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 12 / Train]: Loss = nan | LR = 4.9412e-11                                                                                             
[Epoch 13 / Train]: 0it [00:00, ?it/s]
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 13 / Train]: Loss = nan | LR = 5.2941e-11                                                                                             
[Epoch 14 / Train]: 0it [00:00, ?it/s]
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 14 / Train]: Loss = nan | LR = 5.6471e-11                                                                                             
[Epoch 15 / Train]: 0it [00:00, ?it/s]
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 15 / Train]: Loss = nan | LR = 6e-11                                                                                                  
[Epoch 16 / Train]: 0it [00:00, ?it/s]
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 16 / Train]: Loss = nan | LR = 6.3529e-11                                                                                             
[Epoch 17 / Train]: 0it [00:00, ?it/s]
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 17 / Train]: Loss = nan | LR = 6.7059e-11                                                                                             
[Epoch 18 / Train]: 0it [00:00, ?it/s]
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 18 / Train]: Loss = nan | LR = 7.0588e-11                                                                                             
[Epoch 19 / Train]: 0it [00:00, ?it/s]
                    INFO     colossalai - colossalai - INFO: /usr/local/lib/python3.6/dist-packages/colossalai/trainer/hooks/_log_hook.py:92 after_train_epoch                                            
                    INFO     colossalai - colossalai - INFO: [Epoch 19 / Train]: Loss = nan | LR = 7.4118e-11

Environment
I install ColossalAI via
pip install colossalai==0.1.6+torch1.10cu10.2 -f https://release.colossalai.org

GPU: v100, memory:32510MiB

Answered by BoxiangW

Jun 15, 2022

Hi, I have looked into your log info. I am trying to reproduce this error and find the reason right now.

View full answer

BoxiangW · 2022-06-15T03:29:30Z

BoxiangW
Jun 15, 2022

Hi, I have looked into your log info. I am trying to reproduce this error and find the reason right now.

13 replies

BoxiangW Jun 15, 2022

This is because your BATCH_SIZE is too big for your GPU. You can lower your BATCH_SIZE and increase gradient_accumulation. (gradient_accumulation is introduced to relieve the OOM situation)

BoxiangW Jun 15, 2022

For a V100 40G GPU, I recommend you to try BATCH_SIZE around 128 and increase gradient_accumulation.

zhangqianYY Jun 15, 2022
Author

thx,i got A100 memory is 40G or 80G from nvidia-a100, v100 32G indeeds does less them.Thank you again for your reply.

zhangqianYY Jun 15, 2022
Author

I actually want to use colossal-AI to train clip(this is similar to Simclr, but image-text for positive pair) ,To improve the accuracy, I need to increase the batchsize,In theory, The bigger the better. What paradigms of parallelism do you recommend( Data Parallel? Model Parallel? Tensor Parallel? Pipeline Parallel? ) only for increase batchsize? Some examples are better for reference.

BoxiangW Jun 15, 2022

If you could use a lot of GPUs, you can try data parallel. While greater gradient_accumulation could help you achieve a greater batch size. (total_batch_size = BATCH_SIZE * gradient_accumulation).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

image->vit->data_parallel->cifar10 loss=Nan #1116

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 13 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

image->vit->data_parallel->cifar10 loss=Nan #1116

zhangqianYY Jun 15, 2022

Replies: 1 comment · 13 replies

BoxiangW Jun 15, 2022

BoxiangW Jun 15, 2022

BoxiangW Jun 15, 2022

zhangqianYY Jun 15, 2022 Author

zhangqianYY Jun 15, 2022 Author

BoxiangW Jun 15, 2022

zhangqianYY
Jun 15, 2022

Replies: 1 comment 13 replies

BoxiangW
Jun 15, 2022

zhangqianYY Jun 15, 2022
Author

zhangqianYY Jun 15, 2022
Author