loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1 loaded library: loaded library: loaded library: loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1/usr/lib/x86_64-linux-gnu/libibverbs.so.1 /usr/lib/x86_64-linux-gnu/libibverbs.so.1 /usr/lib/x86_64-linux-gnu/libibverbs.so.1 [07/05 14:06:43 libai]: Rank of current process: 0. World size: 4 [07/05 14:06:43 libai]: Command line arguments: Namespace(config_file='configs/bert_nl24_nah16_hs1024.py', eval_only=False, fast_dev_run=False, opts=['model.cfg.hidden_layers=24', 'train.dist.pipeline_num_layers=24', 'train.train_micro_batch_size=128', 'train.global_batch_size=2048', 'train.dist.tensor_parallel_size=2', 'train.dist.pipeline_parallel_size=1', 'train.amp.enabled=true', 'train.activation_checkpoint.enabled=true', 'train.train_iter=220', 'train.log_period=100', 'train.output_dir=test_logs/01b1d32/1n4g/LibAI_bert_nl24_nah16_hs1024_FP16_actrue_mp2_pp1_mb128_gb2048_1n4g_20220705_140640645048573'], resume=False) [07/05 14:06:43 libai]: Contents of args.config_file=configs/bert_nl24_nah16_hs1024.py: from libai.config import LazyCall from libai.evaluation import PPLEvaluator from .common.models.bert import pretrain_model as model from .common.models.graph import graph from .common.train import train from .common.optim import optim from .common.data.bert_dataset import dataloader, tokenization #vocab_file = "/workspace/dataset/bert-base-chinese-vocab.txt" #data_prefix = "/workspace/dataset/loss_compara_content_sentence" vocab_file = "/dataset/source/dataset/bert-base-chinese-vocab.txt" data_prefix = "/dataset/source/dataset/loss_compara_content_sentence" tokenization.tokenizer.vocab_file = vocab_file dataloader.train.dataset[0].data_prefix = data_prefix dataloader.train.dataset[0].indexed_dataset.data_prefix = data_prefix # dataloader.train.num_workers = 4 # Bert-large model config #model.cfg.hidden_layers = 24 model.cfg.num_attention_heads = 16 model.cfg.hidden_size = 1024 #train.dist.pipeline_num_layers = model.cfg.hidden_layers train.test_micro_batch_size = 4 train.evaluation.evaluator = LazyCall(PPLEvaluator)() train.input_placement_device = "cpu" train.evaluation.enabled = False train.evaluation.eval_iter = 30 [07/05 14:06:43 libai]: Full config saved to test_logs/01b1d32/1n4g/LibAI_bert_nl24_nah16_hs1024_FP16_actrue_mp2_pp1_mb128_gb2048_1n4g_20220705_140640645048573/config.yaml [07/05 14:06:43 lb.engine.default]: > compiling dataset index builder ... make: Entering directory '/dataset/xyn/libai_bench/libai/libai/data/data_utils' make: Nothing to be done for 'default'. make: Leaving directory '/dataset/xyn/libai_bench/libai/libai/data/data_utils' [07/05 14:06:43 lb.engine.default]: >>> done with dataset index builder. Compilation time: 0.041 seconds [07/05 14:06:43 lb.engine.default]: >>> done with compiling. Compilation time: 0.042 seconds [07/05 14:06:43 lb.engine.default]: Prepare training, validating, testing set [07/05 14:06:43 lb.data.data_utils.indexed_dataset]: building dataset index ... [07/05 14:06:43 lb.data.data_utils.indexed_dataset]: warming up index mmap file... [07/05 14:06:43 lb.data.data_utils.indexed_dataset]: reading sizes... [07/05 14:06:43 lb.data.data_utils.indexed_dataset]: reading pointers... [07/05 14:06:43 lb.data.data_utils.indexed_dataset]: reading document index... [07/05 14:06:43 lb.data.data_utils.indexed_dataset]: warming up data mmap file... [07/05 14:06:43 lb.data.data_utils.indexed_dataset]: creating numpy buffer of mmap... [07/05 14:06:43 lb.data.data_utils.indexed_dataset]: creating memory view of numpy buffer... [07/05 14:06:43 lb.data.data_utils.indexed_dataset]: Finished creating indexed dataset in 0.105216 seconds [07/05 14:06:43 lb.data.data_utils.indexed_dataset]: indexed dataset stats: [07/05 14:06:43 lb.data.data_utils.indexed_dataset]: number of documents: 50000 [07/05 14:06:43 lb.data.data_utils.indexed_dataset]: number of sentences: 1249934 [07/05 14:06:43 lb.data.data_utils.dataset_utils]:  > loading indexed mapping from /dataset/source/dataset/loss_compara_content_sentence_bert_indexmap_450560mns_509msl_0.10ssp_1234s.npy [07/05 14:06:43 lb.data.data_utils.dataset_utils]:  loaded indexed file in 0.004 seconds [07/05 14:06:43 lb.data.data_utils.dataset_utils]:  total number of samples: 452417 [07/05 14:06:43 lb.data.data_utils.dataset_utils]:  > loading indexed mapping from /dataset/source/dataset/loss_compara_content_sentence_bert_indexmap_8mns_509msl_0.10ssp_1234s.npy [07/05 14:06:43 lb.data.data_utils.dataset_utils]:  loaded indexed file in 0.001 seconds [07/05 14:06:43 lb.data.data_utils.dataset_utils]:  total number of samples: 5884 [07/05 14:06:43 lb.data.data_utils.dataset_utils]:  > loading indexed mapping from /dataset/source/dataset/loss_compara_content_sentence_bert_indexmap_8mns_509msl_0.10ssp_1234s.npy [07/05 14:06:43 lb.data.data_utils.dataset_utils]:  loaded indexed file in 0.001 seconds [07/05 14:06:43 lb.data.data_utils.dataset_utils]:  total number of samples: 5884 [07/05 14:06:44 lb.engine.default]: Auto-scaling the config to train.train_iter=220, train.warmup_iter=0 iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol. iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol. iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO P2P plugin IBext iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Using network IBext NCCL version 2.12.10+cuda11.2 iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol. iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol. iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO P2P plugin IBext iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol. iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol. iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol. iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol. iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO P2P plugin IBext iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO P2P plugin IBext iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Using network IBext iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO Using network IBext iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Using network IBext iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO PXN Disabled as plugin is v4 iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO PXN Disabled as plugin is v4 iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO PXN Disabled as plugin is v4 iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO PXN Disabled as plugin is v4 iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 00/08 : 0 2 3 1 iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 0/-1/-1->2->3 [2] 3/-1/-1->2->0 [3] 0/-1/-1->2->3 [4] 3/-1/-1->2->0 [5] 0/-1/-1->2->3 [6] 3/-1/-1->2->0 [7] 0/-1/-1->2->3 iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO Trees [0] -1/-1/-1->1->3 [1] 3/-1/-1->1->-1 [2] -1/-1/-1->1->3 [3] 3/-1/-1->1->-1 [4] -1/-1/-1->1->3 [5] 3/-1/-1->1->-1 [6] -1/-1/-1->1->3 [7] 3/-1/-1->1->-1 iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Trees [0] 1/-1/-1->3->2 [1] 2/-1/-1->3->1 [2] 1/-1/-1->3->2 [3] 2/-1/-1->3->1 [4] 1/-1/-1->3->2 [5] 2/-1/-1->3->1 [6] 1/-1/-1->3->2 [7] 2/-1/-1->3->1 iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 01/08 : 0 2 1 3 iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 02/08 : 0 1 3 2 iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 03/08 : 0 3 1 2 iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 04/08 : 0 2 3 1 iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 05/08 : 0 2 1 3 iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 06/08 : 0 1 3 2 iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 07/08 : 0 3 1 2 iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Trees [0] 2/-1/-1->0->-1 [1] -1/-1/-1->0->2 [2] 2/-1/-1->0->-1 [3] -1/-1/-1->0->2 [4] 2/-1/-1->0->-1 [5] -1/-1/-1->0->2 [6] 2/-1/-1->0->-1 [7] -1/-1/-1->0->2 iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Channel 01 : 3[67020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Channel 04 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO Channel 03 : 1[65020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 02 : 0[65010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Channel 05 : 3[67020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO Channel 07 : 1[65020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 06 : 0[65010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Channel 02 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Channel 03 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Channel 03 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO Channel 02 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Channel 06 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Channel 04 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO Channel 05 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 04 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Channel 07 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Channel 07 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO Channel 06 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 05 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Channel 02 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO Channel 00 : 1[65020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Channel 01 : 2[67010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 03 : 0[65010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Channel 06 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO Channel 04 : 1[65020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Channel 05 : 2[67010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 07 : 0[65010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Channel 02 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Channel 03 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Channel 05 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Channel 06 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 02 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO Channel 03 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Channel 07 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 03 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO Channel 04 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 06 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO Channel 07 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Channel 07 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Channel 02 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Channel 04 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Channel 05 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Channel 05 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Channel 06 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Channel 03 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Channel 04 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Channel 05 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Channel 07 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34657:35241 [1] NCCL INFO comm 0x7f7c305342b0 rank 1 nranks 4 cudaDev 1 busId 65020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO comm 0x7f3bc0766b70 rank 0 nranks 4 cudaDev 0 busId 65010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34659:35245 [3] NCCL INFO comm 0x7ff250841670 rank 3 nranks 4 cudaDev 3 busId 67020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34658:35242 [2] NCCL INFO comm 0x7f455489faf0 rank 2 nranks 4 cudaDev 2 busId 67010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34656:35243 [0] NCCL INFO Launch mode Parallel [07/05 14:06:57 lb.engine.default]: Model: BertForPreTraining( (bert): BertModel( (embeddings): BertEmbeddings( (vocab_embeddings): VocabEmbedding(num_embeddings=21248, embedding_dim=1024) (position_embeddings): Embedding(num_embeddings=512, embedding_dim=1024) (tokentype_embeddings): Embedding(num_embeddings=2, embedding_dim=1024) (embedding_dropout): Dropout(p=0.1, inplace=False) ) (extended_attn_mask): BertExtendedAttnMask() (encoders): ModuleList( (0): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (1): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (2): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (3): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (4): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (5): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (6): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (7): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (8): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (9): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (10): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (11): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (12): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (13): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (14): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (15): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (16): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (17): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (18): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (19): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (20): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (21): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (22): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (23): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) ) (final_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (pooler): BertPooler( (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=col) (activation_func): Tanh() ) ) (cls_head): BertPreTrainingHeads( (predictions): BertLMPredictionHead( (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=data) (activation_func): GELU() (layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) ) (seq_relationship): Linear1D(in_features=1024, out_features=2, bias=True, parallel=data) (lm_logits): LMLogits() (loss_func): BertLoss( (lm_loss): ParallelCrossEntropyLoss() ) ) ) WARNING [07/05 14:06:57 lb.scheduler.lr_scheduler]: warmup iters equals to zero, return CosineLR [07/05 14:06:58 lb.engine.trainer]: Starting training from iteration 0 [07/05 14:07:04 lb.models.utils.graph_base]: Start compling the train graph which may take some time. Please wait for a moment ... iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 00/08 : 0 2 3 1 iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 01/08 : 0 2 1 3 iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 02/08 : 0 1 3 2 iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Trees [0] 1/-1/-1->3->2 [1] 2/-1/-1->3->1 [2] 1/-1/-1->3->2 [3] 2/-1/-1->3->1 [4] 1/-1/-1->3->2 [5] 2/-1/-1->3->1 [6] 1/-1/-1->3->2 [7] 2/-1/-1->3->1 iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 0/-1/-1->2->3 [2] 3/-1/-1->2->0 [3] 0/-1/-1->2->3 [4] 3/-1/-1->2->0 [5] 0/-1/-1->2->3 [6] 3/-1/-1->2->0 [7] 0/-1/-1->2->3 iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 03/08 : 0 3 1 2 iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Trees [0] -1/-1/-1->1->3 [1] 3/-1/-1->1->-1 [2] -1/-1/-1->1->3 [3] 3/-1/-1->1->-1 [4] -1/-1/-1->1->3 [5] 3/-1/-1->1->-1 [6] -1/-1/-1->1->3 [7] 3/-1/-1->1->-1 iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 04/08 : 0 2 3 1 iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 05/08 : 0 2 1 3 iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 06/08 : 0 1 3 2 iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 07/08 : 0 3 1 2 iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Trees [0] 2/-1/-1->0->-1 [1] -1/-1/-1->0->2 [2] 2/-1/-1->0->-1 [3] -1/-1/-1->0->2 [4] 2/-1/-1->0->-1 [5] -1/-1/-1->0->2 [6] 2/-1/-1->0->-1 [7] -1/-1/-1->0->2 iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 02 : 0[65010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 01 : 3[67020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 06 : 0[65010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 03 : 1[65020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 05 : 3[67020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 07 : 1[65020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 04 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 02 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 02 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 03 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 03 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 04 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 05 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 04 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 06 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 05 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 06 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 07 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 07 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 00 : 1[65020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 02 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 03 : 0[65010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 01 : 2[67010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 04 : 1[65020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 07 : 0[65010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 06 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 05 : 2[67010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 02 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 03 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 05 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 02 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 06 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 03 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 03 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 07 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 06 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 04 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 07 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 07 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 02 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 05 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 04 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 06 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 05 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 03 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 04 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 05 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 07 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO comm 0x564e09c38990 rank 0 nranks 4 cudaDev 0 busId 65010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO comm 0x55f02fd3b720 rank 1 nranks 4 cudaDev 1 busId 65020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO comm 0x55bc1b55fba0 rank 3 nranks 4 cudaDev 3 busId 67020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO comm 0x565433fe9c50 rank 2 nranks 4 cudaDev 2 busId 67010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 00/08 : 0 2 3 1 iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 01/08 : 0 2 1 3 iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 02/08 : 0 1 3 2 iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Trees [0] 1/-1/-1->3->2 [1] 2/-1/-1->3->1 [2] 1/-1/-1->3->2 [3] 2/-1/-1->3->1 [4] 1/-1/-1->3->2 [5] 2/-1/-1->3->1 [6] 1/-1/-1->3->2 [7] 2/-1/-1->3->1 iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 03/08 : 0 3 1 2 iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 0/-1/-1->2->3 [2] 3/-1/-1->2->0 [3] 0/-1/-1->2->3 [4] 3/-1/-1->2->0 [5] 0/-1/-1->2->3 [6] 3/-1/-1->2->0 [7] 0/-1/-1->2->3 iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Trees [0] -1/-1/-1->1->3 [1] 3/-1/-1->1->-1 [2] -1/-1/-1->1->3 [3] 3/-1/-1->1->-1 [4] -1/-1/-1->1->3 [5] 3/-1/-1->1->-1 [6] -1/-1/-1->1->3 [7] 3/-1/-1->1->-1 iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 04/08 : 0 2 3 1 iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 05/08 : 0 2 1 3 iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 06/08 : 0 1 3 2 iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 07/08 : 0 3 1 2 iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Trees [0] 2/-1/-1->0->-1 [1] -1/-1/-1->0->2 [2] 2/-1/-1->0->-1 [3] -1/-1/-1->0->2 [4] 2/-1/-1->0->-1 [5] -1/-1/-1->0->2 [6] 2/-1/-1->0->-1 [7] -1/-1/-1->0->2 iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 01 : 3[67020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 05 : 3[67020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 02 : 0[65010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 03 : 1[65020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 04 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 06 : 0[65010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 07 : 1[65020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 02 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 03 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 03 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 02 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 04 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 04 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 06 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 05 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 07 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 05 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 07 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 06 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 03 : 0[65010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 02 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 01 : 2[67010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 00 : 1[65020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 07 : 0[65010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 05 : 2[67010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 06 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 04 : 1[65020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 02 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 03 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 05 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 02 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 06 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 07 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 03 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 03 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 06 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 04 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Channel 07 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Channel 07 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 02 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 05 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 04 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 06 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Channel 05 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 03 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 04 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 05 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Channel 07 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO comm 0x55f02fe76ab0 rank 1 nranks 4 cudaDev 1 busId 65020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO comm 0x564e09d77560 rank 0 nranks 4 cudaDev 0 busId 65010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO comm 0x56543412b860 rank 2 nranks 4 cudaDev 2 busId 67010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO comm 0x55bc1b6a1c80 rank 3 nranks 4 cudaDev 3 busId 67020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Launch mode Parallel NCCL version 2.12.10+cuda11.2 iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Channel 00/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Channel 01/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Channel 02/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Channel 03/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO Channel 00/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO Channel 01/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO Channel 02/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO Channel 03/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Channel 00 : 1[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Channel 00 : 0[65010] -> 1[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Channel 01 : 1[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Channel 01 : 0[65010] -> 1[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Channel 02 : 1[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Channel 02 : 0[65010] -> 1[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Channel 03 : 1[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Channel 03 : 0[65010] -> 1[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO Channel 00 : 1[67020] -> 0[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO Channel 00 : 0[65020] -> 1[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO Channel 01 : 1[67020] -> 0[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO Channel 01 : 0[65020] -> 1[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO Channel 02 : 1[67020] -> 0[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO Channel 02 : 0[65020] -> 1[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO Channel 03 : 1[67020] -> 0[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO Channel 03 : 0[65020] -> 1[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO comm 0x7f4014425350 rank 1 nranks 2 cudaDev 2 busId 67010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO comm 0x7f330451e250 rank 0 nranks 2 cudaDev 0 busId 65010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Launch mode Parallel NCCL version 2.12.10+cuda11.2 iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO comm 0x7fed00411c20 rank 1 nranks 2 cudaDev 3 busId 67020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO comm 0x7f76ec4201d0 rank 0 nranks 2 cudaDev 1 busId 65020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO Launch mode Parallel iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Channel 00/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Channel 01/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Channel 02/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Channel 03/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Channel 00/02 : 0 1 iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Channel 01/02 : 0 1 iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Channel 00 : 0[65010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO Channel 00 : 1[65020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO Channel 00 : 1[67020] -> 0[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Channel 00 : 0[67010] -> 1[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Channel 01 : 0[65010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO Channel 01 : 1[65020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO Channel 01 : 1[67020] -> 0[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Channel 01 : 0[67010] -> 1[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO Channel 02 : 1[67020] -> 0[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Channel 02 : 0[67010] -> 1[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO Channel 03 : 1[67020] -> 0[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Channel 03 : 0[67010] -> 1[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO comm 0x7f33045f1c60 rank 0 nranks 2 cudaDev 0 busId 65010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34656:35867 [0] NCCL INFO Launch mode Parallel iv-2udaavw4l02thdv8lcrl:34657:35876 [1] NCCL INFO comm 0x7f76ec4f5e40 rank 1 nranks 2 cudaDev 1 busId 65020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34659:35870 [3] NCCL INFO comm 0x7fed004e93a0 rank 1 nranks 2 cudaDev 3 busId 67020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO comm 0x7f40148efed0 rank 0 nranks 2 cudaDev 2 busId 67010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34658:35763 [2] NCCL INFO Launch mode Parallel iv-2udaavw4l02thdv8lcrl:34656:35973 [0] NCCL INFO Channel 01 : 0[65010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35973 [0] NCCL INFO Channel 05 : 0[65010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35974 [1] NCCL INFO Channel 03 : 1[65020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35974 [1] NCCL INFO Channel 07 : 1[65020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35976 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35976 [2] NCCL INFO Channel 05 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35975 [3] NCCL INFO Channel 03 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35975 [3] NCCL INFO Channel 07 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35978 [1] NCCL INFO Channel 00 : 1[65020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35977 [0] NCCL INFO Channel 00 : 0[65010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35978 [1] NCCL INFO Channel 01 : 1[65020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35977 [0] NCCL INFO Channel 01 : 0[65010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35979 [3] NCCL INFO Channel 00 : 1[67020] -> 0[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35980 [2] NCCL INFO Channel 00 : 0[67010] -> 1[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35979 [3] NCCL INFO Channel 01 : 1[67020] -> 0[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35980 [2] NCCL INFO Channel 01 : 0[67010] -> 1[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35979 [3] NCCL INFO Channel 02 : 1[67020] -> 0[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35980 [2] NCCL INFO Channel 02 : 0[67010] -> 1[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35979 [3] NCCL INFO Channel 03 : 1[67020] -> 0[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35980 [2] NCCL INFO Channel 03 : 0[67010] -> 1[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35864 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34659:35900 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34657:35864 [1] NCCL INFO Channel 00/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34659:35900 [3] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 iv-2udaavw4l02thdv8lcrl:34657:35864 [1] NCCL INFO Channel 01/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34657:35864 [1] NCCL INFO Channel 02/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34657:35864 [1] NCCL INFO Channel 03/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34657:35864 [1] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 iv-2udaavw4l02thdv8lcrl:34658:35756 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34656:35849 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:34658:35756 [2] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 iv-2udaavw4l02thdv8lcrl:34656:35849 [0] NCCL INFO Channel 00/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34656:35849 [0] NCCL INFO Channel 01/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34656:35849 [0] NCCL INFO Channel 02/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34656:35849 [0] NCCL INFO Channel 03/04 : 0 1 iv-2udaavw4l02thdv8lcrl:34656:35849 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 iv-2udaavw4l02thdv8lcrl:34659:35900 [3] NCCL INFO Channel 00 : 1[67020] -> 0[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35864 [1] NCCL INFO Channel 00 : 0[65020] -> 1[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35900 [3] NCCL INFO Channel 01 : 1[67020] -> 0[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35864 [1] NCCL INFO Channel 01 : 0[65020] -> 1[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35900 [3] NCCL INFO Channel 02 : 1[67020] -> 0[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35864 [1] NCCL INFO Channel 02 : 0[65020] -> 1[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35864 [1] NCCL INFO Channel 03 : 0[65020] -> 1[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:35900 [3] NCCL INFO Channel 03 : 1[67020] -> 0[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35756 [2] NCCL INFO Channel 00 : 1[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35849 [0] NCCL INFO Channel 00 : 0[65010] -> 1[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35756 [2] NCCL INFO Channel 01 : 1[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35849 [0] NCCL INFO Channel 01 : 0[65010] -> 1[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35756 [2] NCCL INFO Channel 02 : 1[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35849 [0] NCCL INFO Channel 02 : 0[65010] -> 1[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35756 [2] NCCL INFO Channel 03 : 1[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35849 [0] NCCL INFO Channel 03 : 0[65010] -> 1[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:35864 [1] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34657:35864 [1] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34657:35864 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34657:35864 [1] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34659:35900 [3] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34659:35900 [3] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34659:35900 [3] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34659:35900 [3] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34656:35849 [0] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34656:35849 [0] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34656:35849 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34656:35849 [0] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34658:35756 [2] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:34658:35756 [2] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:34658:35756 [2] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:34658:35756 [2] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer iv-2udaavw4l02thdv8lcrl:34657:35864 [1] NCCL INFO comm 0x7f76f8c07040 rank 0 nranks 2 cudaDev 1 busId 65020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34657:35864 [1] NCCL INFO Launch mode Parallel iv-2udaavw4l02thdv8lcrl:34659:35900 [3] NCCL INFO comm 0x7fed0cc393b0 rank 1 nranks 2 cudaDev 3 busId 67020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34657:36056 [1] NCCL INFO Channel 01 : 1[65020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:36056 [1] NCCL INFO Channel 05 : 1[65020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:36057 [3] NCCL INFO Channel 01 : 3[67020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:35849 [0] NCCL INFO comm 0x7f3310e31e90 rank 0 nranks 2 cudaDev 0 busId 65010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34656:35849 [0] NCCL INFO Launch mode Parallel iv-2udaavw4l02thdv8lcrl:34656:35849 [0] NCCL INFO Launch mode Parallel iv-2udaavw4l02thdv8lcrl:34659:36057 [3] NCCL INFO Channel 05 : 3[67020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:35756 [2] NCCL INFO comm 0x7f4020bf2770 rank 1 nranks 2 cudaDev 2 busId 67010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:34656:36058 [0] NCCL INFO Channel 01 : 0[65010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:36059 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:36058 [0] NCCL INFO Channel 05 : 0[65010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:36059 [2] NCCL INFO Channel 05 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:36058 [0] NCCL INFO Channel 02 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:36056 [1] NCCL INFO Channel 02 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:36057 [3] NCCL INFO Channel 02 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:36059 [2] NCCL INFO Channel 02 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:36058 [0] NCCL INFO Channel 06 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:36056 [1] NCCL INFO Channel 06 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:36059 [2] NCCL INFO Channel 06 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:36057 [3] NCCL INFO Channel 06 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:36058 [0] NCCL INFO Channel 03 : 0[65010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:36059 [2] NCCL INFO Channel 03 : 2[67010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:36056 [1] NCCL INFO Channel 03 : 1[65020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:36057 [3] NCCL INFO Channel 03 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34658:36059 [2] NCCL INFO Channel 07 : 2[67010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34656:36058 [0] NCCL INFO Channel 07 : 0[65010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34657:36056 [1] NCCL INFO Channel 07 : 1[65020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:34659:36057 [3] NCCL INFO Channel 07 : 3[67020] -> 2[67010] via P2P/IPC timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/07/05 14:46:22.353, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 70 %, 32510 MiB, 14302 MiB, 18208 MiB 2022/07/05 14:46:22.354, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 70 %, 32510 MiB, 14302 MiB, 18208 MiB 2022/07/05 14:46:22.355, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 14 %, 32510 MiB, 14034 MiB, 18476 MiB 2022/07/05 14:46:22.355, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 9 %, 32510 MiB, 14034 MiB, 18476 MiB 2022/07/05 14:46:22.356, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 32507 MiB, 3 MiB 2022/07/05 14:46:22.357, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 32507 MiB, 3 MiB timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/07/05 14:46:22.357, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 32507 MiB, 3 MiB 2022/07/05 14:46:22.359, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 70 %, 32510 MiB, 14302 MiB, 18208 MiB 2022/07/05 14:46:22.359, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 32507 MiB, 3 MiB 2022/07/05 14:46:22.359, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 70 %, 32510 MiB, 14302 MiB, 18208 MiB 2022/07/05 14:46:22.361, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 70 %, 32510 MiB, 14302 MiB, 18208 MiB 2022/07/05 14:46:22.361, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 70 %, 32510 MiB, 14302 MiB, 18208 MiB 2022/07/05 14:46:22.364, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 14 %, 32510 MiB, 14034 MiB, 18476 MiB 2022/07/05 14:46:22.364, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 14 %, 32510 MiB, 14034 MiB, 18476 MiB 2022/07/05 14:46:22.366, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 9 %, 32510 MiB, 14034 MiB, 18476 MiB 2022/07/05 14:46:22.366, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 9 %, 32510 MiB, 14034 MiB, 18476 MiB 2022/07/05 14:46:22.368, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 32507 MiB, 3 MiB 2022/07/05 14:46:22.368, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 32507 MiB, 3 MiB 2022/07/05 14:46:22.371, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 32507 MiB, 3 MiB 2022/07/05 14:46:22.371, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 32507 MiB, 3 MiB 2022/07/05 14:46:22.372, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 32507 MiB, 3 MiB 2022/07/05 14:46:22.373, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 32507 MiB, 3 MiB 2022/07/05 14:46:22.374, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 32507 MiB, 3 MiB 2022/07/05 14:46:22.374, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 32507 MiB, 3 MiB timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/07/05 14:46:45.463, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 71 %, 32510 MiB, 14302 MiB, 18208 MiB 2022/07/05 14:46:45.464, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 71 %, 32510 MiB, 14302 MiB, 18208 MiB 2022/07/05 14:46:45.465, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 48 %, 32510 MiB, 14034 MiB, 18476 MiB 2022/07/05 14:46:45.465, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 22 %, 32510 MiB, 14034 MiB, 18476 MiB 2022/07/05 14:46:45.466, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 32507 MiB, 3 MiB 2022/07/05 14:46:45.466, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 32507 MiB, 3 MiB 2022/07/05 14:46:45.467, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 32507 MiB, 3 MiB 2022/07/05 14:46:45.468, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 32507 MiB, 3 MiB [07/05 14:47:08 lb.utils.events]:  eta: 0:46:18 iteration: 99/220 consumed_samples: 204800 total_loss: 7.973 lm_loss: 7.275 sop_loss: 0.7003 time: 23.1601 s/iter data_time: 0.0121 s/iter total_throughput: 88.43 samples/s lr: 5.82e-05 [07/05 15:25:42 lb.utils.events]:  eta: 0:07:42 iteration: 199/220 consumed_samples: 409600 total_loss: 7.898 lm_loss: 7.201 sop_loss: 0.6948 time: 23.1499 s/iter data_time: 0.0123 s/iter total_throughput: 88.47 samples/s lr: 3.21e-06 [07/05 15:33:25 lb.utils.events]:  eta: 0:00:00 iteration: 219/220 consumed_samples: 450560 total_loss: 7.892 lm_loss: 7.197 sop_loss: 0.6939 time: 23.1492 s/iter data_time: 0.1148 s/iter total_throughput: 88.47 samples/s lr: 1.01e-06 [07/05 15:33:25 lb.engine.hooks]: Overall training speed: 218 iterations in 1:24:06 (23.1492 s / it) [07/05 15:33:25 lb.engine.hooks]: Total training time: 1:24:06 (0:00:00 on hooks) iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO comm 0x7f7c305342b0 rank 1 nranks 4 cudaDev 1 busId 65020 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO comm 0x7f3bc0766b70 rank 0 nranks 4 cudaDev 0 busId 65010 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO comm 0x7ff250841670 rank 3 nranks 4 cudaDev 3 busId 67020 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO comm 0x7f455489faf0 rank 2 nranks 4 cudaDev 2 busId 67010 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO comm 0x55f02fd3b720 rank 1 nranks 4 cudaDev 1 busId 65020 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO comm 0x564e09c38990 rank 0 nranks 4 cudaDev 0 busId 65010 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO comm 0x565433fe9c50 rank 2 nranks 4 cudaDev 2 busId 67010 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO comm 0x55bc1b55fba0 rank 3 nranks 4 cudaDev 3 busId 67020 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO comm 0x55f02fe76ab0 rank 1 nranks 4 cudaDev 1 busId 65020 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO comm 0x564e09d77560 rank 0 nranks 4 cudaDev 0 busId 65010 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO comm 0x7f76ec4f5e40 rank 1 nranks 2 cudaDev 1 busId 65020 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO comm 0x7f33045f1c60 rank 0 nranks 2 cudaDev 0 busId 65010 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO comm 0x7f76f8c07040 rank 0 nranks 2 cudaDev 1 busId 65020 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO comm 0x7f3310e31e90 rank 0 nranks 2 cudaDev 0 busId 65010 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34657:34657 [1] NCCL INFO comm 0x7f76ec4201d0 rank 0 nranks 2 cudaDev 1 busId 65020 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34656:34656 [0] NCCL INFO comm 0x7f330451e250 rank 0 nranks 2 cudaDev 0 busId 65010 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO comm 0x56543412b860 rank 2 nranks 4 cudaDev 2 busId 67010 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO comm 0x55bc1b6a1c80 rank 3 nranks 4 cudaDev 3 busId 67020 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO comm 0x7f4020bf2770 rank 1 nranks 2 cudaDev 2 busId 67010 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO comm 0x7fed0cc393b0 rank 1 nranks 2 cudaDev 3 busId 67020 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO comm 0x7f4014425350 rank 1 nranks 2 cudaDev 2 busId 67010 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO comm 0x7fed00411c20 rank 1 nranks 2 cudaDev 3 busId 67020 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34658:34658 [2] NCCL INFO comm 0x7f40148efed0 rank 0 nranks 2 cudaDev 2 busId 67010 - Destroy COMPLETE iv-2udaavw4l02thdv8lcrl:34659:34659 [3] NCCL INFO comm 0x7fed004e93a0 rank 1 nranks 2 cudaDev 3 busId 67020 - Destroy COMPLETE ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. *****************************************