loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1 loaded library: loaded library: loaded library: loaded library: loaded library: loaded library: loaded library: loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1 /usr/lib/x86_64-linux-gnu/libibverbs.so.1/usr/lib/x86_64-linux-gnu/libibverbs.so.1 /usr/lib/x86_64-linux-gnu/libibverbs.so.1/usr/lib/x86_64-linux-gnu/libibverbs.so.1/usr/lib/x86_64-linux-gnu/libibverbs.so.1 /usr/lib/x86_64-linux-gnu/libibverbs.so.1 /usr/lib/x86_64-linux-gnu/libibverbs.so.1 [07/05 20:43:17 libai]: Rank of current process: 0. World size: 16 [07/05 20:43:17 libai]: Command line arguments: Namespace(config_file='configs/bert_nl24_nah16_hs1024.py', eval_only=False, fast_dev_run=False, opts=['model.cfg.hidden_layers=24', 'train.dist.pipeline_num_layers=24', 'train.train_micro_batch_size=128', 'train.global_batch_size=2048', 'train.dist.tensor_parallel_size=8', 'train.dist.pipeline_parallel_size=1', 'train.amp.enabled=true', 'train.activation_checkpoint.enabled=true', 'train.train_iter=220', 'train.log_period=100', 'train.output_dir=test_logs/01b1d32/2n8g/LibAI_bert_nl24_nah16_hs1024_FP16_actrue_mp8_pp1_mb128_gb2048_2n8g_20220705_204305155951783'], resume=False) [07/05 20:43:18 libai]: Contents of args.config_file=configs/bert_nl24_nah16_hs1024.py: from libai.config import LazyCall from libai.evaluation import PPLEvaluator from .common.models.bert import pretrain_model as model from .common.models.graph import graph from .common.train import train from .common.optim import optim from .common.data.bert_dataset import dataloader, tokenization #vocab_file = "/workspace/dataset/bert-base-chinese-vocab.txt" #data_prefix = "/workspace/dataset/loss_compara_content_sentence" vocab_file = "/dataset/source/dataset/bert-base-chinese-vocab.txt" data_prefix = "/dataset/source/dataset/loss_compara_content_sentence" tokenization.tokenizer.vocab_file = vocab_file dataloader.train.dataset[0].data_prefix = data_prefix dataloader.train.dataset[0].indexed_dataset.data_prefix = data_prefix # dataloader.train.num_workers = 4 # Bert-large model config #model.cfg.hidden_layers = 24 model.cfg.num_attention_heads = 16 model.cfg.hidden_size = 1024 #train.dist.pipeline_num_layers = model.cfg.hidden_layers train.test_micro_batch_size = 4 train.evaluation.evaluator = LazyCall(PPLEvaluator)() train.input_placement_device = "cpu" train.evaluation.enabled = False train.evaluation.eval_iter = 30 [07/05 20:43:18 libai]: Full config saved to test_logs/01b1d32/2n8g/LibAI_bert_nl24_nah16_hs1024_FP16_actrue_mp8_pp1_mb128_gb2048_2n8g_20220705_204305155951783/config.yaml [07/05 20:43:18 lb.engine.default]: > compiling dataset index builder ... make: Entering directory '/dataset/xyn/libai_bench/libai/libai/data/data_utils' make: Nothing to be done for 'default'. make: Leaving directory '/dataset/xyn/libai_bench/libai/libai/data/data_utils' [07/05 20:43:18 lb.engine.default]: >>> done with dataset index builder. Compilation time: 0.041 seconds [07/05 20:43:18 lb.engine.default]: >>> done with compiling. Compilation time: 0.056 seconds [07/05 20:43:18 lb.engine.default]: Prepare training, validating, testing set [07/05 20:43:18 lb.data.data_utils.indexed_dataset]: building dataset index ... [07/05 20:43:18 lb.data.data_utils.indexed_dataset]: warming up index mmap file... [07/05 20:43:18 lb.data.data_utils.indexed_dataset]: reading sizes... [07/05 20:43:18 lb.data.data_utils.indexed_dataset]: reading pointers... [07/05 20:43:18 lb.data.data_utils.indexed_dataset]: reading document index... [07/05 20:43:18 lb.data.data_utils.indexed_dataset]: warming up data mmap file... [07/05 20:43:18 lb.data.data_utils.indexed_dataset]: creating numpy buffer of mmap... [07/05 20:43:18 lb.data.data_utils.indexed_dataset]: creating memory view of numpy buffer... [07/05 20:43:18 lb.data.data_utils.indexed_dataset]: Finished creating indexed dataset in 0.102189 seconds [07/05 20:43:18 lb.data.data_utils.indexed_dataset]: indexed dataset stats: [07/05 20:43:18 lb.data.data_utils.indexed_dataset]: number of documents: 50000 [07/05 20:43:18 lb.data.data_utils.indexed_dataset]: number of sentences: 1249934 [07/05 20:43:18 lb.data.data_utils.dataset_utils]:  > loading indexed mapping from /dataset/source/dataset/loss_compara_content_sentence_bert_indexmap_450560mns_509msl_0.10ssp_1234s.npy [07/05 20:43:18 lb.data.data_utils.dataset_utils]:  loaded indexed file in 0.004 seconds [07/05 20:43:18 lb.data.data_utils.dataset_utils]:  total number of samples: 452417 [07/05 20:43:18 lb.data.data_utils.dataset_utils]:  > loading indexed mapping from /dataset/source/dataset/loss_compara_content_sentence_bert_indexmap_8mns_509msl_0.10ssp_1234s.npy [07/05 20:43:18 lb.data.data_utils.dataset_utils]:  loaded indexed file in 0.001 seconds [07/05 20:43:18 lb.data.data_utils.dataset_utils]:  total number of samples: 5884 [07/05 20:43:18 lb.data.data_utils.dataset_utils]:  > loading indexed mapping from /dataset/source/dataset/loss_compara_content_sentence_bert_indexmap_8mns_509msl_0.10ssp_1234s.npy [07/05 20:43:18 lb.data.data_utils.dataset_utils]:  loaded indexed file in 0.001 seconds [07/05 20:43:18 lb.data.data_utils.dataset_utils]:  total number of samples: 5884 [07/05 20:43:21 lb.engine.default]: Auto-scaling the config to train.train_iter=220, train.warmup_iter=0 iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Bootstrap : Using eth0:192.168.11.230<0> iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol. iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol. iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO P2P plugin IBext iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.230<0> iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Using network IBext NCCL version 2.12.10+cuda11.2 iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO Bootstrap : Using eth0:192.168.11.230<0> iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Bootstrap : Using eth0:192.168.11.230<0> iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO Bootstrap : Using eth0:192.168.11.230<0> iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO Bootstrap : Using eth0:192.168.11.230<0> iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO Bootstrap : Using eth0:192.168.11.230<0> iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol. iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol. iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO P2P plugin IBext iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol. iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol. iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol. iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol. iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO P2P plugin IBext iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO P2P plugin IBext iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol. iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol. iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO P2P plugin IBext iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol. iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol. iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO P2P plugin IBext iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.230<0> iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO Using network IBext iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.230<0> iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO Using network IBext iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.230<0> iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Using network IBext iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.230<0> iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO Using network IBext iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.230<0> iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO Using network IBext iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO Bootstrap : Using eth0:192.168.11.230<0> iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol. iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol. iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO P2P plugin IBext iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.230<0> iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO Using network IBext iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO Bootstrap : Using eth0:192.168.11.230<0> iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol. iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol. iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO P2P plugin IBext iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.230<0> iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO Using network IBext iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO PXN Disabled as plugin is v4 iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO PXN Disabled as plugin is v4 iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO PXN Disabled as plugin is v4 iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO PXN Disabled as plugin is v4 iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO PXN Disabled as plugin is v4 iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO PXN Disabled as plugin is v4 iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO PXN Disabled as plugin is v4 iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO PXN Disabled as plugin is v4 iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Trees [0] 5/12/-1->4->-1 [1] 5/-1/-1->4->12 iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO Trees [0] -1/-1/-1->6->1 [1] -1/-1/-1->6->1 iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO Trees [0] 7/-1/-1->5->4 [1] 7/-1/-1->5->4 iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO Trees [0] 0/-1/-1->7->5 [1] 0/-1/-1->7->5 iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO Trees [0] 1/-1/-1->3->2 [1] 1/-1/-1->3->2 iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 3/-1/-1->2->0 iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO Trees [0] 6/-1/-1->1->3 [1] 6/-1/-1->1->3 iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Channel 00/02 : 0 1 3 2 5 12 14 15 8 9 11 10 13 4 6 7 iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Channel 01/02 : 0 1 3 2 5 12 14 15 8 9 11 10 13 4 6 7 iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Trees [0] 2/-1/-1->0->7 [1] 2/-1/-1->0->7 iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Channel 00 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO Channel 00 : 6[6b010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO Channel 00 : 2[67010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Channel 00 : 0[65010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Channel 01 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO Channel 01 : 6[6b010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO Channel 01 : 2[67010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Channel 01 : 0[65010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO Channel 00 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO Channel 01 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO Channel 00/0 : 5[69020] -> 12[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO Channel 00 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO Channel 01 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Channel 00/0 : 13[69020] -> 4[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO Channel 00 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO Channel 01 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO Channel 00 : 3[67020] -> 5[69020] via P2P/indirect/4[69010] iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Channel 00 : 0[65010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO Channel 00 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Channel 01 : 0[65010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO Channel 01 : 3[67020] -> 5[69020] via P2P/indirect/4[69010] iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO Channel 01 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO Channel 01/0 : 5[69020] -> 12[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Channel 01/0 : 13[69020] -> 4[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Channel 00 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Channel 01 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO Channel 00 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO Channel 01 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO Channel 00 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO Channel 01 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO Channel 00 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO Channel 00 : 2[67010] -> 4[69010] via P2P/indirect/5[69020] iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO Channel 01 : 2[67010] -> 4[69010] via P2P/indirect/5[69020] iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO Channel 01 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Channel 00 : 0[65010] -> 4[69010] via P2P/indirect/3[67020] iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Channel 01 : 0[65010] -> 4[69010] via P2P/indirect/3[67020] iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Channel 00/0 : 12[69010] -> 4[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Channel 01/0 : 12[69010] -> 4[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Channel 00/0 : 4[69010] -> 12[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Channel 01/0 : 4[69010] -> 12[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO Channel 00 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO Channel 00 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO Channel 01 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO Channel 01 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO Channel 00 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO Channel 01 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO Channel 00 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO Channel 00 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO Channel 01 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO Channel 01 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO Channel 00 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Channel 00 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Channel 01 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO Channel 01 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Channel 00 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Channel 01 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO Channel 00 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Channel 00 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO Channel 01 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Channel 01 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO Channel 00 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO Channel 01 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO Channel 00 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO Channel 01 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO Channel 00 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020] iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO Channel 01 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020] iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO Channel 00 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO Channel 00 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO Channel 01 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO Channel 01 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO Channel 00 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO Channel 00 : 5[69020] -> 3[67020] via P2P/indirect/2[67010] iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Channel 00 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO Channel 01 : 5[69020] -> 3[67020] via P2P/indirect/2[67010] iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO Channel 01 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Channel 01 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO Channel 00 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO Channel 01 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Channel 00 : 4[69010] -> 2[67010] via P2P/indirect/3[67020] iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO Channel 01 : 4[69010] -> 2[67010] via P2P/indirect/3[67020] iv-ybpu7pvmiu5m57lh5kdd:42285:43472 [4] NCCL INFO comm 0x7ff5dfb31680 rank 4 nranks 16 cudaDev 4 busId 69010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42283:43476 [2] NCCL INFO comm 0x7fdce98d3d20 rank 2 nranks 16 cudaDev 2 busId 67010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42286:43478 [5] NCCL INFO comm 0x7fe03560f9e0 rank 5 nranks 16 cudaDev 5 busId 69020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42288:43473 [7] NCCL INFO comm 0x7f2b883aa7d0 rank 7 nranks 16 cudaDev 7 busId 6b020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42287:43470 [6] NCCL INFO comm 0x7fe0ef595930 rank 6 nranks 16 cudaDev 6 busId 6b010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO comm 0x7f8113900790 rank 0 nranks 16 cudaDev 0 busId 65010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42284:43471 [3] NCCL INFO comm 0x7f8720ca58a0 rank 3 nranks 16 cudaDev 3 busId 67020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42282:43480 [1] NCCL INFO comm 0x7f2c008153e0 rank 1 nranks 16 cudaDev 1 busId 65020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42281:43475 [0] NCCL INFO Launch mode Parallel [07/05 20:43:36 lb.engine.default]: Model: BertForPreTraining( (bert): BertModel( (embeddings): BertEmbeddings( (vocab_embeddings): VocabEmbedding(num_embeddings=21504, embedding_dim=1024) (position_embeddings): Embedding(num_embeddings=512, embedding_dim=1024) (tokentype_embeddings): Embedding(num_embeddings=2, embedding_dim=1024) (embedding_dropout): Dropout(p=0.1, inplace=False) ) (extended_attn_mask): BertExtendedAttnMask() (encoders): ModuleList( (0): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (1): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (2): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (3): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (4): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (5): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (6): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (7): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (8): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (9): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (10): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (11): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (12): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (13): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (14): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (15): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (16): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (17): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (18): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (19): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (20): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (21): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (22): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) (23): TransformerLayer( (drop_path): Identity() (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (self_attention): MultiheadAttention( hidden_size=1024, num_heads=16, is_cross_attention=False (dropout): Dropout(p=0.1, inplace=False) (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col) (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row) ) (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): MLP( bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1 (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col) (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row) ) ) ) (final_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (pooler): BertPooler( (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=col) (activation_func): Tanh() ) ) (cls_head): BertPreTrainingHeads( (predictions): BertLMPredictionHead( (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=data) (activation_func): GELU() (layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) ) (seq_relationship): Linear1D(in_features=1024, out_features=2, bias=True, parallel=data) (lm_logits): LMLogits() (loss_func): BertLoss( (lm_loss): ParallelCrossEntropyLoss() ) ) ) WARNING [07/05 20:43:36 lb.scheduler.lr_scheduler]: warmup iters equals to zero, return CosineLR [07/05 20:43:37 lb.engine.trainer]: Starting training from iteration 0 [07/05 20:44:32 lb.models.utils.graph_base]: Start compling the train graph which may take some time. Please wait for a moment ... iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Trees [0] 6/-1/-1->1->3 [1] 6/-1/-1->1->3 iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 3/-1/-1->2->0 iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 00/02 : 0 1 3 2 5 12 14 15 8 9 11 10 13 4 6 7 iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 01/02 : 0 1 3 2 5 12 14 15 8 9 11 10 13 4 6 7 iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Trees [0] 2/-1/-1->0->7 [1] 2/-1/-1->0->7 iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Trees [0] 0/-1/-1->7->5 [1] 0/-1/-1->7->5 iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Trees [0] -1/-1/-1->6->1 [1] -1/-1/-1->6->1 iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Trees [0] 7/-1/-1->5->4 [1] 7/-1/-1->5->4 iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Trees [0] 5/12/-1->4->-1 [1] 5/-1/-1->4->12 iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Trees [0] 1/-1/-1->3->2 [1] 1/-1/-1->3->2 iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 00 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 00 : 6[6b010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 00 : 0[65010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 00 : 2[67010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 01 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 01 : 6[6b010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 01 : 0[65010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 01 : 2[67010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 00 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 01 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 00/0 : 5[69020] -> 12[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 00 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 01 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 00/0 : 13[69020] -> 4[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 00 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 01 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 00 : 0[65010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 01 : 0[65010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 00 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 00 : 3[67020] -> 5[69020] via P2P/indirect/4[69010] iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 01 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 01 : 3[67020] -> 5[69020] via P2P/indirect/4[69010] iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 01/0 : 5[69020] -> 12[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 01/0 : 13[69020] -> 4[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 00 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 01 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 00 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 01 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 00 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 01 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 00 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 00 : 2[67010] -> 4[69010] via P2P/indirect/5[69020] iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 01 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 00 : 0[65010] -> 4[69010] via P2P/indirect/3[67020] iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 01 : 2[67010] -> 4[69010] via P2P/indirect/5[69020] iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 01 : 0[65010] -> 4[69010] via P2P/indirect/3[67020] iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 00/0 : 12[69010] -> 4[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 01/0 : 12[69010] -> 4[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 00/0 : 4[69010] -> 12[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 01/0 : 4[69010] -> 12[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 00 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 00 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 01 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 01 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 00 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 00 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 01 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 01 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 00 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 00 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 01 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 00 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 01 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 01 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 00 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 01 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 00 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 00 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 01 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 01 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 00 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 01 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 00 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 00 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020] iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 01 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 01 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020] iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 00 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 00 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 01 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 01 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 00 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 00 : 5[69020] -> 3[67020] via P2P/indirect/2[67010] iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 00 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 01 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 01 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 01 : 5[69020] -> 3[67020] via P2P/indirect/2[67010] iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 00 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 01 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 00 : 4[69010] -> 2[67010] via P2P/indirect/3[67020] iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 01 : 4[69010] -> 2[67010] via P2P/indirect/3[67020] iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO comm 0x56347f801d00 rank 7 nranks 16 cudaDev 7 busId 6b020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO comm 0x55c46788cc30 rank 3 nranks 16 cudaDev 3 busId 67020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO comm 0x55dd11333890 rank 1 nranks 16 cudaDev 1 busId 65020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO comm 0x55b88ffe3130 rank 5 nranks 16 cudaDev 5 busId 69020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO comm 0x56521cee47e0 rank 4 nranks 16 cudaDev 4 busId 69010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO comm 0x55fe38031c70 rank 0 nranks 16 cudaDev 0 busId 65010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO comm 0x55f6a6503a10 rank 2 nranks 16 cudaDev 2 busId 67010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO comm 0x557f31896bf0 rank 6 nranks 16 cudaDev 6 busId 6b010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 3/-1/-1->2->0 iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Trees [0] 0/-1/-1->7->5 [1] 0/-1/-1->7->5 iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Trees [0] -1/-1/-1->6->1 [1] -1/-1/-1->6->1 iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Trees [0] 7/-1/-1->5->4 [1] 7/-1/-1->5->4 iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Trees [0] 5/12/-1->4->-1 [1] 5/-1/-1->4->12 iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Trees [0] 1/-1/-1->3->2 [1] 1/-1/-1->3->2 iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Trees [0] 6/-1/-1->1->3 [1] 6/-1/-1->1->3 iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 00/02 : 0 1 3 2 5 12 14 15 8 9 11 10 13 4 6 7 iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 01/02 : 0 1 3 2 5 12 14 15 8 9 11 10 13 4 6 7 iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Trees [0] 2/-1/-1->0->7 [1] 2/-1/-1->0->7 iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 00 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 00 : 6[6b010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 00 : 0[65010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 00 : 2[67010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 01 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 01 : 6[6b010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 01 : 0[65010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 01 : 2[67010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 00 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 01 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 00/0 : 5[69020] -> 12[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 00 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 01 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 00/0 : 13[69020] -> 4[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 00 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 01 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 00 : 3[67020] -> 5[69020] via P2P/indirect/4[69010] iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 00 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 00 : 0[65010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 01 : 3[67020] -> 5[69020] via P2P/indirect/4[69010] iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 01 : 0[65010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 01 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 01/0 : 5[69020] -> 12[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 01/0 : 13[69020] -> 4[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 00 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 01 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 00 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 01 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 00 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 01 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 00 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 01 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 00 : 0[65010] -> 4[69010] via P2P/indirect/3[67020] iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 00 : 2[67010] -> 4[69010] via P2P/indirect/5[69020] iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 01 : 0[65010] -> 4[69010] via P2P/indirect/3[67020] iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 01 : 2[67010] -> 4[69010] via P2P/indirect/5[69020] iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 00/0 : 12[69010] -> 4[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 01/0 : 12[69010] -> 4[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 00/0 : 4[69010] -> 12[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 01/0 : 4[69010] -> 12[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 00 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 00 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 01 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 01 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 00 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 00 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO Channel 01 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 01 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 00 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 00 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 00 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 01 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 00 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO Channel 01 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO Channel 01 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 01 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 00 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 00 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 00 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 01 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO Channel 01 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 01 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 00 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 00 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 01 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 01 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 00 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020] iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 00 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO Channel 01 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 01 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020] iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 00 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 00 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 01 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO Channel 01 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 00 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 00 : 5[69020] -> 3[67020] via P2P/indirect/2[67010] iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 01 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO Channel 01 : 5[69020] -> 3[67020] via P2P/indirect/2[67010] iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 00 : 4[69010] -> 2[67010] via P2P/indirect/3[67020] iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO Channel 01 : 4[69010] -> 2[67010] via P2P/indirect/3[67020] iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO comm 0x55f6a67d48f0 rank 2 nranks 16 cudaDev 2 busId 67010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO comm 0x557f31b670f0 rank 6 nranks 16 cudaDev 6 busId 6b010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO comm 0x56521d1d0600 rank 4 nranks 16 cudaDev 4 busId 69010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO comm 0x55c467b5d260 rank 3 nranks 16 cudaDev 3 busId 67020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO comm 0x56347fad6000 rank 7 nranks 16 cudaDev 7 busId 6b020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO comm 0x55fe38304fa0 rank 0 nranks 16 cudaDev 0 busId 65010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO comm 0x55b8902c0df0 rank 5 nranks 16 cudaDev 5 busId 69020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO comm 0x55dd11604a20 rank 1 nranks 16 cudaDev 1 busId 65020 - Init COMPLETE NCCL version 2.12.10+cuda11.2 NCCL version 2.12.10+cuda11.2 NCCL version 2.12.10+cuda11.2 NCCL version 2.12.10+cuda11.2 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Launch mode Parallel NCCL version 2.12.10+cuda11.2 NCCL version 2.12.10+cuda11.2 NCCL version 2.12.10+cuda11.2 iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 00/0 : 1[67010] -> 0[67010] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 01/0 : 1[67010] -> 0[67010] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 00/0 : 0[67010] -> 1[67010] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 01/0 : 0[67010] -> 1[67010] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 00/0 : 1[69010] -> 0[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 01/0 : 1[69010] -> 0[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 00/0 : 0[69010] -> 1[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 01/0 : 0[69010] -> 1[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 00/0 : 1[65010] -> 0[65010] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 01/0 : 1[65010] -> 0[65010] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 00/0 : 0[65010] -> 1[65010] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 01/0 : 0[65010] -> 1[65010] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 00/0 : 1[65020] -> 0[65020] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 01/0 : 1[65020] -> 0[65020] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 00/0 : 0[65020] -> 1[65020] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 01/0 : 0[65020] -> 1[65020] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 00/0 : 1[69020] -> 0[69020] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 01/0 : 1[69020] -> 0[69020] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 00/0 : 0[69020] -> 1[69020] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 01/0 : 0[69020] -> 1[69020] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 00/0 : 1[6b010] -> 0[6b010] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 01/0 : 1[6b010] -> 0[6b010] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 00/0 : 0[6b010] -> 1[6b010] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 01/0 : 0[6b010] -> 1[6b010] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 00/0 : 1[67020] -> 0[67020] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 01/0 : 1[67020] -> 0[67020] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 00/0 : 0[67020] -> 1[67020] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 00/0 : 1[6b020] -> 0[6b020] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 01/0 : 0[67020] -> 1[67020] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 01/0 : 1[6b020] -> 0[6b020] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 00/0 : 0[6b020] -> 1[6b020] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 01/0 : 0[6b020] -> 1[6b020] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO comm 0x7fda26375c50 rank 0 nranks 2 cudaDev 2 busId 67010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Launch mode Parallel iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO comm 0x7f292637bc80 rank 0 nranks 2 cudaDev 1 busId 65020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Launch mode Parallel iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO comm 0x7f7ae278ff30 rank 0 nranks 2 cudaDev 0 busId 65010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Launch mode Parallel iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO comm 0x7fddfe3778a0 rank 0 nranks 2 cudaDev 6 busId 6b010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO comm 0x7f843e358cc0 rank 0 nranks 2 cudaDev 3 busId 67020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Launch mode Parallel iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Launch mode Parallel iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO comm 0x7f28be361230 rank 0 nranks 2 cudaDev 7 busId 6b020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Launch mode Parallel iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO comm 0x7ff2f23550d0 rank 0 nranks 2 cudaDev 4 busId 69010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Launch mode Parallel iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO comm 0x7fdd6237a660 rank 0 nranks 2 cudaDev 5 busId 69020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Launch mode Parallel iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 00/12 : 0 2 3 1 6 4 5 7 iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Trees [0] 6/-1/-1->1->3 [1] 6/-1/-1->1->3 [2] 3/-1/-1->1->6 [3] 3/-1/-1->1->6 [4] 2/-1/-1->1->0 [5] -1/-1/-1->1->2 [6] 6/-1/-1->1->3 [7] 6/-1/-1->1->3 [8] 3/-1/-1->1->6 [9] 3/-1/-1->1->6 [10] 2/-1/-1->1->0 [11] -1/-1/-1->1->2 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 01/12 : 0 2 3 1 6 4 5 7 iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Trees [0] 5/-1/-1->4->6 [1] 5/-1/-1->4->6 [2] 6/-1/-1->4->5 [3] 6/-1/-1->4->5 [4] 3/-1/-1->4->7 [5] 7/-1/-1->4->3 [6] 5/-1/-1->4->6 [7] 5/-1/-1->4->6 [8] 6/-1/-1->4->5 [9] 6/-1/-1->4->5 [10] 3/-1/-1->4->7 [11] 7/-1/-1->4->3 iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Trees [0] 1/-1/-1->3->2 [1] 1/-1/-1->3->2 [2] 2/-1/-1->3->1 [3] 2/-1/-1->3->1 [4] -1/-1/-1->3->4 [5] 4/-1/-1->3->0 [6] 1/-1/-1->3->2 [7] 1/-1/-1->3->2 [8] 2/-1/-1->3->1 [9] 2/-1/-1->3->1 [10] -1/-1/-1->3->4 [11] 4/-1/-1->3->0 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 02/12 : 0 7 5 4 6 1 3 2 iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 3/-1/-1->2->0 [2] -1/-1/-1->2->3 [3] -1/-1/-1->2->3 [4] 5/-1/-1->2->1 [5] 1/-1/-1->2->5 [6] 3/-1/-1->2->0 [7] 3/-1/-1->2->0 [8] -1/-1/-1->2->3 [9] -1/-1/-1->2->3 [10] 5/-1/-1->2->1 [11] 1/-1/-1->2->5 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 03/12 : 0 7 5 4 6 1 3 2 iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Trees [0] -1/-1/-1->7->5 [1] -1/-1/-1->7->5 [2] 5/-1/-1->7->0 [3] 5/-1/-1->7->0 [4] 4/-1/-1->7->6 [5] 6/-1/-1->7->4 [6] -1/-1/-1->7->5 [7] -1/-1/-1->7->5 [8] 5/-1/-1->7->0 [9] 5/-1/-1->7->0 [10] 4/-1/-1->7->6 [11] 6/-1/-1->7->4 iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Trees [0] 4/-1/-1->6->1 [1] 4/-1/-1->6->1 [2] 1/-1/-1->6->4 [3] 1/-1/-1->6->4 [4] 7/-1/-1->6->5 [5] 5/-1/-1->6->7 [6] 4/-1/-1->6->1 [7] 4/-1/-1->6->1 [8] 1/-1/-1->6->4 [9] 1/-1/-1->6->4 [10] 7/-1/-1->6->5 [11] 5/-1/-1->6->7 iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Trees [0] 7/-1/-1->5->4 [1] 7/-1/-1->5->4 [2] 4/-1/-1->5->7 [3] 4/-1/-1->5->7 [4] 6/-1/-1->5->2 [5] 2/-1/-1->5->6 [6] 7/-1/-1->5->4 [7] 7/-1/-1->5->4 [8] 4/-1/-1->5->7 [9] 4/-1/-1->5->7 [10] 6/-1/-1->5->2 [11] 2/-1/-1->5->6 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 04/12 : 0 1 2 5 6 7 4 3 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 05/12 : 0 3 4 7 6 5 2 1 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 06/12 : 0 2 3 1 6 4 5 7 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 07/12 : 0 2 3 1 6 4 5 7 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 08/12 : 0 7 5 4 6 1 3 2 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 09/12 : 0 7 5 4 6 1 3 2 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 10/12 : 0 1 2 5 6 7 4 3 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 11/12 : 0 3 4 7 6 5 2 1 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Trees [0] 2/-1/-1->0->-1 [1] 2/-1/-1->0->-1 [2] 7/-1/-1->0->-1 [3] 7/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 3/-1/-1->0->-1 [6] 2/-1/-1->0->-1 [7] 2/-1/-1->0->-1 [8] 7/-1/-1->0->-1 [9] 7/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 3/-1/-1->0->-1 iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 04 : 1[65020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 00 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 10 : 1[65020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 04 : 6[6b010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 00 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 01 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 10 : 6[6b010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 01 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 04 : 0[65010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 06 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 05 : 3[67020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 04 : 5[69020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 06 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 06 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 10 : 0[65010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 07 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 11 : 3[67020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 10 : 5[69020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 07 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 07 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 02 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 00 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 03 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 02 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 01 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 08 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 03 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 06 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 09 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 06 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 08 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 07 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 07 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 09 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 05 : 4[69010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 02 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 05 : 0[65010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 04 : 2[67010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 11 : 4[69010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 11 : 0[65010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 10 : 2[67010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 03 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 08 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 09 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 04 : 7[6b020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 04 : 3[67020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 05 : 5[69020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 10 : 7[6b020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 10 : 3[67020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 11 : 5[69020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 00 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 02 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 02 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 01 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 03 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 03 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 06 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 08 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 06 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 08 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 07 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 09 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 09 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 07 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 00 : 6[6b010] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 01 : 6[6b010] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 06 : 6[6b010] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 02 : 0[65010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 02 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 07 : 6[6b010] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 03 : 0[65010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 03 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 08 : 0[65010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 05 : 7[6b020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 05 : 2[67010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 08 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 09 : 0[65010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 11 : 7[6b020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 11 : 2[67010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 09 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 05 : 6[6b010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 05 : 1[65020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 02 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 11 : 6[6b010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 11 : 1[65020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 03 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 04 : 4[69010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 08 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 10 : 4[69010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 09 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 05 : 1[65020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 11 : 1[65020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 05 : 6[6b010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 11 : 6[6b010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 02 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 03 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 08 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 02 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 02 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 09 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 03 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 03 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 05 : 5[69020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 08 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 08 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 04 : 3[67020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 11 : 5[69020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 09 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 09 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 10 : 3[67020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 06 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 02 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 00 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 05 : 2[67010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 07 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 03 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 01 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 11 : 2[67010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 08 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 06 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 09 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 07 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 00 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 04 : 4[69010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 05 : 3[67020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 01 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 10 : 4[69010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 11 : 3[67020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 06 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 07 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 04 : 5[69020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 02 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 05 : 7[6b020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 10 : 5[69020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 03 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 11 : 7[6b020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 08 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 09 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 02 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 00 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 03 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 01 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 08 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 06 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 06 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 09 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 07 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 07 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 02 : 6[6b010] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 04 : 7[6b020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 03 : 6[6b010] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 10 : 7[6b020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 08 : 6[6b010] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 00 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 09 : 6[6b010] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 01 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 04 : 2[67010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 06 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 10 : 2[67010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 07 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 04 : 6[6b010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 04 : 1[65020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 10 : 6[6b010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 10 : 1[65020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 05 : 4[69010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 11 : 4[69010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 06 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 07 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 02 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 10 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 02 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 10 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 02 : 3[67020] -> 5[69020] via P2P/indirect/4[69010] iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 03 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 03 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 02 : 2[67010] -> 4[69010] via P2P/indirect/5[69020] iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 10 : 3[67020] -> 5[69020] via P2P/indirect/4[69010] iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 11 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 10 : 2[67010] -> 4[69010] via P2P/indirect/5[69020] iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 11 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 03 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 03 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 11 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 11 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 04 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 04 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 04 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020] iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 04 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 04 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 12 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 12 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 04 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 04 : 0[65010] -> 4[69010] via P2P/indirect/3[67020] iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 04 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 12 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020] iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO Channel 12 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO Channel 12 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 12 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 12 : 0[65010] -> 4[69010] via P2P/indirect/3[67020] iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 12 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 05 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 05 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 05 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 05 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO Channel 13 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 13 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010] iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 13 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020] iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO Channel 13 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 06 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 06 : 5[69020] -> 3[67020] via P2P/indirect/2[67010] iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 06 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 06 : 4[69010] -> 2[67010] via P2P/indirect/3[67020] iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Channel 14 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020] iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO Channel 14 : 5[69020] -> 3[67020] via P2P/indirect/2[67010] iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO Channel 14 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010] iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO Channel 14 : 4[69010] -> 2[67010] via P2P/indirect/3[67020] iv-ybpu7pvmiu5m57lh5kdd:42286:44815 [5] NCCL INFO comm 0x7fdd6241d5d0 rank 5 nranks 8 cudaDev 5 busId 69020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO comm 0x7f7ae286bf20 rank 0 nranks 8 cudaDev 0 busId 65010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42287:44835 [6] NCCL INFO comm 0x7fddfe430d70 rank 6 nranks 8 cudaDev 6 busId 6b010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42281:44598 [0] NCCL INFO Launch mode Parallel iv-ybpu7pvmiu5m57lh5kdd:42288:44905 [7] NCCL INFO comm 0x7f28be417510 rank 7 nranks 8 cudaDev 7 busId 6b020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42285:44830 [4] NCCL INFO comm 0x7ff2f24005d0 rank 4 nranks 8 cudaDev 4 busId 69010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42284:44799 [3] NCCL INFO comm 0x7f843e434f20 rank 3 nranks 8 cudaDev 3 busId 67020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42282:44811 [1] NCCL INFO comm 0x7f292644fe10 rank 1 nranks 8 cudaDev 1 busId 65020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42283:44668 [2] NCCL INFO comm 0x7fda26451f20 rank 2 nranks 8 cudaDev 2 busId 67010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42281:45025 [0] NCCL INFO Channel 00 : 0[65010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45024 [5] NCCL INFO Channel 00 : 5[69020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45025 [0] NCCL INFO Channel 01 : 0[65010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45024 [5] NCCL INFO Channel 01 : 5[69020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45026 [6] NCCL INFO Channel 00 : 6[6b010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45026 [6] NCCL INFO Channel 01 : 6[6b010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45027 [1] NCCL INFO Channel 00 : 1[65020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45028 [4] NCCL INFO Channel 00 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45030 [3] NCCL INFO Channel 00 : 3[67020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45029 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45027 [1] NCCL INFO Channel 01 : 1[65020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45028 [4] NCCL INFO Channel 01 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45029 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45030 [3] NCCL INFO Channel 01 : 3[67020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45025 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45025 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45024 [5] NCCL INFO Channel 00 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45027 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45028 [4] NCCL INFO Channel 00 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45024 [5] NCCL INFO Channel 01 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45027 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45028 [4] NCCL INFO Channel 01 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45029 [2] NCCL INFO Channel 00 : 2[67010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45025 [0] NCCL INFO Channel 00 : 0[65010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45027 [1] NCCL INFO Channel 00 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45029 [2] NCCL INFO Channel 01 : 2[67010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45025 [0] NCCL INFO Channel 01 : 0[65010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45027 [1] NCCL INFO Channel 01 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45028 [4] NCCL INFO Channel 00 : 4[69010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45028 [4] NCCL INFO Channel 01 : 4[69010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45030 [3] NCCL INFO Channel 00 : 3[67020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45025 [0] NCCL INFO Channel 00 : 0[65010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45030 [3] NCCL INFO Channel 01 : 3[67020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45025 [0] NCCL INFO Channel 01 : 0[65010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45024 [5] NCCL INFO Channel 00 : 5[69020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45026 [6] NCCL INFO Channel 00 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45024 [5] NCCL INFO Channel 01 : 5[69020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45026 [6] NCCL INFO Channel 01 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45029 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45026 [6] NCCL INFO Channel 00 : 6[6b010] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45031 [7] NCCL INFO Channel 00 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45029 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45026 [6] NCCL INFO Channel 01 : 6[6b010] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45031 [7] NCCL INFO Channel 01 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45031 [7] NCCL INFO Channel 00 : 7[6b020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45031 [7] NCCL INFO Channel 01 : 7[6b020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45030 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45031 [7] NCCL INFO Channel 00 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45030 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45031 [7] NCCL INFO Channel 01 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45031 [7] NCCL INFO Channel 00 : 7[6b020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45031 [7] NCCL INFO Channel 01 : 7[6b020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45029 [2] NCCL INFO Channel 00 : 2[67010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45027 [1] NCCL INFO Channel 00 : 1[65020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45024 [5] NCCL INFO Channel 00 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45026 [6] NCCL INFO Channel 00 : 6[6b010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45028 [4] NCCL INFO Channel 00 : 4[69010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45030 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45029 [2] NCCL INFO Channel 01 : 2[67010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45024 [5] NCCL INFO Channel 01 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45027 [1] NCCL INFO Channel 01 : 1[65020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45026 [6] NCCL INFO Channel 01 : 6[6b010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45028 [4] NCCL INFO Channel 01 : 4[69010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45030 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45032 [0] NCCL INFO Channel 01 : 0[65010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45033 [7] NCCL INFO Channel 01 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45032 [0] NCCL INFO Channel 09 : 0[65010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45033 [7] NCCL INFO Channel 09 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45034 [1] NCCL INFO Channel 01 : 1[65020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45035 [5] NCCL INFO Channel 01 : 5[69020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45036 [4] NCCL INFO Channel 01 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45037 [6] NCCL INFO Channel 01 : 6[6b010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45034 [1] NCCL INFO Channel 09 : 1[65020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45038 [3] NCCL INFO Channel 01 : 3[67020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45039 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45035 [5] NCCL INFO Channel 09 : 5[69020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45038 [3] NCCL INFO Channel 09 : 3[67020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45036 [4] NCCL INFO Channel 09 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45039 [2] NCCL INFO Channel 09 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45037 [6] NCCL INFO Channel 09 : 6[6b010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45032 [0] NCCL INFO Channel 02 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45036 [4] NCCL INFO Channel 02 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45032 [0] NCCL INFO Channel 10 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45036 [4] NCCL INFO Channel 10 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45035 [5] NCCL INFO Channel 02 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45034 [1] NCCL INFO Channel 02 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45035 [5] NCCL INFO Channel 10 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45034 [1] NCCL INFO Channel 10 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45032 [0] NCCL INFO Channel 03 : 0[65010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45039 [2] NCCL INFO Channel 03 : 2[67010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45037 [6] NCCL INFO Channel 03 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45036 [4] NCCL INFO Channel 03 : 4[69010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45032 [0] NCCL INFO Channel 11 : 0[65010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45039 [2] NCCL INFO Channel 11 : 2[67010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45037 [6] NCCL INFO Channel 11 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45036 [4] NCCL INFO Channel 11 : 4[69010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45034 [1] NCCL INFO Channel 05 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45035 [5] NCCL INFO Channel 05 : 5[69020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45038 [3] NCCL INFO Channel 05 : 3[67020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45033 [7] NCCL INFO Channel 05 : 7[6b020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45034 [1] NCCL INFO Channel 13 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45035 [5] NCCL INFO Channel 13 : 5[69020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45038 [3] NCCL INFO Channel 13 : 3[67020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45033 [7] NCCL INFO Channel 13 : 7[6b020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45039 [2] NCCL INFO Channel 06 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45037 [6] NCCL INFO Channel 06 : 6[6b010] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45038 [3] NCCL INFO Channel 06 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45033 [7] NCCL INFO Channel 06 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45039 [2] NCCL INFO Channel 14 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45037 [6] NCCL INFO Channel 14 : 6[6b010] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45038 [3] NCCL INFO Channel 14 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45033 [7] NCCL INFO Channel 14 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45038 [3] NCCL INFO Channel 07 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45034 [1] NCCL INFO Channel 07 : 1[65020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45039 [2] NCCL INFO Channel 07 : 2[67010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45035 [5] NCCL INFO Channel 07 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45032 [0] NCCL INFO Channel 07 : 0[65010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45033 [7] NCCL INFO Channel 07 : 7[6b020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45037 [6] NCCL INFO Channel 07 : 6[6b010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45036 [4] NCCL INFO Channel 07 : 4[69010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45038 [3] NCCL INFO Channel 15 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45034 [1] NCCL INFO Channel 15 : 1[65020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45039 [2] NCCL INFO Channel 15 : 2[67010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45035 [5] NCCL INFO Channel 15 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45032 [0] NCCL INFO Channel 15 : 0[65010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45033 [7] NCCL INFO Channel 15 : 7[6b020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45037 [6] NCCL INFO Channel 15 : 6[6b010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45036 [4] NCCL INFO Channel 15 : 4[69010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:44648 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42286:44858 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42285:44870 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42287:44724 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42284:44734 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42282:44848 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42281:44607 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-ybpu7pvmiu5m57lh5kdd:42288:44899 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-ybpu7pvmiu5m57lh5kdd:42283:44648 [2] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42283:44648 [2] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42283:44648 [2] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 iv-ybpu7pvmiu5m57lh5kdd:42286:44858 [5] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42286:44858 [5] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42286:44858 [5] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 iv-ybpu7pvmiu5m57lh5kdd:42285:44870 [4] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42285:44870 [4] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42285:44870 [4] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 iv-ybpu7pvmiu5m57lh5kdd:42284:44734 [3] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42284:44734 [3] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42284:44734 [3] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 iv-ybpu7pvmiu5m57lh5kdd:42287:44724 [6] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42287:44724 [6] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42287:44724 [6] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 iv-ybpu7pvmiu5m57lh5kdd:42283:44648 [2] NCCL INFO Channel 00/0 : 1[67010] -> 0[67010] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42282:44848 [1] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42282:44848 [1] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42282:44848 [1] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 iv-ybpu7pvmiu5m57lh5kdd:42283:44648 [2] NCCL INFO Channel 01/0 : 1[67010] -> 0[67010] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42283:44648 [2] NCCL INFO Channel 00/0 : 0[67010] -> 1[67010] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42283:44648 [2] NCCL INFO Channel 01/0 : 0[67010] -> 1[67010] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42281:44607 [0] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42281:44607 [0] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42281:44607 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 iv-ybpu7pvmiu5m57lh5kdd:42288:44899 [7] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42288:44899 [7] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmiu5m57lh5kdd:42288:44899 [7] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 iv-ybpu7pvmiu5m57lh5kdd:42286:44858 [5] NCCL INFO Channel 00/0 : 1[69020] -> 0[69020] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42286:44858 [5] NCCL INFO Channel 01/0 : 1[69020] -> 0[69020] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42286:44858 [5] NCCL INFO Channel 00/0 : 0[69020] -> 1[69020] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42286:44858 [5] NCCL INFO Channel 01/0 : 0[69020] -> 1[69020] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:44870 [4] NCCL INFO Channel 00/0 : 1[69010] -> 0[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:44870 [4] NCCL INFO Channel 01/0 : 1[69010] -> 0[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:44870 [4] NCCL INFO Channel 00/0 : 0[69010] -> 1[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42285:44870 [4] NCCL INFO Channel 01/0 : 0[69010] -> 1[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmiu5m57lh5kdd:42284:44734 [3] NCCL INFO Channel 00/0 : 1[67020] -> 0[67020] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42284:44734 [3] NCCL INFO Channel 01/0 : 1[67020] -> 0[67020] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42284:44734 [3] NCCL INFO Channel 00/0 : 0[67020] -> 1[67020] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42287:44724 [6] NCCL INFO Channel 00/0 : 1[6b010] -> 0[6b010] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42284:44734 [3] NCCL INFO Channel 01/0 : 0[67020] -> 1[67020] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42287:44724 [6] NCCL INFO Channel 01/0 : 1[6b010] -> 0[6b010] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42287:44724 [6] NCCL INFO Channel 00/0 : 0[6b010] -> 1[6b010] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42287:44724 [6] NCCL INFO Channel 01/0 : 0[6b010] -> 1[6b010] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42282:44848 [1] NCCL INFO Channel 00/0 : 1[65020] -> 0[65020] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42282:44848 [1] NCCL INFO Channel 01/0 : 1[65020] -> 0[65020] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42282:44848 [1] NCCL INFO Channel 00/0 : 0[65020] -> 1[65020] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42282:44848 [1] NCCL INFO Channel 01/0 : 0[65020] -> 1[65020] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42281:44607 [0] NCCL INFO Channel 00/0 : 1[65010] -> 0[65010] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42281:44607 [0] NCCL INFO Channel 01/0 : 1[65010] -> 0[65010] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42281:44607 [0] NCCL INFO Channel 00/0 : 0[65010] -> 1[65010] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42281:44607 [0] NCCL INFO Channel 01/0 : 0[65010] -> 1[65010] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42288:44899 [7] NCCL INFO Channel 00/0 : 1[6b020] -> 0[6b020] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42288:44899 [7] NCCL INFO Channel 01/0 : 1[6b020] -> 0[6b020] [receive] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42288:44899 [7] NCCL INFO Channel 00/0 : 0[6b020] -> 1[6b020] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42288:44899 [7] NCCL INFO Channel 01/0 : 0[6b020] -> 1[6b020] [send] via NET/IBext/0 iv-ybpu7pvmiu5m57lh5kdd:42283:44648 [2] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42283:44648 [2] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42283:44648 [2] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42283:44648 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42281:44607 [0] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42281:44607 [0] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42281:44607 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42281:44607 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42287:44724 [6] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42287:44724 [6] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42287:44724 [6] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42287:44724 [6] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42284:44734 [3] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42284:44734 [3] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42284:44734 [3] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42284:44734 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42282:44848 [1] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42282:44848 [1] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42282:44848 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42282:44848 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42288:44899 [7] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42288:44899 [7] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42288:44899 [7] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42288:44899 [7] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42281:44607 [0] NCCL INFO comm 0x7f7aeee9df20 rank 0 nranks 2 cudaDev 0 busId 65010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42283:44648 [2] NCCL INFO comm 0x7fda32a8ef00 rank 0 nranks 2 cudaDev 2 busId 67010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42281:44607 [0] NCCL INFO Launch mode Parallel iv-ybpu7pvmiu5m57lh5kdd:42283:44648 [2] NCCL INFO Launch mode Parallel iv-ybpu7pvmiu5m57lh5kdd:42287:44724 [6] NCCL INFO comm 0x7fde0aa88cd0 rank 0 nranks 2 cudaDev 6 busId 6b010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42287:44724 [6] NCCL INFO Launch mode Parallel iv-ybpu7pvmiu5m57lh5kdd:42281:44607 [0] NCCL INFO Launch mode Parallel iv-ybpu7pvmiu5m57lh5kdd:42284:44734 [3] NCCL INFO comm 0x7f844aab2200 rank 0 nranks 2 cudaDev 3 busId 67020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42282:44848 [1] NCCL INFO comm 0x7f2922a67150 rank 0 nranks 2 cudaDev 1 busId 65020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42284:44734 [3] NCCL INFO Launch mode Parallel iv-ybpu7pvmiu5m57lh5kdd:42282:44848 [1] NCCL INFO Launch mode Parallel iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00/1 : 15[6b020] -> 0[65010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:44899 [7] NCCL INFO comm 0x7f28baad7e40 rank 0 nranks 2 cudaDev 7 busId 6b020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42288:44899 [7] NCCL INFO Launch mode Parallel iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01/1 : 15[6b020] -> 0[65010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00 : 0[65010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01 : 0[65010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00 : 6[6b010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00 : 3[67020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00 : 1[65020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00/1 : 7[6b020] -> 8[65010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01/1 : 7[6b020] -> 8[65010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01 : 3[67020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01 : 1[65020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01 : 6[6b010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00/1 : 15[6b020] -> 1[65020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01/1 : 15[6b020] -> 1[65020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00/1 : 14[6b010] -> 0[65010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01/1 : 14[6b010] -> 0[65010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00/1 : 7[6b020] -> 9[65020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01/1 : 7[6b020] -> 9[65020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00/1 : 15[6b020] -> 2[67010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01/1 : 15[6b020] -> 2[67010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00 : 2[67010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44858 [5] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42286:44858 [5] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42286:44858 [5] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42286:44858 [5] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01 : 2[67010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:44858 [5] NCCL INFO comm 0x7fdd6ea7aaa0 rank 0 nranks 2 cudaDev 5 busId 69020 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42286:44858 [5] NCCL INFO Launch mode Parallel iv-ybpu7pvmiu5m57lh5kdd:42285:44870 [4] NCCL INFO Connected all rings iv-ybpu7pvmiu5m57lh5kdd:42285:44870 [4] NCCL INFO Connected all trees iv-ybpu7pvmiu5m57lh5kdd:42285:44870 [4] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmiu5m57lh5kdd:42285:44870 [4] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00 : 5[69020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01 : 5[69020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00/1 : 6[6b010] -> 8[65010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01/1 : 6[6b010] -> 8[65010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00/1 : 13[69020] -> 0[65010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01/1 : 13[69020] -> 0[65010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00 : 0[65010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01 : 0[65010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:44870 [4] NCCL INFO comm 0x7ff2eea9e040 rank 0 nranks 2 cudaDev 4 busId 69010 - Init COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42285:44870 [4] NCCL INFO Launch mode Parallel iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01 : 4[69010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01 : 5[69020] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01 : 4[69010] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00 : 4[69010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01 : 4[69010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00/1 : 5[69020] -> 8[65010] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01/1 : 5[69020] -> 8[65010] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00/1 : 7[6b020] -> 10[67010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00/1 : 15[6b020] -> 3[67020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01/1 : 7[6b020] -> 10[67010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01/1 : 15[6b020] -> 3[67020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00/1 : 4[69010] -> 8[65010] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01/1 : 4[69010] -> 8[65010] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00/1 : 14[6b010] -> 1[65020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01/1 : 14[6b010] -> 1[65020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00/1 : 6[6b010] -> 9[65020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01/1 : 6[6b010] -> 9[65020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00/1 : 12[69010] -> 0[65010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01/1 : 12[69010] -> 0[65010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00/1 : 13[69020] -> 1[65020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01/1 : 13[69020] -> 1[65020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00/1 : 6[6b010] -> 10[67010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01/1 : 6[6b010] -> 10[67010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00/1 : 14[6b010] -> 2[67010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00/1 : 14[6b010] -> 3[67020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01/1 : 14[6b010] -> 3[67020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00/1 : 3[67020] -> 8[65010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01/1 : 3[67020] -> 8[65010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00/1 : 7[6b020] -> 11[67020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01/1 : 7[6b020] -> 11[67020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00/1 : 11[67020] -> 0[65010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01/1 : 11[67020] -> 0[65010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01/1 : 14[6b010] -> 2[67010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00/1 : 5[69020] -> 9[65020] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00/1 : 6[6b010] -> 11[67020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01/1 : 6[6b010] -> 11[67020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01/1 : 5[69020] -> 9[65020] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00/1 : 7[6b020] -> 12[69010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01/1 : 7[6b020] -> 12[69010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00/1 : 13[69020] -> 2[67010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01/1 : 13[69020] -> 2[67010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00/1 : 12[69010] -> 1[65020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01/1 : 12[69010] -> 1[65020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00/1 : 7[6b020] -> 13[69020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01/1 : 7[6b020] -> 13[69020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00/1 : 10[67010] -> 0[65010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01/1 : 10[67010] -> 0[65010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01 : 1[65020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00/1 : 15[6b020] -> 4[69010] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01/1 : 15[6b020] -> 4[69010] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00/1 : 4[69010] -> 9[65020] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01/1 : 4[69010] -> 9[65020] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00/1 : 12[69010] -> 2[67010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01/1 : 12[69010] -> 2[67010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00/1 : 2[67010] -> 8[65010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01/1 : 2[67010] -> 8[65010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00/1 : 13[69020] -> 3[67020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01/1 : 13[69020] -> 3[67020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00/1 : 3[67020] -> 9[65020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01/1 : 3[67020] -> 9[65020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00/1 : 5[69020] -> 10[67010] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01/1 : 5[69020] -> 10[67010] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00/1 : 6[6b010] -> 12[69010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01/1 : 6[6b010] -> 12[69010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00/1 : 11[67020] -> 1[65020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01/1 : 11[67020] -> 1[65020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00/1 : 12[69010] -> 3[67020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01/1 : 12[69010] -> 3[67020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00/1 : 3[67020] -> 10[67010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01/1 : 3[67020] -> 10[67010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00/1 : 7[6b020] -> 14[6b010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01/1 : 7[6b020] -> 14[6b010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00/1 : 15[6b020] -> 5[69020] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01/1 : 15[6b020] -> 5[69020] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00/1 : 5[69020] -> 11[67020] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01/1 : 5[69020] -> 11[67020] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00/1 : 14[6b010] -> 4[69010] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01/1 : 14[6b010] -> 4[69010] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00/1 : 4[69010] -> 10[67010] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01/1 : 4[69010] -> 10[67010] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00/1 : 10[67010] -> 1[65020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01/1 : 10[67010] -> 1[65020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00/1 : 1[65020] -> 8[65010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01/1 : 1[65020] -> 8[65010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00/1 : 9[65020] -> 0[65010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01/1 : 9[65020] -> 0[65010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00 : 0[65010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00/1 : 15[6b020] -> 6[6b010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01/1 : 15[6b020] -> 6[6b010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00/1 : 6[6b010] -> 13[69020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01/1 : 6[6b010] -> 13[69020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00/1 : 11[67020] -> 2[67010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01/1 : 11[67020] -> 2[67010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00/1 : 2[67010] -> 9[65020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01/1 : 2[67010] -> 9[65020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01 : 0[65010] -> 7[6b020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00/1 : 14[6b010] -> 6[6b010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01/1 : 14[6b010] -> 6[6b010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00/1 : 6[6b010] -> 14[6b010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01/1 : 6[6b010] -> 14[6b010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00/1 : 9[65020] -> 1[65020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01/1 : 9[65020] -> 1[65020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00/1 : 1[65020] -> 9[65020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01/1 : 1[65020] -> 9[65020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00/1 : 15[6b020] -> 7[6b020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01/1 : 15[6b020] -> 7[6b020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00/1 : 7[6b020] -> 15[6b020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01/1 : 7[6b020] -> 15[6b020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00/1 : 8[65010] -> 1[65020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01/1 : 8[65010] -> 1[65020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00/1 : 1[65020] -> 10[67010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01/1 : 1[65020] -> 10[67010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00/1 : 8[65010] -> 0[65010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01/1 : 8[65010] -> 0[65010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00/1 : 0[65010] -> 8[65010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01/1 : 0[65010] -> 8[65010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00/1 : 14[6b010] -> 7[6b020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01/1 : 14[6b010] -> 7[6b020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01 : 7[6b020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00/1 : 14[6b010] -> 5[69020] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01/1 : 14[6b010] -> 5[69020] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00/1 : 5[69020] -> 12[69010] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01/1 : 5[69020] -> 12[69010] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00/1 : 13[69020] -> 4[69010] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01/1 : 13[69020] -> 4[69010] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00/1 : 4[69010] -> 11[67020] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01/1 : 4[69010] -> 11[67020] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00/1 : 0[65010] -> 9[65020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01/1 : 0[65010] -> 9[65020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00/1 : 10[67010] -> 2[67010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01/1 : 10[67010] -> 2[67010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00/1 : 2[67010] -> 10[67010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01/1 : 2[67010] -> 10[67010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00/1 : 11[67020] -> 3[67020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01/1 : 11[67020] -> 3[67020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00/1 : 3[67020] -> 11[67020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01/1 : 3[67020] -> 11[67020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00/1 : 9[65020] -> 2[67010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01/1 : 9[65020] -> 2[67010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00/1 : 2[67010] -> 11[67020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01/1 : 2[67010] -> 11[67020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00/1 : 13[69020] -> 6[6b010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01/1 : 13[69020] -> 6[6b010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00/1 : 6[6b010] -> 15[6b020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01/1 : 6[6b010] -> 15[6b020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00/1 : 0[65010] -> 10[67010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01/1 : 0[65010] -> 10[67010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00/1 : 1[65020] -> 11[67020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01/1 : 1[65020] -> 11[67020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00/1 : 10[67010] -> 3[67020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01/1 : 10[67010] -> 3[67020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00/1 : 3[67020] -> 12[69010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01/1 : 3[67020] -> 12[69010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00/1 : 8[65010] -> 2[67010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01/1 : 8[65010] -> 2[67010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00/1 : 2[67010] -> 12[69010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01/1 : 2[67010] -> 12[69010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00/1 : 0[65010] -> 11[67020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01/1 : 0[65010] -> 11[67020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00/1 : 13[69020] -> 5[69020] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01/1 : 13[69020] -> 5[69020] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00/1 : 5[69020] -> 13[69020] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01/1 : 5[69020] -> 13[69020] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00/1 : 12[69010] -> 4[69010] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01/1 : 12[69010] -> 4[69010] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00/1 : 4[69010] -> 12[69010] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01/1 : 4[69010] -> 12[69010] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00/1 : 12[69010] -> 5[69020] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01/1 : 12[69010] -> 5[69020] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00/1 : 5[69020] -> 14[6b010] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01/1 : 5[69020] -> 14[6b010] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00/1 : 9[65020] -> 3[67020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01/1 : 9[65020] -> 3[67020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00/1 : 3[67020] -> 13[69020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01/1 : 3[67020] -> 13[69020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00/1 : 11[67020] -> 4[69010] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01/1 : 11[67020] -> 4[69010] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00/1 : 4[69010] -> 13[69020] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01/1 : 4[69010] -> 13[69020] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00/1 : 13[69020] -> 7[6b020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01/1 : 13[69020] -> 7[6b020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00/1 : 12[69010] -> 6[6b010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01/1 : 12[69010] -> 6[6b010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00/1 : 1[65020] -> 12[69010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01/1 : 1[65020] -> 12[69010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00/1 : 11[67020] -> 5[69020] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01/1 : 11[67020] -> 5[69020] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00/1 : 5[69020] -> 15[6b020] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01/1 : 5[69020] -> 15[6b020] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00/1 : 2[67010] -> 13[69020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01/1 : 2[67010] -> 13[69020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00/1 : 10[67010] -> 4[69010] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01/1 : 10[67010] -> 4[69010] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00/1 : 4[69010] -> 14[6b010] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01/1 : 4[69010] -> 14[6b010] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00/1 : 8[65010] -> 3[67020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01/1 : 8[65010] -> 3[67020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00/1 : 3[67020] -> 14[6b010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01/1 : 3[67020] -> 14[6b010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00/1 : 11[67020] -> 6[6b010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01/1 : 11[67020] -> 6[6b010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00/1 : 0[65010] -> 12[69010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01/1 : 0[65010] -> 12[69010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01 : 6[6b010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00/1 : 12[69010] -> 7[6b020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01/1 : 12[69010] -> 7[6b020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00/1 : 10[67010] -> 5[69020] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01/1 : 10[67010] -> 5[69020] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00/1 : 10[67010] -> 6[6b010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01/1 : 10[67010] -> 6[6b010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00/1 : 3[67020] -> 15[6b020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01/1 : 3[67020] -> 15[6b020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00/1 : 9[65020] -> 4[69010] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01/1 : 9[65020] -> 4[69010] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00/1 : 4[69010] -> 15[6b020] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01/1 : 4[69010] -> 15[6b020] [send] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00/1 : 9[65020] -> 6[6b010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01/1 : 9[65020] -> 6[6b010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00/1 : 11[67020] -> 7[6b020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01/1 : 11[67020] -> 7[6b020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00/1 : 2[67010] -> 14[6b010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01/1 : 2[67010] -> 14[6b010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00/1 : 1[65020] -> 13[69020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01/1 : 1[65020] -> 13[69020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00 : 3[67020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01 : 3[67020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00/1 : 9[65020] -> 5[69020] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01/1 : 9[65020] -> 5[69020] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00/1 : 10[67010] -> 7[6b020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01/1 : 10[67010] -> 7[6b020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00 : 7[6b020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00/1 : 2[67010] -> 15[6b020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01/1 : 2[67010] -> 15[6b020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01 : 7[6b020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00/1 : 8[65010] -> 6[6b010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01/1 : 8[65010] -> 6[6b010] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00 : 6[6b010] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01 : 6[6b010] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00/1 : 8[65010] -> 4[69010] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01/1 : 8[65010] -> 4[69010] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00/1 : 8[65010] -> 5[69020] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01/1 : 8[65010] -> 5[69020] [receive] via NET/IBext/0/GDRDMA/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00 : 5[69020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00/1 : 1[65020] -> 14[6b010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01/1 : 1[65020] -> 14[6b010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00/1 : 0[65010] -> 13[69020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01/1 : 0[65010] -> 13[69020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01 : 5[69020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00/1 : 1[65020] -> 15[6b020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01/1 : 1[65020] -> 15[6b020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42284:45094 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 00 : 4[69010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42285:45097 [4] NCCL INFO Channel 01 : 4[69010] -> 3[67020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00/1 : 9[65020] -> 7[6b020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01/1 : 9[65020] -> 7[6b020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01 : 7[6b020] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 00 : 1[65020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42282:45093 [1] NCCL INFO Channel 01 : 1[65020] -> 0[65010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 00 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42286:45096 [5] NCCL INFO Channel 01 : 5[69020] -> 4[69010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00/1 : 0[65010] -> 14[6b010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01/1 : 0[65010] -> 14[6b010] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 00 : 2[67010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42283:45091 [2] NCCL INFO Channel 01 : 2[67010] -> 1[65020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 00 : 6[6b010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42287:45092 [6] NCCL INFO Channel 01 : 6[6b010] -> 5[69020] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00/1 : 8[65010] -> 7[6b020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01/1 : 8[65010] -> 7[6b020] [receive] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 00 : 7[6b020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42288:45095 [7] NCCL INFO Channel 01 : 7[6b020] -> 6[6b010] via P2P/IPC iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 00/1 : 0[65010] -> 15[6b020] [send] via NET/IBext/0/Shared iv-ybpu7pvmiu5m57lh5kdd:42281:45090 [0] NCCL INFO Channel 01/1 : 0[65010] -> 15[6b020] [send] via NET/IBext/0/Shared timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/07/05 21:01:54.355, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 77 %, 32510 MiB, 23110 MiB, 9400 MiB 2022/07/05 21:01:54.358, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23154 MiB, 9356 MiB 2022/07/05 21:01:54.361, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23510 MiB, 9000 MiB 2022/07/05 21:01:54.363, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23406 MiB, 9104 MiB 2022/07/05 21:01:54.364, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23274 MiB, 9236 MiB timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/07/05 21:01:54.366, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23178 MiB, 9332 MiB timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/07/05 21:01:54.368, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 77 %, 32510 MiB, 23110 MiB, 9400 MiB 2022/07/05 21:01:54.369, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23098 MiB, 9412 MiB 2022/07/05 21:01:54.369, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 77 %, 32510 MiB, 23110 MiB, 9400 MiB timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/07/05 21:01:54.372, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23154 MiB, 9356 MiB 2022/07/05 21:01:54.372, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23174 MiB, 9336 MiB 2022/07/05 21:01:54.373, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23154 MiB, 9356 MiB 2022/07/05 21:01:54.373, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 77 %, 32510 MiB, 23110 MiB, 9400 MiB timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/07/05 21:01:54.375, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23510 MiB, 9000 MiB 2022/07/05 21:01:54.375, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 77 %, 32510 MiB, 23110 MiB, 9400 MiB 2022/07/05 21:01:54.376, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23510 MiB, 9000 MiB 2022/07/05 21:01:54.377, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23154 MiB, 9356 MiB 2022/07/05 21:01:54.378, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 77 %, 32510 MiB, 23110 MiB, 9400 MiB 2022/07/05 21:01:54.380, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23406 MiB, 9104 MiB 2022/07/05 21:01:54.380, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23154 MiB, 9356 MiB 2022/07/05 21:01:54.381, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23406 MiB, 9104 MiB 2022/07/05 21:01:54.382, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23510 MiB, 9000 MiB 2022/07/05 21:01:54.384, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23154 MiB, 9356 MiB 2022/07/05 21:01:54.386, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23274 MiB, 9236 MiB 2022/07/05 21:01:54.386, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23510 MiB, 9000 MiB 2022/07/05 21:01:54.387, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23274 MiB, 9236 MiB timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/07/05 21:01:54.387, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23406 MiB, 9104 MiB 2022/07/05 21:01:54.389, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23510 MiB, 9000 MiB 2022/07/05 21:01:54.390, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23178 MiB, 9332 MiB 2022/07/05 21:01:54.390, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23406 MiB, 9104 MiB 2022/07/05 21:01:54.391, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23178 MiB, 9332 MiB 2022/07/05 21:01:54.391, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 74 %, 32510 MiB, 23110 MiB, 9400 MiB 2022/07/05 21:01:54.392, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23274 MiB, 9236 MiB 2022/07/05 21:01:54.394, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23406 MiB, 9104 MiB 2022/07/05 21:01:54.398, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23098 MiB, 9412 MiB 2022/07/05 21:01:54.398, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23274 MiB, 9236 MiB 2022/07/05 21:01:54.399, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23098 MiB, 9412 MiB 2022/07/05 21:01:54.400, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23154 MiB, 9356 MiB 2022/07/05 21:01:54.401, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23178 MiB, 9332 MiB 2022/07/05 21:01:54.403, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23274 MiB, 9236 MiB 2022/07/05 21:01:54.404, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 75 %, 32510 MiB, 23174 MiB, 9336 MiB 2022/07/05 21:01:54.404, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 75 %, 32510 MiB, 23178 MiB, 9332 MiB 2022/07/05 21:01:54.405, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 75 %, 32510 MiB, 23174 MiB, 9336 MiB 2022/07/05 21:01:54.406, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23510 MiB, 9000 MiB 2022/07/05 21:01:54.406, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23098 MiB, 9412 MiB 2022/07/05 21:01:54.409, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 75 %, 32510 MiB, 23178 MiB, 9332 MiB 2022/07/05 21:01:54.410, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23098 MiB, 9412 MiB 2022/07/05 21:01:54.412, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23406 MiB, 9104 MiB 2022/07/05 21:01:54.412, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 75 %, 32510 MiB, 23174 MiB, 9336 MiB 2022/07/05 21:01:54.414, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23098 MiB, 9412 MiB 2022/07/05 21:01:54.416, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 75 %, 32510 MiB, 23174 MiB, 9336 MiB 2022/07/05 21:01:54.417, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23274 MiB, 9236 MiB 2022/07/05 21:01:54.419, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 75 %, 32510 MiB, 23174 MiB, 9336 MiB 2022/07/05 21:01:54.423, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 75 %, 32510 MiB, 23178 MiB, 9332 MiB 2022/07/05 21:01:54.427, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23098 MiB, 9412 MiB 2022/07/05 21:01:54.436, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 75 %, 32510 MiB, 23174 MiB, 9336 MiB timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/07/05 21:02:02.104, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23110 MiB, 9400 MiB 2022/07/05 21:02:02.104, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23154 MiB, 9356 MiB 2022/07/05 21:02:02.105, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 75 %, 32510 MiB, 23510 MiB, 9000 MiB 2022/07/05 21:02:02.106, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 75 %, 32510 MiB, 23406 MiB, 9104 MiB 2022/07/05 21:02:02.106, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 74 %, 32510 MiB, 23274 MiB, 9236 MiB 2022/07/05 21:02:02.107, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23178 MiB, 9332 MiB 2022/07/05 21:02:02.108, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 76 %, 32510 MiB, 23098 MiB, 9412 MiB 2022/07/05 21:02:02.108, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 77 %, 32510 MiB, 23174 MiB, 9336 MiB [07/05 21:02:09 lb.utils.events]:  eta: 0:15:39 iteration: 99/220 consumed_samples: 204800 total_loss: 8.057 lm_loss: 7.359 sop_loss: 0.6977 time: 7.8261 s/iter data_time: 0.0296 s/iter total_throughput: 261.69 samples/s lr: 5.82e-05 [07/05 21:15:12 lb.utils.events]:  eta: 0:02:36 iteration: 199/220 consumed_samples: 409600 total_loss: 7.896 lm_loss: 7.202 sop_loss: 0.6938 time: 7.8244 s/iter data_time: 0.0346 s/iter total_throughput: 261.74 samples/s lr: 3.21e-06 [07/05 21:17:48 lb.utils.events]:  eta: 0:00:00 iteration: 219/220 consumed_samples: 450560 total_loss: 7.892 lm_loss: 7.199 sop_loss: 0.6935 time: 7.8235 s/iter data_time: 0.0701 s/iter total_throughput: 261.78 samples/s lr: 1.01e-06 [07/05 21:17:48 lb.engine.hooks]: Overall training speed: 218 iterations in 0:28:25 (7.8235 s / it) [07/05 21:17:48 lb.engine.hooks]: Total training time: 0:28:25 (0:00:00 on hooks) iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO comm 0x7fdce98d3d20 rank 2 nranks 16 cudaDev 2 busId 67010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO comm 0x7f8720ca58a0 rank 3 nranks 16 cudaDev 3 busId 67020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO comm 0x7fe03560f9e0 rank 5 nranks 16 cudaDev 5 busId 69020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO comm 0x7ff5dfb31680 rank 4 nranks 16 cudaDev 4 busId 69010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO comm 0x7f2c008153e0 rank 1 nranks 16 cudaDev 1 busId 65020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO comm 0x7fe0ef595930 rank 6 nranks 16 cudaDev 6 busId 6b010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO comm 0x7f2b883aa7d0 rank 7 nranks 16 cudaDev 7 busId 6b020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO comm 0x7f8113900790 rank 0 nranks 16 cudaDev 0 busId 65010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO comm 0x55f6a6503a10 rank 2 nranks 16 cudaDev 2 busId 67010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO comm 0x55c46788cc30 rank 3 nranks 16 cudaDev 3 busId 67020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO comm 0x55dd11333890 rank 1 nranks 16 cudaDev 1 busId 65020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO comm 0x557f31896bf0 rank 6 nranks 16 cudaDev 6 busId 6b010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO comm 0x56347f801d00 rank 7 nranks 16 cudaDev 7 busId 6b020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO comm 0x56521cee47e0 rank 4 nranks 16 cudaDev 4 busId 69010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO comm 0x55b88ffe3130 rank 5 nranks 16 cudaDev 5 busId 69020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO comm 0x55fe38031c70 rank 0 nranks 16 cudaDev 0 busId 65010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO comm 0x55f6a67d48f0 rank 2 nranks 16 cudaDev 2 busId 67010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO comm 0x55c467b5d260 rank 3 nranks 16 cudaDev 3 busId 67020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO comm 0x55dd11604a20 rank 1 nranks 16 cudaDev 1 busId 65020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO comm 0x56347fad6000 rank 7 nranks 16 cudaDev 7 busId 6b020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO comm 0x557f31b670f0 rank 6 nranks 16 cudaDev 6 busId 6b010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO comm 0x56521d1d0600 rank 4 nranks 16 cudaDev 4 busId 69010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO comm 0x55b8902c0df0 rank 5 nranks 16 cudaDev 5 busId 69020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO comm 0x55fe38304fa0 rank 0 nranks 16 cudaDev 0 busId 65010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO comm 0x7fda26451f20 rank 2 nranks 8 cudaDev 2 busId 67010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO comm 0x7fda32a8ef00 rank 0 nranks 2 cudaDev 2 busId 67010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO comm 0x7f843e434f20 rank 3 nranks 8 cudaDev 3 busId 67020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42283:42283 [2] NCCL INFO comm 0x7fda26375c50 rank 0 nranks 2 cudaDev 2 busId 67010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO comm 0x7f844aab2200 rank 0 nranks 2 cudaDev 3 busId 67020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO comm 0x7ff2f24005d0 rank 4 nranks 8 cudaDev 4 busId 69010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO comm 0x7f28be417510 rank 7 nranks 8 cudaDev 7 busId 6b020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42284:42284 [3] NCCL INFO comm 0x7f843e358cc0 rank 0 nranks 2 cudaDev 3 busId 67020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO comm 0x7f292644fe10 rank 1 nranks 8 cudaDev 1 busId 65020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO comm 0x7fdd6241d5d0 rank 5 nranks 8 cudaDev 5 busId 69020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO comm 0x7f7ae286bf20 rank 0 nranks 8 cudaDev 0 busId 65010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO comm 0x7f28baad7e40 rank 0 nranks 2 cudaDev 7 busId 6b020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO comm 0x7fddfe430d70 rank 6 nranks 8 cudaDev 6 busId 6b010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO comm 0x7f2922a67150 rank 0 nranks 2 cudaDev 1 busId 65020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO comm 0x7ff2eea9e040 rank 0 nranks 2 cudaDev 4 busId 69010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO comm 0x7fdd6ea7aaa0 rank 0 nranks 2 cudaDev 5 busId 69020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO comm 0x7f7aeee9df20 rank 0 nranks 2 cudaDev 0 busId 65010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42288:42288 [7] NCCL INFO comm 0x7f28be361230 rank 0 nranks 2 cudaDev 7 busId 6b020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO comm 0x7fde0aa88cd0 rank 0 nranks 2 cudaDev 6 busId 6b010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42282:42282 [1] NCCL INFO comm 0x7f292637bc80 rank 0 nranks 2 cudaDev 1 busId 65020 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42285:42285 [4] NCCL INFO comm 0x7ff2f23550d0 rank 0 nranks 2 cudaDev 4 busId 69010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42287:42287 [6] NCCL INFO comm 0x7fddfe3778a0 rank 0 nranks 2 cudaDev 6 busId 6b010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42281:42281 [0] NCCL INFO comm 0x7f7ae278ff30 rank 0 nranks 2 cudaDev 0 busId 65010 - Destroy COMPLETE iv-ybpu7pvmiu5m57lh5kdd:42286:42286 [5] NCCL INFO comm 0x7fdd6237a660 rank 0 nranks 2 cudaDev 5 busId 69020 - Destroy COMPLETE ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. *****************************************