loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
loaded library: loaded library: loaded library: loaded library: loaded library: loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1/usr/lib/x86_64-linux-gnu/libibverbs.so.1/usr/lib/x86_64-linux-gnu/libibverbs.so.1
/usr/lib/x86_64-linux-gnu/libibverbs.so.1/usr/lib/x86_64-linux-gnu/libibverbs.so.1

/usr/lib/x86_64-linux-gnu/libibverbs.so.1


loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
W20220705 04:31:09.906327 11320 rpc_client.cpp:190] LoadServer 198.18.8.22 Failed at 0 times error_code 14 error_message failed to connect to all addresses
W20220705 04:31:09.906394 11315 rpc_client.cpp:190] LoadServer 198.18.8.22 Failed at 0 times error_code 14 error_message failed to connect to all addresses
W20220705 04:31:09.906394 11318 rpc_client.cpp:190] LoadServer 198.18.8.22 Failed at 0 times error_code 14 error_message failed to connect to all addresses
[07/05 04:31:21 libai]: Rank of current process: 0. World size: 32
[07/05 04:31:21 libai]: Command line arguments: Namespace(config_file='configs/gpt2_nl24_nah16_hs1024.py', eval_only=False, fast_dev_run=False, opts=['model.cfg.num_layers=24', 'train.dist.pipeline_num_layers=24', 'train.train_micro_batch_size=64', 'train.global_batch_size=1024', 'train.dist.tensor_parallel_size=4', 'train.dist.pipeline_parallel_size=4', 'train.amp.enabled=true', 'train.activation_checkpoint.enabled=true', 'train.train_iter=220', 'train.log_period=100', 'train.output_dir=test_logs/01b1d32/4n8g/LibAI_gpt2_nl24_nah16_hs1024_FP16_actrue_mp4_pp4_mb64_gb1024_4n8g_20220705_043108406236792'], resume=False)
[07/05 04:31:21 libai]: Contents of args.config_file=configs/gpt2_nl24_nah16_hs1024.py:
from libai.config import LazyCall
from libai.evaluation import PPLEvaluator
from libai.config import LazyCall
from .common.models.gpt import pretrain_model as model
from .common.train import train
from .common.optim import optim
from .common.data.gpt_dataset import dataloader, tokenization

from .common.models.graph import graph

#vocab_file = "/workspace/dataset/gpt2-vocab.json"
#merges_file = "/workspace/dataset/gpt2-merges.txt"
#data_prefix = "/workspace/dataset/loss_compara_content_sentence"

vocab_file = "/dataset/source/dataset/gpt2-vocab.json"
merges_file = "/dataset/source/dataset/gpt2-merges.txt"
data_prefix = "/dataset/source/dataset/loss_compara_content_sentence"

tokenization.tokenizer.vocab_file = vocab_file
tokenization.tokenizer.merges_file = merges_file
dataloader.train.dataset[0].data_prefix = data_prefix
dataloader.train.dataset[0].indexed_dataset.data_prefix = data_prefix
# dataloader.train.num_workers = 4

# GPT-2 model config
model.cfg.embedding_dropout_prob = 0.1
model.cfg.attention_dropout_prob = 0.1
model.cfg.num_attention_heads = 16
model.cfg.hidden_size = 1024
model.cfg.ffn_hidden_size = 4096
#model.cfg.num_layers = 24
model.cfg.max_seq_length = 1024
#model.cfg.initializer_range = 0.006

# model.cfg.bias_dropout_fusion = True
# model.cfg.bias_gelu_fusion = True
# model.cfg.scale_mask_softmax_fusion = True


train.input_placement_device = "cpu"


for ds in dataloader.train.dataset:
    ds.max_seq_length = model.cfg.max_seq_length

optim.lr = 1.5e-4

#train.dist.pipeline_num_layers = model.cfg.num_layers

train.test_micro_batch_size = 4
train.evaluation.evaluator = LazyCall(PPLEvaluator)()

train.evaluation.enabled = False
train.evaluation.eval_iter = 30

[07/05 04:31:21 libai]: Full config saved to test_logs/01b1d32/4n8g/LibAI_gpt2_nl24_nah16_hs1024_FP16_actrue_mp4_pp4_mb64_gb1024_4n8g_20220705_043108406236792/config.yaml
[07/05 04:31:21 lb.engine.default]: > compiling dataset index builder ...
make: Entering directory '/dataset/xyn/libai_bench/libai/libai/data/data_utils'
make: Nothing to be done for 'default'.
make: Leaving directory '/dataset/xyn/libai_bench/libai/libai/data/data_utils'
[07/05 04:31:21 lb.engine.default]: >>> done with dataset index builder. Compilation time: 0.041 seconds
[07/05 04:31:21 lb.engine.default]: >>> done with compiling. Compilation time: 0.043 seconds
[07/05 04:31:21 lb.engine.default]: Prepare training, validating, testing set
[07/05 04:31:21 lb.data.data_utils.indexed_dataset]: building dataset index ...
[07/05 04:31:21 lb.data.data_utils.indexed_dataset]: warming up index mmap file...
[07/05 04:31:21 lb.data.data_utils.indexed_dataset]: reading sizes...
[07/05 04:31:21 lb.data.data_utils.indexed_dataset]: reading pointers...
[07/05 04:31:21 lb.data.data_utils.indexed_dataset]: reading document index...
[07/05 04:31:21 lb.data.data_utils.indexed_dataset]: warming up data mmap file...
[07/05 04:31:22 lb.data.data_utils.indexed_dataset]: creating numpy buffer of mmap...
[07/05 04:31:22 lb.data.data_utils.indexed_dataset]: creating memory view of numpy buffer...
[07/05 04:31:22 lb.data.data_utils.indexed_dataset]: Finished creating indexed dataset in 0.094948 seconds
[07/05 04:31:22 lb.data.data_utils.indexed_dataset]: indexed dataset stats:
[07/05 04:31:22 lb.data.data_utils.indexed_dataset]: number of documents: 50000
[07/05 04:31:22 lb.data.data_utils.indexed_dataset]: number of sentences: 1249934
[07/05 04:31:22 lb.data.datasets.gpt_dataset]:  > loading doc-idx mapping from /dataset/source/dataset/loss_compara_content_sentence_gpt-2_indexmap_225280ns_1024sl_1234s_doc_idx.npy
[07/05 04:31:22 lb.data.datasets.gpt_dataset]:  > loading sample-idx mapping from /dataset/source/dataset/loss_compara_content_sentence_gpt-2_indexmap_225280ns_1024sl_1234s_sample_idx.npy
[07/05 04:31:22 lb.data.datasets.gpt_dataset]:  > loading shuffle-idx mapping from /dataset/source/dataset/loss_compara_content_sentence_gpt-2_indexmap_225280ns_1024sl_1234s_shuffle_idx.npy
[07/05 04:31:22 lb.data.datasets.gpt_dataset]:     loaded indexed file in 0.006 seconds
[07/05 04:31:22 lb.data.datasets.gpt_dataset]:     total number of samples: 229329
[07/05 04:31:22 lb.data.datasets.gpt_dataset]:     total number of epochs: 4
[07/05 04:31:22 lb.data.datasets.gpt_dataset]:  > loading doc-idx mapping from /dataset/source/dataset/loss_compara_content_sentence_gpt-2_indexmap_8ns_1024sl_1234s_doc_idx.npy
[07/05 04:31:22 lb.data.datasets.gpt_dataset]:  > loading sample-idx mapping from /dataset/source/dataset/loss_compara_content_sentence_gpt-2_indexmap_8ns_1024sl_1234s_sample_idx.npy
[07/05 04:31:22 lb.data.datasets.gpt_dataset]:  > loading shuffle-idx mapping from /dataset/source/dataset/loss_compara_content_sentence_gpt-2_indexmap_8ns_1024sl_1234s_shuffle_idx.npy
[07/05 04:31:22 lb.data.datasets.gpt_dataset]:     loaded indexed file in 0.002 seconds
[07/05 04:31:22 lb.data.datasets.gpt_dataset]:     total number of samples: 57333
[07/05 04:31:22 lb.data.datasets.gpt_dataset]:     total number of epochs: 1
[07/05 04:31:22 lb.data.datasets.gpt_dataset]:  > loading doc-idx mapping from /dataset/source/dataset/loss_compara_content_sentence_gpt-2_indexmap_8ns_1024sl_1234s_doc_idx.npy
[07/05 04:31:22 lb.data.datasets.gpt_dataset]:  > loading sample-idx mapping from /dataset/source/dataset/loss_compara_content_sentence_gpt-2_indexmap_8ns_1024sl_1234s_sample_idx.npy
[07/05 04:31:22 lb.data.datasets.gpt_dataset]:  > loading shuffle-idx mapping from /dataset/source/dataset/loss_compara_content_sentence_gpt-2_indexmap_8ns_1024sl_1234s_shuffle_idx.npy
[07/05 04:31:22 lb.data.datasets.gpt_dataset]:     loaded indexed file in 0.002 seconds
[07/05 04:31:22 lb.data.datasets.gpt_dataset]:     total number of samples: 57333
[07/05 04:31:22 lb.data.datasets.gpt_dataset]:     total number of epochs: 1
[07/05 04:31:26 lb.engine.default]: Auto-scaling the config to train.train_iter=220, train.warmup_iter=0
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0>
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol.
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol.
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO P2P plugin IBext
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0>
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Using network IBext
NCCL version 2.12.10+cuda11.2
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0>
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0>
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0>
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0>
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0>
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol.
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol.
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol.
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol.
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol.
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol.
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol.
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol.
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO P2P plugin IBext
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO P2P plugin IBext
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO P2P plugin IBext
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO P2P plugin IBext
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol.
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol.
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO P2P plugin IBext
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0>
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol.
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol.
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO P2P plugin IBext
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0>
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Using network IBext
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0>
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Using network IBext
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0>
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Using network IBext
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0>
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Using network IBext
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0>
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Using network IBext
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0>
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Using network IBext
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0>
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v5 symbol.
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v5 symbol.
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO P2P plugin IBext
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0>
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Using network IBext
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3.
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3.
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3.
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3.
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3.
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3.
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3.
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3.
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23.
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7.
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23.
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7.
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23.
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23.
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7.
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7.
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23.
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23.
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7.
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7.
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23.
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7.
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23.
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7.
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO PXN Disabled as plugin is v4
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO PXN Disabled as plugin is v4
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO PXN Disabled as plugin is v4
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO PXN Disabled as plugin is v4
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO PXN Disabled as plugin is v4
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO PXN Disabled as plugin is v4
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO PXN Disabled as plugin is v4
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO PXN Disabled as plugin is v4
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Trees [0] -1/-1/-1->7->5 [1] -1/-1/-1->7->5 [2] 5/-1/-1->7->0 [3] 5/-1/-1->7->0 [4] 4/-1/-1->7->6 [5] 6/-1/-1->7->4 [6] -1/-1/-1->7->5 [7] -1/-1/-1->7->5 [8] 5/-1/-1->7->0 [9] 5/-1/-1->7->0 [10] 4/-1/-1->7->6 [11] 6/-1/-1->7->4
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 00/12 :    0   2   3   1   6   4   5   7
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Trees [0] 4/-1/-1->6->1 [1] 4/-1/-1->6->1 [2] 1/-1/-1->6->4 [3] 1/-1/-1->6->4 [4] 7/-1/-1->6->5 [5] 5/-1/-1->6->7 [6] 4/-1/-1->6->1 [7] 4/-1/-1->6->1 [8] 1/-1/-1->6->4 [9] 1/-1/-1->6->4 [10] 7/-1/-1->6->5 [11] 5/-1/-1->6->7
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Trees [0] 6/-1/-1->1->3 [1] 6/-1/-1->1->3 [2] 3/-1/-1->1->6 [3] 3/-1/-1->1->6 [4] 2/-1/-1->1->0 [5] -1/-1/-1->1->2 [6] 6/-1/-1->1->3 [7] 6/-1/-1->1->3 [8] 3/-1/-1->1->6 [9] 3/-1/-1->1->6 [10] 2/-1/-1->1->0 [11] -1/-1/-1->1->2
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 3/-1/-1->2->0 [2] -1/-1/-1->2->3 [3] -1/-1/-1->2->3 [4] 5/-1/-1->2->1 [5] 1/-1/-1->2->5 [6] 3/-1/-1->2->0 [7] 3/-1/-1->2->0 [8] -1/-1/-1->2->3 [9] -1/-1/-1->2->3 [10] 5/-1/-1->2->1 [11] 1/-1/-1->2->5
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Trees [0] 5/-1/-1->4->6 [1] 5/-1/-1->4->6 [2] 6/-1/-1->4->5 [3] 6/-1/-1->4->5 [4] 3/-1/-1->4->7 [5] 7/-1/-1->4->3 [6] 5/-1/-1->4->6 [7] 5/-1/-1->4->6 [8] 6/-1/-1->4->5 [9] 6/-1/-1->4->5 [10] 3/-1/-1->4->7 [11] 7/-1/-1->4->3
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Trees [0] 7/-1/-1->5->4 [1] 7/-1/-1->5->4 [2] 4/-1/-1->5->7 [3] 4/-1/-1->5->7 [4] 6/-1/-1->5->2 [5] 2/-1/-1->5->6 [6] 7/-1/-1->5->4 [7] 7/-1/-1->5->4 [8] 4/-1/-1->5->7 [9] 4/-1/-1->5->7 [10] 6/-1/-1->5->2 [11] 2/-1/-1->5->6
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Trees [0] 1/-1/-1->3->2 [1] 1/-1/-1->3->2 [2] 2/-1/-1->3->1 [3] 2/-1/-1->3->1 [4] -1/-1/-1->3->4 [5] 4/-1/-1->3->0 [6] 1/-1/-1->3->2 [7] 1/-1/-1->3->2 [8] 2/-1/-1->3->1 [9] 2/-1/-1->3->1 [10] -1/-1/-1->3->4 [11] 4/-1/-1->3->0
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 01/12 :    0   2   3   1   6   4   5   7
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 02/12 :    0   7   5   4   6   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 03/12 :    0   7   5   4   6   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 04/12 :    0   1   2   5   6   7   4   3
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 05/12 :    0   3   4   7   6   5   2   1
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 06/12 :    0   2   3   1   6   4   5   7
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 07/12 :    0   2   3   1   6   4   5   7
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 08/12 :    0   7   5   4   6   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 09/12 :    0   7   5   4   6   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 10/12 :    0   1   2   5   6   7   4   3
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 11/12 :    0   3   4   7   6   5   2   1
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Trees [0] 2/-1/-1->0->-1 [1] 2/-1/-1->0->-1 [2] 7/-1/-1->0->-1 [3] 7/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 3/-1/-1->0->-1 [6] 2/-1/-1->0->-1 [7] 2/-1/-1->0->-1 [8] 7/-1/-1->0->-1 [9] 7/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 3/-1/-1->0->-1
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 00 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 00 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 04 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 04 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 01 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 01 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 10 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 10 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 06 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 06 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 04 : 0[65010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 07 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 04 : 5[69020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 06 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 07 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 05 : 3[67020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 10 : 0[65010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 10 : 5[69020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 07 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 11 : 3[67020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 02 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 00 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 03 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 02 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 01 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 08 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 03 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 06 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 06 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 09 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 08 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 07 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 07 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 09 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 05 : 4[69010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 02 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 04 : 2[67010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 05 : 0[65010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 11 : 4[69010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 03 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 10 : 2[67010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 11 : 0[65010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 08 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 09 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 04 : 7[6b020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 04 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 05 : 5[69020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 10 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 10 : 7[6b020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 11 : 5[69020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 00 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 02 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 02 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 01 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 03 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 03 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 06 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 06 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 08 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 08 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 07 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 07 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 09 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 09 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 00 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 01 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 02 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 06 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 02 : 0[65010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 03 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 07 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 03 : 0[65010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 08 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 05 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 08 : 0[65010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 05 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 09 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 11 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 09 : 0[65010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 11 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 05 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 05 : 6[6b010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 11 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 11 : 6[6b010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 04 : 4[69010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 10 : 4[69010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 02 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 05 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 03 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 11 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 08 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 09 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 05 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 11 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 02 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 03 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 08 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 09 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 05 : 5[69020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 02 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 02 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 11 : 5[69020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 03 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 03 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 08 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 04 : 3[67020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 08 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 09 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 10 : 3[67020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 09 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 02 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 00 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 05 : 2[67010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 03 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 01 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 11 : 2[67010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 08 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 06 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 09 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 07 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 06 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 07 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 00 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 04 : 4[69010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 10 : 4[69010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 05 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 01 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 05 : 7[6b020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 04 : 5[69020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 11 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 06 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 11 : 7[6b020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 10 : 5[69020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 07 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 02 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 00 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 03 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 01 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 08 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 06 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 02 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 09 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 06 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 07 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 03 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 07 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 08 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 09 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 04 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 10 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 00 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 01 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 02 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 06 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 04 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 03 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 07 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 10 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 08 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 09 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 04 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 04 : 6[6b010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 10 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 10 : 6[6b010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 05 : 4[69010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 06 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 11 : 4[69010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 07 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 02 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 10 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 02 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 10 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 03 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 02 : 3[67020] -> 5[69020] via P2P/indirect/4[69010]
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 02 : 2[67010] -> 4[69010] via P2P/indirect/5[69020]
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 03 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 11 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 10 : 3[67020] -> 5[69020] via P2P/indirect/4[69010]
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 10 : 2[67010] -> 4[69010] via P2P/indirect/5[69020]
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 11 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 03 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 03 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 11 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 11 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 04 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020]
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 04 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 04 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 12 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020]
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 04 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 04 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 12 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 04 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 12 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 04 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 04 : 0[65010] -> 4[69010] via P2P/indirect/3[67020]
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO Channel 12 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 12 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO Channel 12 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 12 : 0[65010] -> 4[69010] via P2P/indirect/3[67020]
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 12 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 05 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 05 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO Channel 13 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 05 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 05 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO Channel 13 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 13 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 13 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 06 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 06 : 5[69020] -> 3[67020] via P2P/indirect/2[67010]
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 06 : 4[69010] -> 2[67010] via P2P/indirect/3[67020]
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 06 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Channel 14 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO Channel 14 : 4[69010] -> 2[67010] via P2P/indirect/3[67020]
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO Channel 14 : 5[69020] -> 3[67020] via P2P/indirect/2[67010]
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO Channel 14 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11320:12530 [7] NCCL INFO comm 0x7f232f75ef40 rank 7 nranks 8 cudaDev 7 busId 6b020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11318:12527 [5] NCCL INFO comm 0x7fae141cc3c0 rank 5 nranks 8 cudaDev 5 busId 69020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:12526 [4] NCCL INFO comm 0x7fb9304c6cd0 rank 4 nranks 8 cudaDev 4 busId 69010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11319:12529 [6] NCCL INFO comm 0x7f7f0d008e90 rank 6 nranks 8 cudaDev 6 busId 6b010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO comm 0x7faf759eecc0 rank 0 nranks 8 cudaDev 0 busId 65010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11315:12533 [2] NCCL INFO comm 0x7f8619061990 rank 2 nranks 8 cudaDev 2 busId 67010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11316:12528 [3] NCCL INFO comm 0x7f9fd8f8b3b0 rank 3 nranks 8 cudaDev 3 busId 67020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11314:12532 [1] NCCL INFO comm 0x7f187446e790 rank 1 nranks 8 cudaDev 1 busId 65020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:12531 [0] NCCL INFO Launch mode Parallel
[07/05 04:31:46 lb.engine.default]: Model:
GPTForPreTraining(
  (GPT_model): GPTModel(
    (embeddings): GPTEmbedding(
      (token_embeddings): VocabEmbedding(num_embeddings=50688, embedding_dim=1024)
      (position_embeddings): Embedding(num_embeddings=1024, embedding_dim=1024)
      (dropout): Dropout(p=0.1, inplace=False)
    )
    (transformer): Transformer(
      (layers): ModuleList(
        (0): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (1): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (2): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (3): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (4): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (5): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (6): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (7): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (8): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (9): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (10): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (11): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (12): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (13): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (14): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (15): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (16): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (17): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (18): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (19): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (20): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (21): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (22): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
        (23): TransformerLayer(
          (drop_path): Identity()
          (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (self_attention): MultiheadAttention(
            hidden_size=1024, num_heads=16, is_cross_attention=False
            (dropout): Dropout(p=0.1, inplace=False)
            (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
            (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
          )
          (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0
            (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
          )
        )
      )
      (layernorm_f): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
    )
    (lm_head): LMLogits()
  )
  (loss_func): GPTLoss(
    (lm_loss): ParallelCrossEntropyLoss()
  )
)
WARNING [07/05 04:31:46 lb.scheduler.lr_scheduler]: warmup iters equals to zero, return CosineLR
[07/05 04:31:46 lb.engine.trainer]: Starting training from iteration 0
[07/05 04:35:17 lb.models.utils.graph_base]: Start compling the train graph which may take some time. Please wait for a moment ...
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Trees [0] -1/-1/-1->7->5 [1] -1/-1/-1->7->5 [2] 5/-1/-1->7->0 [3] 5/-1/-1->7->0 [4] 4/-1/-1->7->6 [5] 6/-1/-1->7->4 [6] -1/-1/-1->7->5 [7] -1/-1/-1->7->5 [8] 5/-1/-1->7->0 [9] 5/-1/-1->7->0 [10] 4/-1/-1->7->6 [11] 6/-1/-1->7->4
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 3/-1/-1->2->0 [2] -1/-1/-1->2->3 [3] -1/-1/-1->2->3 [4] 5/-1/-1->2->1 [5] 1/-1/-1->2->5 [6] 3/-1/-1->2->0 [7] 3/-1/-1->2->0 [8] -1/-1/-1->2->3 [9] -1/-1/-1->2->3 [10] 5/-1/-1->2->1 [11] 1/-1/-1->2->5
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 00/12 :    0   2   3   1   6   4   5   7
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Trees [0] 5/-1/-1->4->6 [1] 5/-1/-1->4->6 [2] 6/-1/-1->4->5 [3] 6/-1/-1->4->5 [4] 3/-1/-1->4->7 [5] 7/-1/-1->4->3 [6] 5/-1/-1->4->6 [7] 5/-1/-1->4->6 [8] 6/-1/-1->4->5 [9] 6/-1/-1->4->5 [10] 3/-1/-1->4->7 [11] 7/-1/-1->4->3
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Trees [0] 7/-1/-1->5->4 [1] 7/-1/-1->5->4 [2] 4/-1/-1->5->7 [3] 4/-1/-1->5->7 [4] 6/-1/-1->5->2 [5] 2/-1/-1->5->6 [6] 7/-1/-1->5->4 [7] 7/-1/-1->5->4 [8] 4/-1/-1->5->7 [9] 4/-1/-1->5->7 [10] 6/-1/-1->5->2 [11] 2/-1/-1->5->6
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Trees [0] 4/-1/-1->6->1 [1] 4/-1/-1->6->1 [2] 1/-1/-1->6->4 [3] 1/-1/-1->6->4 [4] 7/-1/-1->6->5 [5] 5/-1/-1->6->7 [6] 4/-1/-1->6->1 [7] 4/-1/-1->6->1 [8] 1/-1/-1->6->4 [9] 1/-1/-1->6->4 [10] 7/-1/-1->6->5 [11] 5/-1/-1->6->7
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Trees [0] 6/-1/-1->1->3 [1] 6/-1/-1->1->3 [2] 3/-1/-1->1->6 [3] 3/-1/-1->1->6 [4] 2/-1/-1->1->0 [5] -1/-1/-1->1->2 [6] 6/-1/-1->1->3 [7] 6/-1/-1->1->3 [8] 3/-1/-1->1->6 [9] 3/-1/-1->1->6 [10] 2/-1/-1->1->0 [11] -1/-1/-1->1->2
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Trees [0] 1/-1/-1->3->2 [1] 1/-1/-1->3->2 [2] 2/-1/-1->3->1 [3] 2/-1/-1->3->1 [4] -1/-1/-1->3->4 [5] 4/-1/-1->3->0 [6] 1/-1/-1->3->2 [7] 1/-1/-1->3->2 [8] 2/-1/-1->3->1 [9] 2/-1/-1->3->1 [10] -1/-1/-1->3->4 [11] 4/-1/-1->3->0
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 01/12 :    0   2   3   1   6   4   5   7
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 02/12 :    0   7   5   4   6   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 03/12 :    0   7   5   4   6   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 04/12 :    0   1   2   5   6   7   4   3
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 05/12 :    0   3   4   7   6   5   2   1
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 06/12 :    0   2   3   1   6   4   5   7
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 07/12 :    0   2   3   1   6   4   5   7
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 08/12 :    0   7   5   4   6   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 09/12 :    0   7   5   4   6   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 10/12 :    0   1   2   5   6   7   4   3
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 11/12 :    0   3   4   7   6   5   2   1
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Trees [0] 2/-1/-1->0->-1 [1] 2/-1/-1->0->-1 [2] 7/-1/-1->0->-1 [3] 7/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 3/-1/-1->0->-1 [6] 2/-1/-1->0->-1 [7] 2/-1/-1->0->-1 [8] 7/-1/-1->0->-1 [9] 7/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 3/-1/-1->0->-1
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 00 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 00 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 04 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 01 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 01 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 04 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 10 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 06 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 06 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 10 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 04 : 5[69020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 06 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 04 : 0[65010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 07 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 07 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 05 : 3[67020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 10 : 5[69020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 07 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 10 : 0[65010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 11 : 3[67020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 02 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 00 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 03 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 02 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 01 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 08 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 03 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 06 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 09 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 06 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 08 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 07 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 07 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 09 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 04 : 2[67010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 05 : 0[65010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 05 : 4[69010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 02 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 10 : 2[67010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 11 : 0[65010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 11 : 4[69010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 03 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 08 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 09 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 05 : 5[69020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 04 : 7[6b020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 04 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 11 : 5[69020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 10 : 7[6b020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 10 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 02 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 02 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 00 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 03 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 03 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 01 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 08 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 08 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 06 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 06 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 09 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 09 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 07 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 07 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 00 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 01 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 02 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 02 : 0[65010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 06 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 03 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 03 : 0[65010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 07 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 08 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 05 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 08 : 0[65010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 05 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 09 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 11 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 09 : 0[65010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 11 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 05 : 6[6b010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 05 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 02 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 11 : 6[6b010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 11 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 03 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 08 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 09 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 05 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 04 : 4[69010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 11 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 10 : 4[69010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 05 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 11 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 02 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 03 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 08 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 02 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 09 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 03 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 08 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 02 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 09 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 03 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 05 : 5[69020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 08 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 04 : 3[67020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 11 : 5[69020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 09 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 10 : 3[67020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 06 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 07 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 02 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 05 : 2[67010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 00 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 03 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 01 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 08 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 06 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 09 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 11 : 2[67010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 07 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 05 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 00 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 04 : 4[69010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 11 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 01 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 10 : 4[69010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 06 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 04 : 5[69020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 07 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 02 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 05 : 7[6b020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 10 : 5[69020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 03 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 11 : 7[6b020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 08 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 09 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 02 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 00 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 03 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 01 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 06 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 06 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 08 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 07 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 07 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 09 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 04 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 02 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 10 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 03 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 00 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 08 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 01 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 09 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 04 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 06 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 10 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 07 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 04 : 6[6b010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 04 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 10 : 6[6b010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 10 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 05 : 4[69010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 02 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 06 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 11 : 4[69010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 07 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 10 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 02 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 10 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 02 : 3[67020] -> 5[69020] via P2P/indirect/4[69010]
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 03 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 03 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 02 : 2[67010] -> 4[69010] via P2P/indirect/5[69020]
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 10 : 3[67020] -> 5[69020] via P2P/indirect/4[69010]
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 10 : 2[67010] -> 4[69010] via P2P/indirect/5[69020]
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 11 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 11 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 03 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 03 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 11 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 11 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 04 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 04 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 04 : 0[65010] -> 4[69010] via P2P/indirect/3[67020]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 12 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 04 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020]
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 04 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 12 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 04 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 04 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 04 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 12 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 12 : 0[65010] -> 4[69010] via P2P/indirect/3[67020]
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 12 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 12 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 12 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 12 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 05 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 05 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 05 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 05 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 13 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 13 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 13 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 13 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 06 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 06 : 5[69020] -> 3[67020] via P2P/indirect/2[67010]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 06 : 4[69010] -> 2[67010] via P2P/indirect/3[67020]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 06 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 14 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 14 : 5[69020] -> 3[67020] via P2P/indirect/2[67010]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 14 : 4[69010] -> 2[67010] via P2P/indirect/3[67020]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 14 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO comm 0x559725fa38f0 rank 5 nranks 8 cudaDev 5 busId 69020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO comm 0x56373f6b8ac0 rank 7 nranks 8 cudaDev 7 busId 6b020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO comm 0x55ba328bb9c0 rank 6 nranks 8 cudaDev 6 busId 6b010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO comm 0x565323d3b730 rank 1 nranks 8 cudaDev 1 busId 65020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO comm 0x563f688fffb0 rank 0 nranks 8 cudaDev 0 busId 65010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO comm 0x560b8233a870 rank 3 nranks 8 cudaDev 3 busId 67020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO comm 0x562a4c512860 rank 4 nranks 8 cudaDev 4 busId 69010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO comm 0x562ed9ecd8c0 rank 2 nranks 8 cudaDev 2 busId 67010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 00/12 :    0   2   3   1   6   4   5   7
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 3/-1/-1->2->0 [2] -1/-1/-1->2->3 [3] -1/-1/-1->2->3 [4] 5/-1/-1->2->1 [5] 1/-1/-1->2->5 [6] 3/-1/-1->2->0 [7] 3/-1/-1->2->0 [8] -1/-1/-1->2->3 [9] -1/-1/-1->2->3 [10] 5/-1/-1->2->1 [11] 1/-1/-1->2->5
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Trees [0] 6/-1/-1->1->3 [1] 6/-1/-1->1->3 [2] 3/-1/-1->1->6 [3] 3/-1/-1->1->6 [4] 2/-1/-1->1->0 [5] -1/-1/-1->1->2 [6] 6/-1/-1->1->3 [7] 6/-1/-1->1->3 [8] 3/-1/-1->1->6 [9] 3/-1/-1->1->6 [10] 2/-1/-1->1->0 [11] -1/-1/-1->1->2
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Trees [0] -1/-1/-1->7->5 [1] -1/-1/-1->7->5 [2] 5/-1/-1->7->0 [3] 5/-1/-1->7->0 [4] 4/-1/-1->7->6 [5] 6/-1/-1->7->4 [6] -1/-1/-1->7->5 [7] -1/-1/-1->7->5 [8] 5/-1/-1->7->0 [9] 5/-1/-1->7->0 [10] 4/-1/-1->7->6 [11] 6/-1/-1->7->4
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Trees [0] 4/-1/-1->6->1 [1] 4/-1/-1->6->1 [2] 1/-1/-1->6->4 [3] 1/-1/-1->6->4 [4] 7/-1/-1->6->5 [5] 5/-1/-1->6->7 [6] 4/-1/-1->6->1 [7] 4/-1/-1->6->1 [8] 1/-1/-1->6->4 [9] 1/-1/-1->6->4 [10] 7/-1/-1->6->5 [11] 5/-1/-1->6->7
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 01/12 :    0   2   3   1   6   4   5   7
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Trees [0] 7/-1/-1->5->4 [1] 7/-1/-1->5->4 [2] 4/-1/-1->5->7 [3] 4/-1/-1->5->7 [4] 6/-1/-1->5->2 [5] 2/-1/-1->5->6 [6] 7/-1/-1->5->4 [7] 7/-1/-1->5->4 [8] 4/-1/-1->5->7 [9] 4/-1/-1->5->7 [10] 6/-1/-1->5->2 [11] 2/-1/-1->5->6
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Trees [0] 1/-1/-1->3->2 [1] 1/-1/-1->3->2 [2] 2/-1/-1->3->1 [3] 2/-1/-1->3->1 [4] -1/-1/-1->3->4 [5] 4/-1/-1->3->0 [6] 1/-1/-1->3->2 [7] 1/-1/-1->3->2 [8] 2/-1/-1->3->1 [9] 2/-1/-1->3->1 [10] -1/-1/-1->3->4 [11] 4/-1/-1->3->0
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Trees [0] 5/-1/-1->4->6 [1] 5/-1/-1->4->6 [2] 6/-1/-1->4->5 [3] 6/-1/-1->4->5 [4] 3/-1/-1->4->7 [5] 7/-1/-1->4->3 [6] 5/-1/-1->4->6 [7] 5/-1/-1->4->6 [8] 6/-1/-1->4->5 [9] 6/-1/-1->4->5 [10] 3/-1/-1->4->7 [11] 7/-1/-1->4->3
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 02/12 :    0   7   5   4   6   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 03/12 :    0   7   5   4   6   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 04/12 :    0   1   2   5   6   7   4   3
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 05/12 :    0   3   4   7   6   5   2   1
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 06/12 :    0   2   3   1   6   4   5   7
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 07/12 :    0   2   3   1   6   4   5   7
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 08/12 :    0   7   5   4   6   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 09/12 :    0   7   5   4   6   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 10/12 :    0   1   2   5   6   7   4   3
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 11/12 :    0   3   4   7   6   5   2   1
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Trees [0] 2/-1/-1->0->-1 [1] 2/-1/-1->0->-1 [2] 7/-1/-1->0->-1 [3] 7/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 3/-1/-1->0->-1 [6] 2/-1/-1->0->-1 [7] 2/-1/-1->0->-1 [8] 7/-1/-1->0->-1 [9] 7/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 3/-1/-1->0->-1
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 00 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 04 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 00 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 04 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 01 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 10 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 01 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 10 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 04 : 5[69020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 06 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 06 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 05 : 3[67020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 06 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 04 : 0[65010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 10 : 5[69020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 07 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 07 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 11 : 3[67020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 07 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 10 : 0[65010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 02 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 00 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 02 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 03 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 01 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 03 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 08 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 06 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 08 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 09 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 06 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 07 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 09 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 07 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 02 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 05 : 4[69010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 04 : 2[67010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 05 : 0[65010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 03 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 11 : 4[69010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 11 : 0[65010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 10 : 2[67010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 08 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 09 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 04 : 7[6b020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 04 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 05 : 5[69020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 10 : 7[6b020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 10 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 11 : 5[69020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 02 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 00 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 02 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 03 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 01 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 03 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 08 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 06 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 06 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 08 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 09 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 07 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 07 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 09 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 00 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 01 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 06 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 02 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 02 : 0[65010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 03 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 07 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 03 : 0[65010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 08 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 05 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 08 : 0[65010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 05 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 11 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 09 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 09 : 0[65010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 11 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 05 : 6[6b010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 02 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 05 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 11 : 6[6b010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 03 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 11 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 04 : 4[69010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 08 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 10 : 4[69010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 09 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 05 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 11 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 05 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 11 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 02 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 03 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 02 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 08 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 02 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 03 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 09 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 03 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 05 : 5[69020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 04 : 3[67020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 08 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 08 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 11 : 5[69020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 10 : 3[67020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 09 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 09 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 06 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 02 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 00 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 05 : 2[67010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 07 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 03 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 01 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 11 : 2[67010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 08 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 06 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 09 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 07 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 05 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 04 : 4[69010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 00 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 11 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 10 : 4[69010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 01 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 06 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 07 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 04 : 5[69020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 02 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 05 : 7[6b020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 10 : 5[69020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 03 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 11 : 7[6b020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 08 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 09 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 02 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 00 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 03 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 01 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 06 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 08 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 06 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 09 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 07 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 07 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 02 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 04 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 03 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 10 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 00 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 08 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 01 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 09 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 06 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 04 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 07 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 10 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 04 : 6[6b010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 04 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 10 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 10 : 6[6b010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 06 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 07 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 05 : 4[69010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 11 : 4[69010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 02 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 02 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 10 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 02 : 2[67010] -> 4[69010] via P2P/indirect/5[69020]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 10 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 10 : 2[67010] -> 4[69010] via P2P/indirect/5[69020]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 02 : 3[67020] -> 5[69020] via P2P/indirect/4[69010]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 03 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 03 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 10 : 3[67020] -> 5[69020] via P2P/indirect/4[69010]
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 11 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 11 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 03 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 03 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 11 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 11 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 04 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 04 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 04 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 12 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 04 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 12 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 04 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 04 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 04 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 12 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 04 : 0[65010] -> 4[69010] via P2P/indirect/3[67020]
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 12 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 12 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020]
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 12 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 12 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 12 : 0[65010] -> 4[69010] via P2P/indirect/3[67020]
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 05 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 05 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 13 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 05 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 13 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 05 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 13 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 13 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 06 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 06 : 5[69020] -> 3[67020] via P2P/indirect/2[67010]
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 06 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 06 : 4[69010] -> 2[67010] via P2P/indirect/3[67020]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 14 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 14 : 5[69020] -> 3[67020] via P2P/indirect/2[67010]
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 14 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 14 : 4[69010] -> 2[67010] via P2P/indirect/3[67020]
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO comm 0x56373f9699b0 rank 7 nranks 8 cudaDev 7 busId 6b020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO comm 0x5597262578c0 rank 5 nranks 8 cudaDev 5 busId 69020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO comm 0x562a4c7c50f0 rank 4 nranks 8 cudaDev 4 busId 69010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO comm 0x55ba32b6fd50 rank 6 nranks 8 cudaDev 6 busId 6b010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO comm 0x560b825ee780 rank 3 nranks 8 cudaDev 3 busId 67020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO comm 0x565323fedc30 rank 1 nranks 8 cudaDev 1 busId 65020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO comm 0x563f68bad9c0 rank 0 nranks 8 cudaDev 0 busId 65010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO comm 0x562eda17d8d0 rank 2 nranks 8 cudaDev 2 busId 67010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Trees [0] 4/-1/-1->6->1 [1] 4/-1/-1->6->1 [2] 1/-1/-1->6->4 [3] 1/-1/-1->6->4 [4] 7/-1/-1->6->5 [5] 5/-1/-1->6->7 [6] 4/-1/-1->6->1 [7] 4/-1/-1->6->1 [8] 1/-1/-1->6->4 [9] 1/-1/-1->6->4 [10] 7/-1/-1->6->5 [11] 5/-1/-1->6->7
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 00/12 :    0   2   3   1   6   4   5   7
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Trees [0] 7/-1/-1->5->4 [1] 7/-1/-1->5->4 [2] 4/-1/-1->5->7 [3] 4/-1/-1->5->7 [4] 6/-1/-1->5->2 [5] 2/-1/-1->5->6 [6] 7/-1/-1->5->4 [7] 7/-1/-1->5->4 [8] 4/-1/-1->5->7 [9] 4/-1/-1->5->7 [10] 6/-1/-1->5->2 [11] 2/-1/-1->5->6
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 3/-1/-1->2->0 [2] -1/-1/-1->2->3 [3] -1/-1/-1->2->3 [4] 5/-1/-1->2->1 [5] 1/-1/-1->2->5 [6] 3/-1/-1->2->0 [7] 3/-1/-1->2->0 [8] -1/-1/-1->2->3 [9] -1/-1/-1->2->3 [10] 5/-1/-1->2->1 [11] 1/-1/-1->2->5
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Trees [0] 6/-1/-1->1->3 [1] 6/-1/-1->1->3 [2] 3/-1/-1->1->6 [3] 3/-1/-1->1->6 [4] 2/-1/-1->1->0 [5] -1/-1/-1->1->2 [6] 6/-1/-1->1->3 [7] 6/-1/-1->1->3 [8] 3/-1/-1->1->6 [9] 3/-1/-1->1->6 [10] 2/-1/-1->1->0 [11] -1/-1/-1->1->2
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Trees [0] 1/-1/-1->3->2 [1] 1/-1/-1->3->2 [2] 2/-1/-1->3->1 [3] 2/-1/-1->3->1 [4] -1/-1/-1->3->4 [5] 4/-1/-1->3->0 [6] 1/-1/-1->3->2 [7] 1/-1/-1->3->2 [8] 2/-1/-1->3->1 [9] 2/-1/-1->3->1 [10] -1/-1/-1->3->4 [11] 4/-1/-1->3->0
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Trees [0] 5/-1/-1->4->6 [1] 5/-1/-1->4->6 [2] 6/-1/-1->4->5 [3] 6/-1/-1->4->5 [4] 3/-1/-1->4->7 [5] 7/-1/-1->4->3 [6] 5/-1/-1->4->6 [7] 5/-1/-1->4->6 [8] 6/-1/-1->4->5 [9] 6/-1/-1->4->5 [10] 3/-1/-1->4->7 [11] 7/-1/-1->4->3
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Trees [0] -1/-1/-1->7->5 [1] -1/-1/-1->7->5 [2] 5/-1/-1->7->0 [3] 5/-1/-1->7->0 [4] 4/-1/-1->7->6 [5] 6/-1/-1->7->4 [6] -1/-1/-1->7->5 [7] -1/-1/-1->7->5 [8] 5/-1/-1->7->0 [9] 5/-1/-1->7->0 [10] 4/-1/-1->7->6 [11] 6/-1/-1->7->4
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 01/12 :    0   2   3   1   6   4   5   7
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 02/12 :    0   7   5   4   6   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 03/12 :    0   7   5   4   6   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 04/12 :    0   1   2   5   6   7   4   3
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 05/12 :    0   3   4   7   6   5   2   1
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 06/12 :    0   2   3   1   6   4   5   7
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 07/12 :    0   2   3   1   6   4   5   7
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 08/12 :    0   7   5   4   6   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 09/12 :    0   7   5   4   6   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 10/12 :    0   1   2   5   6   7   4   3
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 11/12 :    0   3   4   7   6   5   2   1
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Trees [0] 2/-1/-1->0->-1 [1] 2/-1/-1->0->-1 [2] 7/-1/-1->0->-1 [3] 7/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 3/-1/-1->0->-1 [6] 2/-1/-1->0->-1 [7] 2/-1/-1->0->-1 [8] 7/-1/-1->0->-1 [9] 7/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 3/-1/-1->0->-1
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 00 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 00 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 04 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 04 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 01 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 01 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 10 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 10 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 06 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 06 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 05 : 3[67020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 04 : 5[69020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 06 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 04 : 0[65010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 07 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 07 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 11 : 3[67020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 10 : 5[69020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 07 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 10 : 0[65010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 02 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 02 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 00 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 03 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 03 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 01 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 08 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 08 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 06 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 09 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 06 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 09 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 07 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 07 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 02 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 05 : 4[69010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 04 : 2[67010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 05 : 0[65010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 03 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 11 : 4[69010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 10 : 2[67010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 11 : 0[65010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 08 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 09 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 04 : 7[6b020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 04 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 05 : 5[69020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 10 : 7[6b020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 10 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 11 : 5[69020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 00 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 02 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 02 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 01 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 03 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 03 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 06 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 06 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 08 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 08 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 07 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 09 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 07 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 09 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 00 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 01 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 06 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 02 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 02 : 0[65010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 07 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 03 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 03 : 0[65010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 05 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 08 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 08 : 0[65010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 05 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 11 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 09 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 09 : 0[65010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 11 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 05 : 6[6b010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 05 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 02 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 11 : 6[6b010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 11 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 03 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 08 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 09 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 04 : 4[69010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 05 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 10 : 4[69010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 11 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 05 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 11 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 02 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 03 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 02 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 08 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 03 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 09 : 7[6b020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 08 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 02 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 09 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 03 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 05 : 5[69020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 08 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 04 : 3[67020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 11 : 5[69020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 09 : 4[69010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 10 : 3[67020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 06 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 07 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 02 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 05 : 2[67010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 00 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 03 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 11 : 2[67010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 01 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 08 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 06 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 09 : 5[69020] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 07 : 4[69010] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 00 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 04 : 4[69010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 05 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 01 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 10 : 4[69010] -> 7[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 11 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 06 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 04 : 5[69020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 07 : 6[6b010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 02 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 05 : 7[6b020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 10 : 5[69020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 03 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 11 : 7[6b020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 08 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 09 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 02 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 00 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 03 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 01 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 06 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 08 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 06 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 07 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 09 : 1[65020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 07 : 7[6b020] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 02 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 04 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 03 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 10 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 00 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 08 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 09 : 6[6b010] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 01 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 04 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 06 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 10 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 07 : 5[69020] -> 4[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 04 : 6[6b010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 04 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 10 : 6[6b010] -> 5[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 10 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 05 : 4[69010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 02 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 06 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 11 : 4[69010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 10 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 07 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 02 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 10 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 03 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 03 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 02 : 3[67020] -> 5[69020] via P2P/indirect/4[69010]
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 02 : 2[67010] -> 4[69010] via P2P/indirect/5[69020]
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 11 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 11 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 10 : 3[67020] -> 5[69020] via P2P/indirect/4[69010]
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 10 : 2[67010] -> 4[69010] via P2P/indirect/5[69020]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 03 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 03 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 11 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 11 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 04 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020]
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 04 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 04 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 04 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 04 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 04 : 0[65010] -> 4[69010] via P2P/indirect/3[67020]
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 12 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020]
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 12 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 12 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 04 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 12 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 04 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO Channel 12 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 12 : 0[65010] -> 4[69010] via P2P/indirect/3[67020]
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO Channel 12 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 05 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 12 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 05 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 05 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO Channel 13 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO Channel 13 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 05 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 13 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 13 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 06 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 06 : 5[69020] -> 3[67020] via P2P/indirect/2[67010]
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 06 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 06 : 4[69010] -> 2[67010] via P2P/indirect/3[67020]
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO Channel 14 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020]
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO Channel 14 : 5[69020] -> 3[67020] via P2P/indirect/2[67010]
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO Channel 14 : 4[69010] -> 2[67010] via P2P/indirect/3[67020]
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO Channel 14 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010]
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO comm 0x5653243ebf20 rank 1 nranks 8 cudaDev 1 busId 65020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO comm 0x56373fd65540 rank 7 nranks 8 cudaDev 7 busId 6b020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO comm 0x5597266558e0 rank 5 nranks 8 cudaDev 5 busId 69020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO comm 0x560b829ed140 rank 3 nranks 8 cudaDev 3 busId 67020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO comm 0x562eda579740 rank 2 nranks 8 cudaDev 2 busId 67010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO comm 0x55ba32f6fba0 rank 6 nranks 8 cudaDev 6 busId 6b010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO comm 0x562a4cbc4790 rank 4 nranks 8 cudaDev 4 busId 69010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO comm 0x563f68fa4ee0 rank 0 nranks 8 cudaDev 0 busId 65010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Launch mode Parallel
NCCL version 2.12.10+cuda11.2
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11320:14299 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11319:14460 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11314:14434 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 00/08 :    0   1   3   2
iv-2udaavw4l02thdv8lcrl:11320:14299 [7] NCCL INFO Trees [0] -1/-1/-1->3->1 [1] 1/-1/-1->3->-1 [2] -1/-1/-1->3->1 [3] 1/-1/-1->3->-1 [4] -1/-1/-1->3->1 [5] 1/-1/-1->3->-1 [6] -1/-1/-1->3->1 [7] 1/-1/-1->3->-1
iv-2udaavw4l02thdv8lcrl:11319:14460 [6] NCCL INFO Trees [0] 0/-1/-1->2->-1 [1] -1/-1/-1->2->0 [2] 0/-1/-1->2->-1 [3] -1/-1/-1->2->0 [4] 0/-1/-1->2->-1 [5] -1/-1/-1->2->0 [6] 0/-1/-1->2->-1 [7] -1/-1/-1->2->0
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 01/08 :    0   3   1   2
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Trees [0] 3/-1/-1->1->0 [1] 0/-1/-1->1->3 [2] 3/-1/-1->1->0 [3] 0/-1/-1->1->3 [4] 3/-1/-1->1->0 [5] 0/-1/-1->1->3 [6] 3/-1/-1->1->0 [7] 0/-1/-1->1->3
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 02/08 :    0   2   3   1
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 03/08 :    0   2   1   3
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 04/08 :    0   1   3   2
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 05/08 :    0   3   1   2
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 06/08 :    0   2   3   1
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 07/08 :    0   2   1   3
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Trees [0] 1/-1/-1->0->2 [1] 2/-1/-1->0->1 [2] 1/-1/-1->0->2 [3] 2/-1/-1->0->1 [4] 1/-1/-1->0->2 [5] 2/-1/-1->0->1 [6] 1/-1/-1->0->2 [7] 2/-1/-1->0->1
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 0/-1/-1->2->3 [2] 3/-1/-1->2->0 [3] 0/-1/-1->2->3 [4] 3/-1/-1->2->0 [5] 0/-1/-1->2->3 [6] 3/-1/-1->2->0 [7] 0/-1/-1->2->3
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 00/08 :    0   2   3   1
iv-2udaavw4l02thdv8lcrl:11314:14434 [1] NCCL INFO Trees [0] -1/-1/-1->1->3 [1] 3/-1/-1->1->-1 [2] -1/-1/-1->1->3 [3] 3/-1/-1->1->-1 [4] -1/-1/-1->1->3 [5] 3/-1/-1->1->-1 [6] -1/-1/-1->1->3 [7] 3/-1/-1->1->-1
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Trees [0] 1/-1/-1->3->2 [1] 2/-1/-1->3->1 [2] 1/-1/-1->3->2 [3] 2/-1/-1->3->1 [4] 1/-1/-1->3->2 [5] 2/-1/-1->3->1 [6] 1/-1/-1->3->2 [7] 2/-1/-1->3->1
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 01/08 :    0   2   1   3
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 02/08 :    0   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 03/08 :    0   3   1   2
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 04/08 :    0   2   3   1
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 05/08 :    0   2   1   3
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 06/08 :    0   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 07/08 :    0   3   1   2
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Trees [0] 2/-1/-1->0->-1 [1] -1/-1/-1->0->2 [2] 2/-1/-1->0->-1 [3] -1/-1/-1->0->2 [4] 2/-1/-1->0->-1 [5] -1/-1/-1->0->2 [6] 2/-1/-1->0->-1 [7] -1/-1/-1->0->2
iv-2udaavw4l02thdv8lcrl:11320:14299 [7] NCCL INFO Channel 03 : 3[6b020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14460 [6] NCCL INFO Channel 02 : 2[6b010] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 00 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14299 [7] NCCL INFO Channel 07 : 3[6b020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Channel 01 : 1[69020] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 04 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14460 [6] NCCL INFO Channel 06 : 2[6b010] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Channel 05 : 1[69020] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 02 : 0[65010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14434 [1] NCCL INFO Channel 03 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 06 : 0[65010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Channel 01 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14434 [1] NCCL INFO Channel 07 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Channel 04 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Channel 05 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14299 [7] NCCL INFO Channel 01 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 02 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14460 [6] NCCL INFO Channel 00 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Channel 00 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14299 [7] NCCL INFO Channel 02 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 03 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14460 [6] NCCL INFO Channel 01 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Channel 03 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14299 [7] NCCL INFO Channel 05 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 06 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14460 [6] NCCL INFO Channel 04 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Channel 04 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14299 [7] NCCL INFO Channel 06 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 07 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14460 [6] NCCL INFO Channel 05 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Channel 07 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14434 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Channel 02 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14434 [1] NCCL INFO Channel 02 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Channel 03 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Channel 03 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14434 [1] NCCL INFO Channel 05 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 04 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Channel 06 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Channel 04 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14434 [1] NCCL INFO Channel 06 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 05 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Channel 07 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Channel 07 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14460 [6] NCCL INFO Channel 03 : 2[6b010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 01 : 0[69010] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14299 [7] NCCL INFO Channel 00 : 3[6b020] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Channel 02 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14460 [6] NCCL INFO Channel 07 : 2[6b010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 05 : 0[69010] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14299 [7] NCCL INFO Channel 04 : 3[6b020] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Channel 06 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 03 : 0[65010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14434 [1] NCCL INFO Channel 00 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Channel 01 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Channel 02 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 07 : 0[65010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 01 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14460 [6] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11320:14299 [7] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11314:14434 [1] NCCL INFO Channel 04 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Channel 05 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Channel 06 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 02 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 03 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 05 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14434 [1] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 06 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14460 [6] NCCL INFO Channel 02 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14299 [7] NCCL INFO Channel 00 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Channel 02 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 07 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14460 [6] NCCL INFO Channel 03 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14299 [7] NCCL INFO Channel 03 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Channel 03 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14460 [6] NCCL INFO Channel 06 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14299 [7] NCCL INFO Channel 04 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Channel 05 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14460 [6] NCCL INFO Channel 07 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14299 [7] NCCL INFO Channel 07 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14434 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Channel 06 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 02 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14434 [1] NCCL INFO Channel 03 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Channel 07 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 03 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14434 [1] NCCL INFO Channel 04 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 06 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14434 [1] NCCL INFO Channel 07 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Channel 07 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Channel 01 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 00 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Channel 02 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 01 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Channel 05 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 04 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Channel 06 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Channel 05 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Channel 02 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Channel 04 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Channel 05 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Channel 05 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Channel 06 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Channel 00 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14299 [7] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11320:14299 [7] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11320:14299 [7] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11319:14460 [6] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11319:14460 [6] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11319:14460 [6] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Channel 01 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Channel 03 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Channel 04 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14434 [1] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Channel 05 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14434 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11314:14434 [1] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Channel 07 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Channel 03 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Channel 04 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Channel 05 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Channel 07 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11318:14295 [5] NCCL INFO comm 0x7faa549cd620 rank 1 nranks 4 cudaDev 5 busId 69020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11319:14460 [6] NCCL INFO comm 0x7f7b449c4420 rank 2 nranks 4 cudaDev 6 busId 6b010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO comm 0x7fb5709c43e0 rank 0 nranks 4 cudaDev 4 busId 69010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:14229 [4] NCCL INFO Launch mode Parallel
iv-2udaavw4l02thdv8lcrl:11314:14434 [1] NCCL INFO comm 0x7f14c09c5d00 rank 1 nranks 4 cudaDev 1 busId 65020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO comm 0x7fac7cbe53b0 rank 0 nranks 4 cudaDev 0 busId 65010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:14360 [0] NCCL INFO Launch mode Parallel
iv-2udaavw4l02thdv8lcrl:11315:14279 [2] NCCL INFO comm 0x7f82649c2180 rank 2 nranks 4 cudaDev 2 busId 67010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11316:14453 [3] NCCL INFO comm 0x7f9c089c99e0 rank 3 nranks 4 cudaDev 3 busId 67020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11320:14299 [7] NCCL INFO comm 0x7f1f5c9c4090 rank 3 nranks 4 cudaDev 7 busId 6b020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11320:14274 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11319:14429 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11314:14414 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 00/08 :    0   1   3   2
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Trees [0] 3/-1/-1->1->0 [1] 0/-1/-1->1->3 [2] 3/-1/-1->1->0 [3] 0/-1/-1->1->3 [4] 3/-1/-1->1->0 [5] 0/-1/-1->1->3 [6] 3/-1/-1->1->0 [7] 0/-1/-1->1->3
iv-2udaavw4l02thdv8lcrl:11320:14274 [7] NCCL INFO Trees [0] -1/-1/-1->3->1 [1] 1/-1/-1->3->-1 [2] -1/-1/-1->3->1 [3] 1/-1/-1->3->-1 [4] -1/-1/-1->3->1 [5] 1/-1/-1->3->-1 [6] -1/-1/-1->3->1 [7] 1/-1/-1->3->-1
iv-2udaavw4l02thdv8lcrl:11319:14429 [6] NCCL INFO Trees [0] 0/-1/-1->2->-1 [1] -1/-1/-1->2->0 [2] 0/-1/-1->2->-1 [3] -1/-1/-1->2->0 [4] 0/-1/-1->2->-1 [5] -1/-1/-1->2->0 [6] 0/-1/-1->2->-1 [7] -1/-1/-1->2->0
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 01/08 :    0   3   1   2
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 02/08 :    0   2   3   1
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 03/08 :    0   2   1   3
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 04/08 :    0   1   3   2
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 05/08 :    0   3   1   2
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 06/08 :    0   2   3   1
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 07/08 :    0   2   1   3
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Trees [0] 1/-1/-1->0->2 [1] 2/-1/-1->0->1 [2] 1/-1/-1->0->2 [3] 2/-1/-1->0->1 [4] 1/-1/-1->0->2 [5] 2/-1/-1->0->1 [6] 1/-1/-1->0->2 [7] 2/-1/-1->0->1
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 00/08 :    0   2   3   1
iv-2udaavw4l02thdv8lcrl:11314:14414 [1] NCCL INFO Trees [0] -1/-1/-1->1->3 [1] 3/-1/-1->1->-1 [2] -1/-1/-1->1->3 [3] 3/-1/-1->1->-1 [4] -1/-1/-1->1->3 [5] 3/-1/-1->1->-1 [6] -1/-1/-1->1->3 [7] 3/-1/-1->1->-1
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 0/-1/-1->2->3 [2] 3/-1/-1->2->0 [3] 0/-1/-1->2->3 [4] 3/-1/-1->2->0 [5] 0/-1/-1->2->3 [6] 3/-1/-1->2->0 [7] 0/-1/-1->2->3
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Trees [0] 1/-1/-1->3->2 [1] 2/-1/-1->3->1 [2] 1/-1/-1->3->2 [3] 2/-1/-1->3->1 [4] 1/-1/-1->3->2 [5] 2/-1/-1->3->1 [6] 1/-1/-1->3->2 [7] 2/-1/-1->3->1
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 01/08 :    0   2   1   3
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 02/08 :    0   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 03/08 :    0   3   1   2
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 04/08 :    0   2   3   1
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 05/08 :    0   2   1   3
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 06/08 :    0   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 07/08 :    0   3   1   2
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Trees [0] 2/-1/-1->0->-1 [1] -1/-1/-1->0->2 [2] 2/-1/-1->0->-1 [3] -1/-1/-1->0->2 [4] 2/-1/-1->0->-1 [5] -1/-1/-1->0->2 [6] 2/-1/-1->0->-1 [7] -1/-1/-1->0->2
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 00 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14274 [7] NCCL INFO Channel 03 : 3[6b020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 04 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14429 [6] NCCL INFO Channel 02 : 2[6b010] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Channel 01 : 1[69020] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14274 [7] NCCL INFO Channel 07 : 3[6b020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14429 [6] NCCL INFO Channel 06 : 2[6b010] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Channel 05 : 1[69020] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14414 [1] NCCL INFO Channel 03 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Channel 01 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 02 : 0[65010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14414 [1] NCCL INFO Channel 07 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Channel 04 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Channel 05 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 06 : 0[65010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14274 [7] NCCL INFO Channel 01 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14429 [6] NCCL INFO Channel 00 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 02 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Channel 00 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14274 [7] NCCL INFO Channel 02 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 03 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14429 [6] NCCL INFO Channel 01 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Channel 03 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14274 [7] NCCL INFO Channel 05 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 06 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14429 [6] NCCL INFO Channel 04 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Channel 04 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14274 [7] NCCL INFO Channel 06 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 07 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14429 [6] NCCL INFO Channel 05 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Channel 07 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Channel 02 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14414 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Channel 03 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Channel 03 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14414 [1] NCCL INFO Channel 02 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Channel 06 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Channel 04 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14414 [1] NCCL INFO Channel 05 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 04 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Channel 07 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Channel 07 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14414 [1] NCCL INFO Channel 06 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 05 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14274 [7] NCCL INFO Channel 00 : 3[6b020] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 01 : 0[69010] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Channel 02 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14429 [6] NCCL INFO Channel 03 : 2[6b010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14274 [7] NCCL INFO Channel 04 : 3[6b020] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 05 : 0[69010] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Channel 06 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14429 [6] NCCL INFO Channel 07 : 2[6b010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14274 [7] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 01 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11319:14429 [6] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Channel 02 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14414 [1] NCCL INFO Channel 00 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Channel 01 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 03 : 0[65010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 02 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Channel 06 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14414 [1] NCCL INFO Channel 04 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Channel 05 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 07 : 0[65010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 03 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 05 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14274 [7] NCCL INFO Channel 00 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 06 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14429 [6] NCCL INFO Channel 02 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14274 [7] NCCL INFO Channel 03 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 07 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14429 [6] NCCL INFO Channel 03 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14414 [1] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11320:14274 [7] NCCL INFO Channel 04 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Channel 02 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14429 [6] NCCL INFO Channel 06 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14274 [7] NCCL INFO Channel 07 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14429 [6] NCCL INFO Channel 07 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Channel 03 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Channel 05 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Channel 06 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14414 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 02 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Channel 07 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14414 [1] NCCL INFO Channel 03 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 03 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14414 [1] NCCL INFO Channel 04 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 06 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14414 [1] NCCL INFO Channel 07 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Channel 07 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Channel 01 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 00 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Channel 02 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 01 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Channel 05 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 04 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Channel 05 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Channel 06 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Channel 02 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Channel 05 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Channel 04 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Channel 06 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Channel 05 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Channel 00 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14429 [6] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11319:14429 [6] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11319:14429 [6] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11320:14274 [7] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11320:14274 [7] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11320:14274 [7] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Channel 01 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Channel 03 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Channel 04 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Channel 05 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Channel 07 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11314:14414 [1] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11314:14414 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11314:14414 [1] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Channel 03 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Channel 04 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Channel 05 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Channel 07 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11319:14429 [6] NCCL INFO comm 0x7f7f455ab1f0 rank 2 nranks 4 cudaDev 6 busId 6b010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11320:14274 [7] NCCL INFO comm 0x7f1f695a79b0 rank 3 nranks 4 cudaDev 7 busId 6b020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11318:14337 [5] NCCL INFO comm 0x7faa4d5a2dd0 rank 1 nranks 4 cudaDev 5 busId 69020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO comm 0x7fb5795aa380 rank 0 nranks 4 cudaDev 4 busId 69010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:14226 [4] NCCL INFO Launch mode Parallel
iv-2udaavw4l02thdv8lcrl:11314:14414 [1] NCCL INFO comm 0x7f18a15ac350 rank 1 nranks 4 cudaDev 1 busId 65020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO comm 0x7faca17f4f70 rank 0 nranks 4 cudaDev 0 busId 65010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:14372 [0] NCCL INFO Launch mode Parallel
iv-2udaavw4l02thdv8lcrl:11315:14264 [2] NCCL INFO comm 0x7f82695b20a0 rank 2 nranks 4 cudaDev 2 busId 67010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11316:14431 [3] NCCL INFO comm 0x7f9c195a7960 rank 3 nranks 4 cudaDev 3 busId 67020 - Init COMPLETE
NCCL version 2.12.10+cuda11.2NCCL version 2.12.10+cuda11.2

NCCL version 2.12.10+cuda11.2
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 00/02 :    0   1
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 01/02 :    0   1
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 00/02 :    0   1
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 01/02 :    0   1
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 00/02 :    0   1
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 01/02 :    0   1
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Channel 00/02 :    0   1
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Channel 01/02 :    0   1
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Channel 00 : 1[6b010] -> 0[67010] via direct shared memory
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 00 : 0[67010] -> 1[6b010] via direct shared memory
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Channel 00 : 1[6b020] -> 0[67020] via direct shared memory
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 00 : 0[67020] -> 1[6b020] via direct shared memory
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Channel 01 : 1[6b010] -> 0[67010] via direct shared memory
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 01 : 0[67010] -> 1[6b010] via direct shared memory
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Channel 01 : 1[6b020] -> 0[67020] via direct shared memory
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 00 : 1[69010] -> 0[65010] via direct shared memory
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 01 : 0[67020] -> 1[6b020] via direct shared memory
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 01 : 1[69010] -> 0[65010] via direct shared memory
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 00 : 0[65010] -> 1[69010] via direct shared memory
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 00 : 1[69020] -> 0[65020] via direct shared memory
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 01 : 0[65010] -> 1[69010] via direct shared memory
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 01 : 1[69020] -> 0[65020] via direct shared memory
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Channel 00 : 0[65020] -> 1[69020] via direct shared memory
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Channel 01 : 0[65020] -> 1[69020] via direct shared memory
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO comm 0x7f7f4a67dc70 rank 1 nranks 2 cudaDev 6 busId 6b010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO comm 0x7f825e694020 rank 0 nranks 2 cudaDev 2 busId 67010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Launch mode Parallel
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO comm 0x7f1f566b7830 rank 1 nranks 2 cudaDev 7 busId 6b020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO comm 0x7f9c1268fda0 rank 0 nranks 2 cudaDev 3 busId 67020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Launch mode Parallel
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO comm 0x7f14ca6b3610 rank 0 nranks 2 cudaDev 1 busId 65020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Launch mode Parallel
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO comm 0x7faa6267f070 rank 1 nranks 2 cudaDev 5 busId 69020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO comm 0x7fb56a6825f0 rank 1 nranks 2 cudaDev 4 busId 69010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO comm 0x7fac6a8b5b00 rank 0 nranks 2 cudaDev 0 busId 65010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Launch mode Parallel
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 00/08 :    0   1   3   2
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Trees [0] -1/-1/-1->3->1 [1] 1/-1/-1->3->-1 [2] -1/-1/-1->3->1 [3] 1/-1/-1->3->-1 [4] -1/-1/-1->3->1 [5] 1/-1/-1->3->-1 [6] -1/-1/-1->3->1 [7] 1/-1/-1->3->-1
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Trees [0] 0/-1/-1->2->-1 [1] -1/-1/-1->2->0 [2] 0/-1/-1->2->-1 [3] -1/-1/-1->2->0 [4] 0/-1/-1->2->-1 [5] -1/-1/-1->2->0 [6] 0/-1/-1->2->-1 [7] -1/-1/-1->2->0
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 01/08 :    0   3   1   2
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 02/08 :    0   2   3   1
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 03/08 :    0   2   1   3
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Trees [0] 3/-1/-1->1->0 [1] 0/-1/-1->1->3 [2] 3/-1/-1->1->0 [3] 0/-1/-1->1->3 [4] 3/-1/-1->1->0 [5] 0/-1/-1->1->3 [6] 3/-1/-1->1->0 [7] 0/-1/-1->1->3
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 04/08 :    0   1   3   2
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 05/08 :    0   3   1   2
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 06/08 :    0   2   3   1
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 07/08 :    0   2   1   3
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Trees [0] 1/-1/-1->0->2 [1] 2/-1/-1->0->1 [2] 1/-1/-1->0->2 [3] 2/-1/-1->0->1 [4] 1/-1/-1->0->2 [5] 2/-1/-1->0->1 [6] 1/-1/-1->0->2 [7] 2/-1/-1->0->1
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 00/08 :    0   2   3   1
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Trees [0] -1/-1/-1->1->3 [1] 3/-1/-1->1->-1 [2] -1/-1/-1->1->3 [3] 3/-1/-1->1->-1 [4] -1/-1/-1->1->3 [5] 3/-1/-1->1->-1 [6] -1/-1/-1->1->3 [7] 3/-1/-1->1->-1
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Trees [0] 1/-1/-1->3->2 [1] 2/-1/-1->3->1 [2] 1/-1/-1->3->2 [3] 2/-1/-1->3->1 [4] 1/-1/-1->3->2 [5] 2/-1/-1->3->1 [6] 1/-1/-1->3->2 [7] 2/-1/-1->3->1
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 01/08 :    0   2   1   3
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 0/-1/-1->2->3 [2] 3/-1/-1->2->0 [3] 0/-1/-1->2->3 [4] 3/-1/-1->2->0 [5] 0/-1/-1->2->3 [6] 3/-1/-1->2->0 [7] 0/-1/-1->2->3
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 02/08 :    0   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 03/08 :    0   3   1   2
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 04/08 :    0   2   3   1
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 05/08 :    0   2   1   3
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 06/08 :    0   1   3   2
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 07/08 :    0   3   1   2
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Trees [0] 2/-1/-1->0->-1 [1] -1/-1/-1->0->2 [2] 2/-1/-1->0->-1 [3] -1/-1/-1->0->2 [4] 2/-1/-1->0->-1 [5] -1/-1/-1->0->2 [6] 2/-1/-1->0->-1 [7] -1/-1/-1->0->2
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Channel 03 : 3[6b020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Channel 02 : 2[6b010] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 00 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Channel 07 : 3[6b020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 01 : 1[69020] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Channel 06 : 2[6b010] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 04 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 05 : 1[69020] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Channel 03 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 01 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 02 : 0[65010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Channel 07 : 1[65020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 05 : 3[67020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 04 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 06 : 0[65010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Channel 01 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 02 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Channel 00 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 00 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Channel 02 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 03 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Channel 01 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 03 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Channel 05 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 06 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Channel 04 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 04 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Channel 06 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 07 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Channel 05 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 07 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 02 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 03 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Channel 02 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 03 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 04 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Channel 05 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 06 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 04 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Channel 06 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 07 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 07 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 05 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 01 : 0[69010] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Channel 03 : 2[6b010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Channel 00 : 3[6b020] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 02 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Channel 07 : 2[6b010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 05 : 0[69010] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Channel 04 : 3[6b020] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 06 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Channel 00 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 02 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 01 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 03 : 0[65010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 06 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Channel 04 : 1[65020] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 01 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 05 : 2[67010] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 07 : 0[65010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 02 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 03 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 05 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Connected all rings
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Channel 00 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Channel 02 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 06 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 02 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Channel 03 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Channel 03 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 07 : 0[69010] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 03 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Channel 04 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Channel 06 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 05 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Channel 07 : 3[6b020] -> 1[69020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Channel 07 : 2[6b010] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 06 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 02 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Channel 03 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 07 : 2[67010] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 03 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Channel 04 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 06 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Channel 07 : 1[65020] -> 3[67020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Channel 07 : 0[65010] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 00 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 01 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 01 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 02 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 04 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 05 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Channel 05 : 0[69010] -> 2[6b010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 06 : 1[69020] -> 3[6b020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 02 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 05 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 04 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 06 : 3[67020] -> 1[65020] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Channel 05 : 2[67010] -> 0[65010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 00 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 01 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 03 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 04 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 05 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 03 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Channel 07 : 1[69020] -> 0[69010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 04 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 05 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Channel 07 : 3[67020] -> 2[67010] via P2P/IPC
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO Connected all trees
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer
iv-2udaavw4l02thdv8lcrl:11320:14342 [7] NCCL INFO comm 0x7f1f56793dd0 rank 3 nranks 4 cudaDev 7 busId 6b020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11318:14284 [5] NCCL INFO comm 0x7faa6275e8f0 rank 1 nranks 4 cudaDev 5 busId 69020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO comm 0x7fb56a761900 rank 0 nranks 4 cudaDev 4 busId 69010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11319:14418 [6] NCCL INFO comm 0x7f7f4a769740 rank 2 nranks 4 cudaDev 6 busId 6b010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:14232 [4] NCCL INFO Launch mode Parallel
iv-2udaavw4l02thdv8lcrl:11314:14424 [1] NCCL INFO comm 0x7f14ca7874d0 rank 1 nranks 4 cudaDev 1 busId 65020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO comm 0x7fac6a98f2f0 rank 0 nranks 4 cudaDev 0 busId 65010 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:14369 [0] NCCL INFO Launch mode Parallel
iv-2udaavw4l02thdv8lcrl:11316:14441 [3] NCCL INFO comm 0x7f9c12774b40 rank 3 nranks 4 cudaDev 3 busId 67020 - Init COMPLETE
iv-2udaavw4l02thdv8lcrl:11315:14291 [2] NCCL INFO comm 0x7f825e776b40 rank 2 nranks 4 cudaDev 2 busId 67010 - Init COMPLETE
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/07/05 04:50:07.588, Tesla V100-SXM2-32GB, 470.57.02, 19 %, 5 %, 32510 MiB, 16153 MiB, 16357 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/07/05 04:50:07.589, Tesla V100-SXM2-32GB, 470.57.02, 19 %, 5 %, 32510 MiB, 16153 MiB, 16357 MiB
2022/07/05 04:50:07.591, Tesla V100-SXM2-32GB, 470.57.02, 15 %, 1 %, 32510 MiB, 16233 MiB, 16277 MiB
2022/07/05 04:50:07.592, Tesla V100-SXM2-32GB, 470.57.02, 19 %, 5 %, 32510 MiB, 16153 MiB, 16357 MiB
2022/07/05 04:50:07.592, Tesla V100-SXM2-32GB, 470.57.02, 15 %, 1 %, 32510 MiB, 16233 MiB, 16277 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/07/05 04:50:07.594, Tesla V100-SXM2-32GB, 470.57.02, 7 %, 1 %, 32510 MiB, 16385 MiB, 16125 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/07/05 04:50:07.595, Tesla V100-SXM2-32GB, 470.57.02, 15 %, 1 %, 32510 MiB, 16233 MiB, 16277 MiB
2022/07/05 04:50:07.595, Tesla V100-SXM2-32GB, 470.57.02, 7 %, 1 %, 32510 MiB, 16385 MiB, 16125 MiB
2022/07/05 04:50:07.596, Tesla V100-SXM2-32GB, 470.57.02, 19 %, 5 %, 32510 MiB, 16153 MiB, 16357 MiB
2022/07/05 04:50:07.596, Tesla V100-SXM2-32GB, 470.57.02, 11 %, 6 %, 32510 MiB, 16209 MiB, 16301 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/07/05 04:50:07.597, Tesla V100-SXM2-32GB, 470.57.02, 7 %, 1 %, 32510 MiB, 16385 MiB, 16125 MiB
2022/07/05 04:50:07.598, Tesla V100-SXM2-32GB, 470.57.02, 11 %, 6 %, 32510 MiB, 16209 MiB, 16301 MiB
2022/07/05 04:50:07.597, Tesla V100-SXM2-32GB, 470.57.02, 19 %, 5 %, 32510 MiB, 16153 MiB, 16357 MiB
2022/07/05 04:50:07.600, Tesla V100-SXM2-32GB, 470.57.02, 15 %, 1 %, 32510 MiB, 16233 MiB, 16277 MiB
2022/07/05 04:50:07.600, Tesla V100-SXM2-32GB, 470.57.02, 91 %, 59 %, 32510 MiB, 16225 MiB, 16285 MiB
2022/07/05 04:50:07.600, Tesla V100-SXM2-32GB, 470.57.02, 19 %, 5 %, 32510 MiB, 16153 MiB, 16357 MiB
2022/07/05 04:50:07.602, Tesla V100-SXM2-32GB, 470.57.02, 11 %, 6 %, 32510 MiB, 16209 MiB, 16301 MiB
2022/07/05 04:50:07.602, Tesla V100-SXM2-32GB, 470.57.02, 91 %, 59 %, 32510 MiB, 16225 MiB, 16285 MiB
2022/07/05 04:50:07.602, Tesla V100-SXM2-32GB, 470.57.02, 15 %, 1 %, 32510 MiB, 16233 MiB, 16277 MiB
2022/07/05 04:50:07.604, Tesla V100-SXM2-32GB, 470.57.02, 7 %, 1 %, 32510 MiB, 16385 MiB, 16125 MiB
2022/07/05 04:50:07.605, Tesla V100-SXM2-32GB, 470.57.02, 56 %, 37 %, 32510 MiB, 16129 MiB, 16381 MiB
2022/07/05 04:50:07.606, Tesla V100-SXM2-32GB, 470.57.02, 15 %, 1 %, 32510 MiB, 16233 MiB, 16277 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/07/05 04:50:07.607, Tesla V100-SXM2-32GB, 470.57.02, 91 %, 59 %, 32510 MiB, 16225 MiB, 16285 MiB
2022/07/05 04:50:07.607, Tesla V100-SXM2-32GB, 470.57.02, 56 %, 37 %, 32510 MiB, 16129 MiB, 16381 MiB
2022/07/05 04:50:07.607, Tesla V100-SXM2-32GB, 470.57.02, 7 %, 1 %, 32510 MiB, 16385 MiB, 16125 MiB
2022/07/05 04:50:07.609, Tesla V100-SXM2-32GB, 470.57.02, 11 %, 6 %, 32510 MiB, 16209 MiB, 16301 MiB
2022/07/05 04:50:07.610, Tesla V100-SXM2-32GB, 470.57.02, 49 %, 31 %, 32510 MiB, 16057 MiB, 16453 MiB
2022/07/05 04:50:07.611, Tesla V100-SXM2-32GB, 470.57.02, 7 %, 1 %, 32510 MiB, 16385 MiB, 16125 MiB
2022/07/05 04:50:07.612, Tesla V100-SXM2-32GB, 470.57.02, 56 %, 37 %, 32510 MiB, 16129 MiB, 16381 MiB
2022/07/05 04:50:07.612, Tesla V100-SXM2-32GB, 470.57.02, 49 %, 31 %, 32510 MiB, 16057 MiB, 16453 MiB
2022/07/05 04:50:07.612, Tesla V100-SXM2-32GB, 470.57.02, 11 %, 6 %, 32510 MiB, 16209 MiB, 16301 MiB
2022/07/05 04:50:07.611, Tesla V100-SXM2-32GB, 470.57.02, 19 %, 5 %, 32510 MiB, 16153 MiB, 16357 MiB
2022/07/05 04:50:07.614, Tesla V100-SXM2-32GB, 470.57.02, 91 %, 59 %, 32510 MiB, 16225 MiB, 16285 MiB
2022/07/05 04:50:07.615, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 28 %, 32510 MiB, 16313 MiB, 16197 MiB
2022/07/05 04:50:07.616, Tesla V100-SXM2-32GB, 470.57.02, 11 %, 6 %, 32510 MiB, 16209 MiB, 16301 MiB
2022/07/05 04:50:07.617, Tesla V100-SXM2-32GB, 470.57.02, 49 %, 31 %, 32510 MiB, 16057 MiB, 16453 MiB
2022/07/05 04:50:07.617, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 28 %, 32510 MiB, 16313 MiB, 16197 MiB
2022/07/05 04:50:07.617, Tesla V100-SXM2-32GB, 470.57.02, 91 %, 59 %, 32510 MiB, 16225 MiB, 16285 MiB
2022/07/05 04:50:07.618, Tesla V100-SXM2-32GB, 470.57.02, 15 %, 1 %, 32510 MiB, 16233 MiB, 16277 MiB
2022/07/05 04:50:07.620, Tesla V100-SXM2-32GB, 470.57.02, 56 %, 37 %, 32510 MiB, 16129 MiB, 16381 MiB
2022/07/05 04:50:07.621, Tesla V100-SXM2-32GB, 470.57.02, 91 %, 59 %, 32510 MiB, 16225 MiB, 16285 MiB
2022/07/05 04:50:07.623, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 28 %, 32510 MiB, 16313 MiB, 16197 MiB
2022/07/05 04:50:07.623, Tesla V100-SXM2-32GB, 470.57.02, 56 %, 37 %, 32510 MiB, 16129 MiB, 16381 MiB
2022/07/05 04:50:07.623, Tesla V100-SXM2-32GB, 470.57.02, 7 %, 1 %, 32510 MiB, 16385 MiB, 16125 MiB
2022/07/05 04:50:07.625, Tesla V100-SXM2-32GB, 470.57.02, 49 %, 31 %, 32510 MiB, 16057 MiB, 16453 MiB
2022/07/05 04:50:07.627, Tesla V100-SXM2-32GB, 470.57.02, 56 %, 37 %, 32510 MiB, 16129 MiB, 16381 MiB
2022/07/05 04:50:07.628, Tesla V100-SXM2-32GB, 470.57.02, 49 %, 31 %, 32510 MiB, 16057 MiB, 16453 MiB
2022/07/05 04:50:07.628, Tesla V100-SXM2-32GB, 470.57.02, 11 %, 6 %, 32510 MiB, 16209 MiB, 16301 MiB
2022/07/05 04:50:07.630, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 28 %, 32510 MiB, 16313 MiB, 16197 MiB
2022/07/05 04:50:07.631, Tesla V100-SXM2-32GB, 470.57.02, 49 %, 31 %, 32510 MiB, 16057 MiB, 16453 MiB
2022/07/05 04:50:07.632, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 28 %, 32510 MiB, 16313 MiB, 16197 MiB
2022/07/05 04:50:07.633, Tesla V100-SXM2-32GB, 470.57.02, 91 %, 59 %, 32510 MiB, 16225 MiB, 16285 MiB
2022/07/05 04:50:07.636, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 28 %, 32510 MiB, 16313 MiB, 16197 MiB
2022/07/05 04:50:07.637, Tesla V100-SXM2-32GB, 470.57.02, 56 %, 37 %, 32510 MiB, 16129 MiB, 16381 MiB
2022/07/05 04:50:07.647, Tesla V100-SXM2-32GB, 470.57.02, 49 %, 31 %, 32510 MiB, 16057 MiB, 16453 MiB
2022/07/05 04:50:07.652, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 28 %, 32510 MiB, 16313 MiB, 16197 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/07/05 04:50:13.035, Tesla V100-SXM2-32GB, 470.57.02, 37 %, 23 %, 32510 MiB, 16153 MiB, 16357 MiB
2022/07/05 04:50:13.035, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 28 %, 32510 MiB, 16233 MiB, 16277 MiB
2022/07/05 04:50:13.036, Tesla V100-SXM2-32GB, 470.57.02, 45 %, 29 %, 32510 MiB, 16385 MiB, 16125 MiB
2022/07/05 04:50:13.037, Tesla V100-SXM2-32GB, 470.57.02, 75 %, 48 %, 32510 MiB, 16209 MiB, 16301 MiB
2022/07/05 04:50:13.037, Tesla V100-SXM2-32GB, 470.57.02, 83 %, 54 %, 32510 MiB, 16225 MiB, 16285 MiB
2022/07/05 04:50:13.038, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 66 %, 32510 MiB, 16129 MiB, 16381 MiB
2022/07/05 04:50:13.039, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 67 %, 32510 MiB, 16057 MiB, 16453 MiB
2022/07/05 04:50:13.039, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 64 %, 32510 MiB, 16313 MiB, 16197 MiB
[07/05 04:50:18 lb.utils.events]:  eta: 0:10:56  iteration: 99/220  consumed_samples: 102400  total_loss: 7.287  time: 5.4740 s/iter  data_time: 0.0093 s/iter total_throughput: 187.07 samples/s lr: 8.74e-05  
[07/05 04:59:27 lb.utils.events]:  eta: 0:01:49  iteration: 199/220  consumed_samples: 204800  total_loss: 6.93  time: 5.4835 s/iter  data_time: 0.0094 s/iter total_throughput: 186.74 samples/s lr: 4.81e-06  
[07/05 05:01:17 lb.utils.events]:  eta: 0:00:00  iteration: 219/220  consumed_samples: 225280  total_loss: 6.67  time: 5.4855 s/iter  data_time: 0.0092 s/iter total_throughput: 186.67 samples/s lr: 1.51e-06  
[07/05 05:01:17 lb.engine.hooks]: Overall training speed: 218 iterations in 0:19:55 (5.4856 s / it)
[07/05 05:01:17 lb.engine.hooks]: Total training time: 0:19:55 (0:00:00 on hooks)
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO comm 0x7f8619061990 rank 2 nranks 8 cudaDev 2 busId 67010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO comm 0x7faf759eecc0 rank 0 nranks 8 cudaDev 0 busId 65010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO comm 0x7f9fd8f8b3b0 rank 3 nranks 8 cudaDev 3 busId 67020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO comm 0x7fb9304c6cd0 rank 4 nranks 8 cudaDev 4 busId 69010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO comm 0x7fae141cc3c0 rank 5 nranks 8 cudaDev 5 busId 69020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO comm 0x7f232f75ef40 rank 7 nranks 8 cudaDev 7 busId 6b020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO comm 0x7f187446e790 rank 1 nranks 8 cudaDev 1 busId 65020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO comm 0x7f7f0d008e90 rank 6 nranks 8 cudaDev 6 busId 6b010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO comm 0x562ed9ecd8c0 rank 2 nranks 8 cudaDev 2 busId 67010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO comm 0x563f688fffb0 rank 0 nranks 8 cudaDev 0 busId 65010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO comm 0x562a4c512860 rank 4 nranks 8 cudaDev 4 busId 69010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO comm 0x560b8233a870 rank 3 nranks 8 cudaDev 3 busId 67020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO comm 0x559725fa38f0 rank 5 nranks 8 cudaDev 5 busId 69020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO comm 0x56373f6b8ac0 rank 7 nranks 8 cudaDev 7 busId 6b020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO comm 0x565323d3b730 rank 1 nranks 8 cudaDev 1 busId 65020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO comm 0x55ba328bb9c0 rank 6 nranks 8 cudaDev 6 busId 6b010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO comm 0x562eda17d8d0 rank 2 nranks 8 cudaDev 2 busId 67010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO comm 0x563f68bad9c0 rank 0 nranks 8 cudaDev 0 busId 65010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO comm 0x562a4c7c50f0 rank 4 nranks 8 cudaDev 4 busId 69010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO comm 0x560b825ee780 rank 3 nranks 8 cudaDev 3 busId 67020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO comm 0x5597262578c0 rank 5 nranks 8 cudaDev 5 busId 69020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO comm 0x56373f9699b0 rank 7 nranks 8 cudaDev 7 busId 6b020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO comm 0x565323fedc30 rank 1 nranks 8 cudaDev 1 busId 65020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO comm 0x55ba32b6fd50 rank 6 nranks 8 cudaDev 6 busId 6b010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO comm 0x562eda579740 rank 2 nranks 8 cudaDev 2 busId 67010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO comm 0x563f68fa4ee0 rank 0 nranks 8 cudaDev 0 busId 65010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO comm 0x562a4cbc4790 rank 4 nranks 8 cudaDev 4 busId 69010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO comm 0x560b829ed140 rank 3 nranks 8 cudaDev 3 busId 67020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO comm 0x7fb56a6825f0 rank 1 nranks 2 cudaDev 4 busId 69010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO comm 0x5597266558e0 rank 5 nranks 8 cudaDev 5 busId 69020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO comm 0x7faa6267f070 rank 1 nranks 2 cudaDev 5 busId 69020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO comm 0x56373fd65540 rank 7 nranks 8 cudaDev 7 busId 6b020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO comm 0x7f1f566b7830 rank 1 nranks 2 cudaDev 7 busId 6b020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO comm 0x7fac6a98f2f0 rank 0 nranks 4 cudaDev 0 busId 65010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO comm 0x5653243ebf20 rank 1 nranks 8 cudaDev 1 busId 65020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO comm 0x7f825e776b40 rank 2 nranks 4 cudaDev 2 busId 67010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO comm 0x55ba32f6fba0 rank 6 nranks 8 cudaDev 6 busId 6b010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO comm 0x7f9c12774b40 rank 3 nranks 4 cudaDev 3 busId 67020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO comm 0x7f7f4a67dc70 rank 1 nranks 2 cudaDev 6 busId 6b010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO comm 0x7fb56a761900 rank 0 nranks 4 cudaDev 4 busId 69010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO comm 0x7f1f56793dd0 rank 3 nranks 4 cudaDev 7 busId 6b020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO comm 0x7fac7cbe53b0 rank 0 nranks 4 cudaDev 0 busId 65010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO comm 0x7f14ca7874d0 rank 1 nranks 4 cudaDev 1 busId 65020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO comm 0x7faa6275e8f0 rank 1 nranks 4 cudaDev 5 busId 69020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO comm 0x7f82649c2180 rank 2 nranks 4 cudaDev 2 busId 67010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO comm 0x7f7f4a769740 rank 2 nranks 4 cudaDev 6 busId 6b010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO comm 0x7f1f5c9c4090 rank 3 nranks 4 cudaDev 7 busId 6b020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO comm 0x7faca17f4f70 rank 0 nranks 4 cudaDev 0 busId 65010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO comm 0x7f14c09c5d00 rank 1 nranks 4 cudaDev 1 busId 65020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO comm 0x7f9c089c99e0 rank 3 nranks 4 cudaDev 3 busId 67020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11313:11313 [0] NCCL INFO comm 0x7fac6a8b5b00 rank 0 nranks 2 cudaDev 0 busId 65010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO comm 0x7fb5709c43e0 rank 0 nranks 4 cudaDev 4 busId 69010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO comm 0x7faa549cd620 rank 1 nranks 4 cudaDev 5 busId 69020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO comm 0x7f7b449c4420 rank 2 nranks 4 cudaDev 6 busId 6b010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11320:11320 [7] NCCL INFO comm 0x7f1f695a79b0 rank 3 nranks 4 cudaDev 7 busId 6b020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO comm 0x7f82695b20a0 rank 2 nranks 4 cudaDev 2 busId 67010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO comm 0x7f18a15ac350 rank 1 nranks 4 cudaDev 1 busId 65020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11315:11315 [2] NCCL INFO comm 0x7f825e694020 rank 0 nranks 2 cudaDev 2 busId 67010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11314:11314 [1] NCCL INFO comm 0x7f14ca6b3610 rank 0 nranks 2 cudaDev 1 busId 65020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO comm 0x7f9c195a7960 rank 3 nranks 4 cudaDev 3 busId 67020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11317:11317 [4] NCCL INFO comm 0x7fb5795aa380 rank 0 nranks 4 cudaDev 4 busId 69010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11319:11319 [6] NCCL INFO comm 0x7f7f455ab1f0 rank 2 nranks 4 cudaDev 6 busId 6b010 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11318:11318 [5] NCCL INFO comm 0x7faa4d5a2dd0 rank 1 nranks 4 cudaDev 5 busId 69020 - Destroy COMPLETE
iv-2udaavw4l02thdv8lcrl:11316:11316 [3] NCCL INFO comm 0x7f9c1268fda0 rank 0 nranks 2 cudaDev 3 busId 67020 - Destroy COMPLETE
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************