The module torch.distributed.launch is deprecated and going to be removed in future.Migrate to torch.distributed.run
WARNING:torch.distributed.run:--use_env is deprecated and will be removed in future releases.
 Please read local_rank from `os.environ('LOCAL_RANK')` instead.
INFO:torch.distributed.launcher.api:Starting elastic_operator with launch configs:
  entrypoint       : pretrain_bert.py
  min_nodes        : 1
  max_nodes        : 1
  nproc_per_node   : 8
  run_id           : none
  rdzv_backend     : static
  rdzv_endpoint    : 127.0.0.1:6000
  rdzv_configs     : {'rank': 0, 'timeout': 900}
  max_restarts     : 3
  monitor_interval : 5
  log_dir          : None
  metrics_cfg      : {}

INFO:torch.distributed.elastic.agent.server.local_elastic_agent:log directory set to: /tmp/torchelastic_37c0x0k5/none_h98t4wud
INFO:torch.distributed.elastic.agent.server.api:[default] starting workers for entrypoint: python
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group
/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/utils/store.py:52: FutureWarning: This is an experimental API and will be changed in future.
  warnings.warn(
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:
  restart_count=0
  master_addr=127.0.0.1
  master_port=6000
  group_rank=0
  group_world_size=1
  local_ranks=[0, 1, 2, 3, 4, 5, 6, 7]
  role_ranks=[0, 1, 2, 3, 4, 5, 6, 7]
  global_ranks=[0, 1, 2, 3, 4, 5, 6, 7]
  role_world_sizes=[8, 8, 8, 8, 8, 8, 8, 8]
  global_world_sizes=[8, 8, 8, 8, 8, 8, 8, 8]

INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_37c0x0k5/none_h98t4wud/attempt_0/0/error.json
INFO:torch.distributed.elastic.multiprocessing:Setting worker1 reply file to: /tmp/torchelastic_37c0x0k5/none_h98t4wud/attempt_0/1/error.json
INFO:torch.distributed.elastic.multiprocessing:Setting worker2 reply file to: /tmp/torchelastic_37c0x0k5/none_h98t4wud/attempt_0/2/error.json
INFO:torch.distributed.elastic.multiprocessing:Setting worker3 reply file to: /tmp/torchelastic_37c0x0k5/none_h98t4wud/attempt_0/3/error.json
INFO:torch.distributed.elastic.multiprocessing:Setting worker4 reply file to: /tmp/torchelastic_37c0x0k5/none_h98t4wud/attempt_0/4/error.json
INFO:torch.distributed.elastic.multiprocessing:Setting worker5 reply file to: /tmp/torchelastic_37c0x0k5/none_h98t4wud/attempt_0/5/error.json
INFO:torch.distributed.elastic.multiprocessing:Setting worker6 reply file to: /tmp/torchelastic_37c0x0k5/none_h98t4wud/attempt_0/6/error.json
INFO:torch.distributed.elastic.multiprocessing:Setting worker7 reply file to: /tmp/torchelastic_37c0x0k5/none_h98t4wud/attempt_0/7/error.json
using world size: 8, data-parallel-size: 1, tensor-model-parallel size: 1, pipeline-model-parallel size: 8 
using torch.float16 for parameters ...
Persistent fused layer norm kernel is supported from pytorch v1.11 (nvidia pytorch container paired with v1.11). Defaulting to no_persist_layer_norm=True
------------------------ arguments ------------------------
  accumulate_allreduce_grads_in_fp32 .............. False
  activations_checkpoint_method ................... uniform
  activations_checkpoint_num_layers ............... 1
  adam_beta1 ...................................... 0.9
  adam_beta2 ...................................... 0.999
  adam_eps ........................................ 1e-08
  adlr_autoresume ................................. False
  adlr_autoresume_interval ........................ 1000
  apply_query_key_layer_scaling ................... True
  apply_residual_connection_post_layernorm ........ False
  attention_dropout ............................... 0.1
  attention_softmax_in_fp32 ....................... False
  bert_binary_head ................................ True
  bert_load ....................................... None
  bf16 ............................................ False
  bias_dropout_fusion ............................. True
  bias_gelu_fusion ................................ True
  biencoder_projection_dim ........................ 0
  biencoder_shared_query_context_model ............ False
  block_data_path ................................. None
  classes_fraction ................................ 1.0
  clip_grad ....................................... 1.0
  consumed_train_samples .......................... 0
  consumed_valid_samples .......................... 0
  data_impl ....................................... mmap
  data_parallel_random_init ....................... False
  data_parallel_size .............................. 1
  data_path ....................................... ['/dataset/source/dataset/loss_compara_content_sentence']
  data_per_class_fraction ......................... 1.0
  data_sharding ................................... True
  dataloader_type ................................. single
  DDP_impl ........................................ local
  decoder_seq_length .............................. None
  distribute_checkpointed_activations ............. False
  distributed_backend ............................. nccl
  embedding_path .................................. None
  empty_unused_memory_level ....................... 0
  encoder_seq_length .............................. 512
  eod_mask_loss ................................... False
  eval_interval ................................... 1000
  eval_iters ...................................... 10
  evidence_data_path .............................. None
  exit_duration_in_mins ........................... None
  exit_interval ................................... None
  exit_signal_handler ............................. False
  ffn_hidden_size ................................. 4096
  finetune ........................................ False
  fp16 ............................................ True
  fp16_lm_cross_entropy ........................... False
  fp32_residual_connection ........................ False
  global_batch_size ............................... 1024
  hidden_dropout .................................. 0.1
  hidden_size ..................................... 1024
  hysteresis ...................................... 2
  ict_head_size ................................... None
  ict_load ........................................ None
  img_h ........................................... 224
  img_w ........................................... 224
  indexer_batch_size .............................. 128
  indexer_log_interval ............................ 1000
  inference_batch_times_seqlen_threshold .......... 512
  init_method_std ................................. 0.02
  init_method_xavier_uniform ...................... False
  initial_loss_scale .............................. 4294967296
  kv_channels ..................................... 64
  layernorm_epsilon ............................... 1e-05
  lazy_mpu_init ................................... None
  load ............................................ None
  local_rank ...................................... 0
  log_batch_size_to_tensorboard ................... False
  log_interval .................................... 100
  log_learning_rate_to_tensorboard ................ True
  log_loss_scale_to_tensorboard ................... True
  log_memory_to_tensorboard ....................... False
  log_num_zeros_in_grad ........................... False
  log_params_norm ................................. False
  log_timers_to_tensorboard ....................... False
  log_validation_ppl_to_tensorboard ............... False
  log_world_size_to_tensorboard ................... False
  loss_scale ...................................... None
  loss_scale_window ............................... 1000
  lr .............................................. 0.0001
  lr_decay_iters .................................. 990000
  lr_decay_samples ................................ None
  lr_decay_style .................................. linear
  lr_warmup_fraction .............................. 0.01
  lr_warmup_iters ................................. 0
  lr_warmup_samples ............................... 0
  make_vocab_size_divisible_by .................... 128
  mask_prob ....................................... 0.15
  masked_softmax_fusion ........................... True
  max_position_embeddings ......................... 512
  merge_file ...................................... None
  micro_batch_size ................................ 64
  min_loss_scale .................................. 1.0
  min_lr .......................................... 1e-05
  mmap_warmup ..................................... False
  no_async_tensor_model_parallel_allreduce ........ False
  no_load_optim ................................... None
  no_load_rng ..................................... None
  no_persist_layer_norm ........................... True
  no_save_optim ................................... None
  no_save_rng ..................................... None
  num_attention_heads ............................. 16
  num_channels .................................... 3
  num_classes ..................................... 1000
  num_layers ...................................... 48
  num_layers_per_virtual_pipeline_stage ........... None
  num_workers ..................................... 2
  onnx_safe ....................................... None
  openai_gelu ..................................... False
  optimizer ....................................... adam
  override_lr_scheduler ........................... False
  params_dtype .................................... torch.float16
  patch_dim ....................................... 16
  pipeline_model_parallel_size .................... 8
  pipeline_model_parallel_split_rank .............. None
  query_in_block_prob ............................. 0.1
  rampup_batch_size ............................... None
  rank ............................................ 0
  reset_attention_mask ............................ False
  reset_position_ids .............................. False
  retriever_report_topk_accuracies ................ []
  retriever_score_scaling ......................... False
  retriever_seq_length ............................ 256
  sample_rate ..................................... 1.0
  save ............................................ None
  save_interval ................................... 10000
  scatter_gather_tensors_in_pipeline .............. True
  seed ............................................ 1234
  seq_length ...................................... 512
  sgd_momentum .................................... 0.9
  short_seq_prob .................................. 0.1
  split ........................................... 949,50,1
  tensor_model_parallel_size ...................... 1
  tensorboard_dir ................................. None
  tensorboard_log_interval ........................ 1
  tensorboard_queue_size .......................... 1000
  titles_data_path ................................ None
  tokenizer_type .................................. BertWordPieceLowerCase
  train_iters ..................................... 220
  train_samples ................................... None
  use_checkpoint_lr_scheduler ..................... False
  use_contiguous_buffers_in_local_ddp ............. True
  use_cpu_initialization .......................... None
  use_one_sent_docs ............................... False
  virtual_pipeline_model_parallel_size ............ None
  vocab_extra_ids ................................. 0
  vocab_file ...................................... /dataset/source/dataset/bert-base-chinese-vocab.txt
  weight_decay .................................... 0.01
  world_size ...................................... 8
-------------------- end of arguments ---------------------
setting number of micro-batches to constant 16
> building BertWordPieceLowerCase tokenizer ...
 > padded vocab (size: 21130) with 118 dummy tokens (new size: 21248)
> initializing torch distributed ...
> initializing tensor model parallel with size 1
> initializing pipeline model parallel with size 8
> setting random seeds to 1234 ...
> initializing model parallel cuda seeds on global rank 0, model parallel rank 0, and data parallel rank 0 with model parallel seed: 3952 and data parallel seed: 1234
> compiling dataset index builder ...
[W ProcessGroupNCCL.cpp:1671] Rank 7 using best-guess GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.
make: Entering directory '/dataset/xyn/Megatron-LM/megatron/data'
[W ProcessGroupNCCL.cpp:1671] Rank 2 using best-guess GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.
[W ProcessGroupNCCL.cpp:1671] Rank 3 using best-guess GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.
[W ProcessGroupNCCL.cpp:1671] Rank 6 using best-guess GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.
[W ProcessGroupNCCL.cpp:1671] Rank 1 using best-guess GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.
[W ProcessGroupNCCL.cpp:1671] Rank 5 using best-guess GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.
[W ProcessGroupNCCL.cpp:1671] Rank 4 using best-guess GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.
make: Nothing to be done for 'default'.
make: Leaving directory '/dataset/xyn/Megatron-LM/megatron/data'
>>> done with dataset index builder. Compilation time: 0.039 seconds
> compiling and loading fused kernels ...
Detected CUDA files, patching ldflags
Emitting ninja build file /dataset/xyn/Megatron-LM/megatron/fused_kernels/build/build.ninja...
Building extension module scaled_upper_triang_masked_softmax_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module scaled_upper_triang_masked_softmax_cuda...
Detected CUDA files, patching ldflags
Emitting ninja build file /dataset/xyn/Megatron-LM/megatron/fused_kernels/build/build.ninja...
Building extension module scaled_masked_softmax_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module scaled_masked_softmax_cuda...
Detected CUDA files, patching ldflags
Emitting ninja build file /dataset/xyn/Megatron-LM/megatron/fused_kernels/build/build.ninja...
Building extension module scaled_softmax_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module scaled_softmax_cuda...
Detected CUDA files, patching ldflags
Emitting ninja build file /dataset/xyn/Megatron-LM/megatron/fused_kernels/build/build.ninja...
Building extension module fused_mix_prec_layer_norm_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module fused_mix_prec_layer_norm_cuda...
[W ProcessGroupNCCL.cpp:1671] Rank 0 using best-guess GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device.
iv-ybpu7pvmiu5m57lh5kdd:9981:9981 [0] NCCL INFO Bootstrap : Using eth0:192.168.11.230<0>
iv-ybpu7pvmiu5m57lh5kdd:9981:9981 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
iv-ybpu7pvmiu5m57lh5kdd:9981:9981 [0] NCCL INFO P2P plugin IBext
iv-ybpu7pvmiu5m57lh5kdd:9981:9981 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
iv-ybpu7pvmiu5m57lh5kdd:9981:9981 [0] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.230<0>
iv-ybpu7pvmiu5m57lh5kdd:9981:9981 [0] NCCL INFO Using network IBext
NCCL version 2.10.3+cuda11.4
iv-ybpu7pvmiu5m57lh5kdd:9986:9986 [5] NCCL INFO Bootstrap : Using eth0:192.168.11.230<0>
iv-ybpu7pvmiu5m57lh5kdd:9983:9983 [2] NCCL INFO Bootstrap : Using eth0:192.168.11.230<0>
iv-ybpu7pvmiu5m57lh5kdd:9988:9988 [7] NCCL INFO Bootstrap : Using eth0:192.168.11.230<0>
iv-ybpu7pvmiu5m57lh5kdd:9985:9985 [4] NCCL INFO Bootstrap : Using eth0:192.168.11.230<0>
iv-ybpu7pvmiu5m57lh5kdd:9987:9987 [6] NCCL INFO Bootstrap : Using eth0:192.168.11.230<0>
iv-ybpu7pvmiu5m57lh5kdd:9983:9983 [2] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
iv-ybpu7pvmiu5m57lh5kdd:9988:9988 [7] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
iv-ybpu7pvmiu5m57lh5kdd:9986:9986 [5] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
iv-ybpu7pvmiu5m57lh5kdd:9988:9988 [7] NCCL INFO P2P plugin IBext
iv-ybpu7pvmiu5m57lh5kdd:9983:9983 [2] NCCL INFO P2P plugin IBext
iv-ybpu7pvmiu5m57lh5kdd:9986:9986 [5] NCCL INFO P2P plugin IBext
iv-ybpu7pvmiu5m57lh5kdd:9988:9988 [7] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
iv-ybpu7pvmiu5m57lh5kdd:9983:9983 [2] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
iv-ybpu7pvmiu5m57lh5kdd:9986:9986 [5] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
iv-ybpu7pvmiu5m57lh5kdd:9985:9985 [4] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
iv-ybpu7pvmiu5m57lh5kdd:9985:9985 [4] NCCL INFO P2P plugin IBext
iv-ybpu7pvmiu5m57lh5kdd:9985:9985 [4] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
iv-ybpu7pvmiu5m57lh5kdd:9987:9987 [6] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
iv-ybpu7pvmiu5m57lh5kdd:9987:9987 [6] NCCL INFO P2P plugin IBext
iv-ybpu7pvmiu5m57lh5kdd:9987:9987 [6] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
iv-ybpu7pvmiu5m57lh5kdd:9982:9982 [1] NCCL INFO Bootstrap : Using eth0:192.168.11.230<0>
iv-ybpu7pvmiu5m57lh5kdd:9982:9982 [1] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
iv-ybpu7pvmiu5m57lh5kdd:9982:9982 [1] NCCL INFO P2P plugin IBext
iv-ybpu7pvmiu5m57lh5kdd:9982:9982 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
iv-ybpu7pvmiu5m57lh5kdd:9983:9983 [2] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.230<0>
iv-ybpu7pvmiu5m57lh5kdd:9983:9983 [2] NCCL INFO Using network IBext
iv-ybpu7pvmiu5m57lh5kdd:9985:9985 [4] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.230<0>
iv-ybpu7pvmiu5m57lh5kdd:9985:9985 [4] NCCL INFO Using network IBext
iv-ybpu7pvmiu5m57lh5kdd:9986:9986 [5] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.230<0>
iv-ybpu7pvmiu5m57lh5kdd:9986:9986 [5] NCCL INFO Using network IBext
iv-ybpu7pvmiu5m57lh5kdd:9988:9988 [7] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.230<0>
iv-ybpu7pvmiu5m57lh5kdd:9988:9988 [7] NCCL INFO Using network IBext
iv-ybpu7pvmiu5m57lh5kdd:9987:9987 [6] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.230<0>
iv-ybpu7pvmiu5m57lh5kdd:9987:9987 [6] NCCL INFO Using network IBext
iv-ybpu7pvmiu5m57lh5kdd:9982:9982 [1] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.230<0>
iv-ybpu7pvmiu5m57lh5kdd:9982:9982 [1] NCCL INFO Using network IBext
iv-ybpu7pvmiu5m57lh5kdd:9984:9984 [3] NCCL INFO Bootstrap : Using eth0:192.168.11.230<0>
iv-ybpu7pvmiu5m57lh5kdd:9984:9984 [3] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
iv-ybpu7pvmiu5m57lh5kdd:9984:9984 [3] NCCL INFO P2P plugin IBext
iv-ybpu7pvmiu5m57lh5kdd:9984:9984 [3] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1.
iv-ybpu7pvmiu5m57lh5kdd:9984:9984 [3] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.230<0>
iv-ybpu7pvmiu5m57lh5kdd:9984:9984 [3] NCCL INFO Using network IBext
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3.
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3.
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3.
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3.
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3.
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3.
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3.
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3.
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23.
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23.
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7.
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7.
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23.
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23.
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7.
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7.
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23.
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7.
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23.
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23.
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7.
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7.
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23.
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7.
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Trees [0] -1/-1/-1->7->5 [1] -1/-1/-1->7->5 [2] 5/-1/-1->7->0 [3] 5/-1/-1->7->0 [4] 4/-1/-1->7->6 [5] 6/-1/-1->7->4 [6] -1/-1/-1->7->5 [7] -1/-1/-1->7->5 [8] 5/-1/-1->7->0 [9] 5/-1/-1->7->0 [10] 4/-1/-1->7->6 [11] 6/-1/-1->7->4
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 00/12 :    0   2   3   1   6   4   5   7
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 01/12 :    0   2   3   1   6   4   5   7
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 02/12 :    0   7   5   4   6   1   3   2
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 03/12 :    0   7   5   4   6   1   3   2
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 04/12 :    0   1   2   5   6   7   4   3
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 05/12 :    0   3   4   7   6   5   2   1
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 06/12 :    0   2   3   1   6   4   5   7
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 07/12 :    0   2   3   1   6   4   5   7
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Trees [0] 6/-1/-1->1->3 [1] 6/-1/-1->1->3 [2] 3/-1/-1->1->6 [3] 3/-1/-1->1->6 [4] 2/-1/-1->1->0 [5] -1/-1/-1->1->2 [6] 6/-1/-1->1->3 [7] 6/-1/-1->1->3 [8] 3/-1/-1->1->6 [9] 3/-1/-1->1->6 [10] 2/-1/-1->1->0 [11] -1/-1/-1->1->2
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 08/12 :    0   7   5   4   6   1   3   2
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 3/-1/-1->2->0 [2] -1/-1/-1->2->3 [3] -1/-1/-1->2->3 [4] 5/-1/-1->2->1 [5] 1/-1/-1->2->5 [6] 3/-1/-1->2->0 [7] 3/-1/-1->2->0 [8] -1/-1/-1->2->3 [9] -1/-1/-1->2->3 [10] 5/-1/-1->2->1 [11] 1/-1/-1->2->5
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 09/12 :    0   7   5   4   6   1   3   2
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 10/12 :    0   1   2   5   6   7   4   3
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 11/12 :    0   3   4   7   6   5   2   1
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Trees [0] 2/-1/-1->0->-1 [1] 2/-1/-1->0->-1 [2] 7/-1/-1->0->-1 [3] 7/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 3/-1/-1->0->-1 [6] 2/-1/-1->0->-1 [7] 2/-1/-1->0->-1 [8] 7/-1/-1->0->-1 [9] 7/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 3/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Trees [0] 1/-1/-1->3->2 [1] 1/-1/-1->3->2 [2] 2/-1/-1->3->1 [3] 2/-1/-1->3->1 [4] -1/-1/-1->3->4 [5] 4/-1/-1->3->0 [6] 1/-1/-1->3->2 [7] 1/-1/-1->3->2 [8] 2/-1/-1->3->1 [9] 2/-1/-1->3->1 [10] -1/-1/-1->3->4 [11] 4/-1/-1->3->0
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Trees [0] 5/-1/-1->4->6 [1] 5/-1/-1->4->6 [2] 6/-1/-1->4->5 [3] 6/-1/-1->4->5 [4] 3/-1/-1->4->7 [5] 7/-1/-1->4->3 [6] 5/-1/-1->4->6 [7] 5/-1/-1->4->6 [8] 6/-1/-1->4->5 [9] 6/-1/-1->4->5 [10] 3/-1/-1->4->7 [11] 7/-1/-1->4->3
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Trees [0] 7/-1/-1->5->4 [1] 7/-1/-1->5->4 [2] 4/-1/-1->5->7 [3] 4/-1/-1->5->7 [4] 6/-1/-1->5->2 [5] 2/-1/-1->5->6 [6] 7/-1/-1->5->4 [7] 7/-1/-1->5->4 [8] 4/-1/-1->5->7 [9] 4/-1/-1->5->7 [10] 6/-1/-1->5->2 [11] 2/-1/-1->5->6
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Trees [0] 4/-1/-1->6->1 [1] 4/-1/-1->6->1 [2] 1/-1/-1->6->4 [3] 1/-1/-1->6->4 [4] 7/-1/-1->6->5 [5] 5/-1/-1->6->7 [6] 4/-1/-1->6->1 [7] 4/-1/-1->6->1 [8] 1/-1/-1->6->4 [9] 1/-1/-1->6->4 [10] 7/-1/-1->6->5 [11] 5/-1/-1->6->7
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 04 : 1[65020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 00 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 00 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 10 : 1[65020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 06 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 01 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 04 : 0[65010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 01 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 05 : 3[67020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 07 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 06 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 10 : 0[65010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 06 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 04 : 5[69020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 11 : 3[67020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 07 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 07 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 10 : 5[69020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 04 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 10 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 02 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 03 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 08 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 02 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 09 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 03 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 00 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 06 : 0[65010] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 08 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 01 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 07 : 0[65010] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 09 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 06 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 07 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 05 : 0[65010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 04 : 2[67010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 05 : 4[69010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 02 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 11 : 0[65010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 10 : 2[67010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 11 : 4[69010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 03 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 08 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 09 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 04 : 3[67020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 10 : 3[67020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 04 : 7[6b020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 05 : 5[69020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 10 : 7[6b020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 11 : 5[69020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 00 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 01 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 02 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 02 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 06 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 06 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 03 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 03 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 07 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 07 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 08 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 08 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 09 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 09 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 00 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 01 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 06 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 02 : 0[65010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 07 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 02 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 03 : 0[65010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 03 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 08 : 0[65010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 05 : 2[67010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 08 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 05 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 09 : 0[65010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 11 : 2[67010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 09 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 11 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 05 : 1[65020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 02 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 05 : 6[6b010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 11 : 1[65020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 03 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 11 : 6[6b010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 08 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 04 : 4[69010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 09 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 10 : 4[69010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 05 : 1[65020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 11 : 1[65020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 05 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 02 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 11 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 03 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 02 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 02 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 08 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 03 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 09 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 03 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 08 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 05 : 5[69020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 04 : 3[67020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 08 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 09 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 10 : 3[67020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 11 : 5[69020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 09 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 06 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 05 : 2[67010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 02 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 00 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 07 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 11 : 2[67010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 01 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 03 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 06 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 08 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 07 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 09 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 05 : 3[67020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 04 : 4[69010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 00 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 11 : 3[67020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 10 : 4[69010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 01 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 06 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 07 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 02 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 05 : 7[6b020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 04 : 5[69020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 03 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 11 : 7[6b020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 10 : 5[69020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 08 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 09 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 02 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 00 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 03 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 01 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 08 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 06 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 06 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 09 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 07 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 07 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 04 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 02 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 10 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 03 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 00 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 08 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 01 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 09 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 06 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 04 : 2[67010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 07 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 10 : 2[67010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 04 : 1[65020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 04 : 6[6b010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 10 : 1[65020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 10 : 6[6b010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 06 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 05 : 4[69010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 07 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 11 : 4[69010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 02 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 10 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 02 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 02 : 2[67010] -> 4[69010] via P2P/indirect/5[69020]
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 02 : 3[67020] -> 5[69020] via P2P/indirect/4[69010]
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 10 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 10 : 2[67010] -> 4[69010] via P2P/indirect/5[69020]
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 10 : 3[67020] -> 5[69020] via P2P/indirect/4[69010]
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 03 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 03 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 11 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 11 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 03 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 03 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 11 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 11 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 04 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020]
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 04 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 04 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 04 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 12 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020]
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 04 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 12 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 12 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 04 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 04 : 0[65010] -> 4[69010] via P2P/indirect/3[67020]
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 04 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO Channel 12 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO Channel 12 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 12 : 0[65010] -> 4[69010] via P2P/indirect/3[67020]
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 12 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 05 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 12 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 05 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO Channel 13 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 05 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 13 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 05 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO Channel 13 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 13 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 06 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 06 : 4[69010] -> 2[67010] via P2P/indirect/3[67020]
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO Channel 14 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO Channel 14 : 4[69010] -> 2[67010] via P2P/indirect/3[67020]
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 06 : 5[69020] -> 3[67020] via P2P/indirect/2[67010]
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 06 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO Channel 14 : 5[69020] -> 3[67020] via P2P/indirect/2[67010]
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO Channel 14 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9986:10221 [5] NCCL INFO comm 0x7f1b10008fb0 rank 5 nranks 8 cudaDev 5 busId 69020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9988:10222 [7] NCCL INFO comm 0x7f7f3c008fb0 rank 7 nranks 8 cudaDev 7 busId 6b020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9982:10224 [1] NCCL INFO comm 0x7fabbc008fb0 rank 1 nranks 8 cudaDev 1 busId 65020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9984:10226 [3] NCCL INFO comm 0x7fe774008fb0 rank 3 nranks 8 cudaDev 3 busId 67020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9983:10219 [2] NCCL INFO comm 0x7f440c008fb0 rank 2 nranks 8 cudaDev 2 busId 67010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9985:10220 [4] NCCL INFO comm 0x7f11ec008fb0 rank 4 nranks 8 cudaDev 4 busId 69010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9981:10212 [0] NCCL INFO comm 0x7f4058008fb0 rank 0 nranks 8 cudaDev 0 busId 65010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9987:10223 [6] NCCL INFO comm 0x7fbdb4008fb0 rank 6 nranks 8 cudaDev 6 busId 6b010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9981:9981 [0] NCCL INFO Launch mode Parallel
>>> done with compiling and loading fused kernels. Compilation time: 6.209 seconds
time to initialize megatron (seconds): 6.647
[after megatron is initialized] datetime: 2022-07-05 12:10:06 
building BERT model ...
 > number of parameters on (tensor, pipeline) model parallel rank (0, 4): 75577344
 > number of parameters on (tensor, pipeline) model parallel rank (0, 1): 75577344
 > number of parameters on (tensor, pipeline) model parallel rank (0, 6): 75577344
 > number of parameters on (tensor, pipeline) model parallel rank (0, 5): 75577344
 > number of parameters on (tensor, pipeline) model parallel rank (0, 2): 75577344
 > number of parameters on (tensor, pipeline) model parallel rank (0, 3): 75577344
iv-ybpu7pvmiu5m57lh5kdd:9981:10294 [0] NCCL INFO Channel 00/04 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9988:10295 [7] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0
iv-ybpu7pvmiu5m57lh5kdd:9981:10294 [0] NCCL INFO Channel 01/04 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9981:10294 [0] NCCL INFO Channel 02/04 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9988:10295 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9981:10294 [0] NCCL INFO Channel 03/04 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9981:10294 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9981:10294 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9981:10294 [0] NCCL INFO Channel 00 : 0[65010] -> 1[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10295 [7] NCCL INFO Channel 00 : 1[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10294 [0] NCCL INFO Channel 01 : 0[65010] -> 1[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10295 [7] NCCL INFO Channel 01 : 1[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10294 [0] NCCL INFO Channel 02 : 0[65010] -> 1[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10295 [7] NCCL INFO Channel 02 : 1[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10294 [0] NCCL INFO Channel 03 : 0[65010] -> 1[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10295 [7] NCCL INFO Channel 03 : 1[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10294 [0] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9981:10294 [0] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9981:10294 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9981:10294 [0] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9988:10295 [7] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9988:10295 [7] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9988:10295 [7] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9988:10295 [7] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9981:10294 [0] NCCL INFO comm 0x7f400c008fb0 rank 0 nranks 2 cudaDev 0 busId 65010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9988:10295 [7] NCCL INFO comm 0x7f7ee8008fb0 rank 1 nranks 2 cudaDev 7 busId 6b020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9981:9981 [0] NCCL INFO Launch mode Parallel
 > number of parameters on (tensor, pipeline) model parallel rank (0, 0): 97861632
 > number of parameters on (tensor, pipeline) model parallel rank (0, 7): 99461890
> learning rate decay style: linear
[after model, optimizer, and learning rate scheduler are built] datetime: 2022-07-05 12:10:06 
> building train, validation, and test datasets ...
 > datasets target sizes (minimum size):
    train:      225280
    validation: 10240
    test:       10240
> building train, validation, and test datasets for BERT ...
 > building dataset index ...
    reading sizes...
    reading pointers...
    reading document index...
    creating numpy buffer of mmap...
    creating memory view of numpy buffer...
 > finished creating indexed dataset in 0.005813 seconds
 > indexed dataset stats:
    number of documents: 50000
    number of sentences: 1249934
 > dataset split:
    train:
     document indices in [0, 47450) total of 47450 documents
     sentence indices in [0, 1188464) total of 1188464 sentences
    validation:
     document indices in [47450, 49950) total of 2500 documents
     sentence indices in [1188464, 1248643) total of 60179 sentences
    test:
     document indices in [49950, 50000) total of 50 documents
     sentence indices in [1248643, 1249934) total of 1291 sentences
NCCL version 2.10.3+cuda11.4
NCCL version 2.10.3+cuda11.4
NCCL version 2.10.3+cuda11.4
NCCL version 2.10.3+cuda11.4
NCCL version 2.10.3+cuda11.4
NCCL version 2.10.3+cuda11.4
NCCL version 2.10.3+cuda11.4
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 00/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 01/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 02/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 03/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 04/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 05/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 06/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 07/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 08/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 09/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 10/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 11/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 12/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 13/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 14/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 15/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 16/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 17/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 18/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 19/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 20/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 21/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 22/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 23/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 24/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 25/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 26/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 27/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 28/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 29/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 30/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Channel 31/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 00/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 01/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 02/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 03/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 04/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 05/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 06/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 07/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 08/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 09/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 10/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 11/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 12/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 13/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 14/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 15/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 16/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 17/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 18/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 19/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 20/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 21/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 22/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 23/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 24/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 25/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 26/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 27/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 28/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 29/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 30/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Channel 31/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 00/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 01/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 02/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 03/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 04/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 05/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 06/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 07/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 08/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 09/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 10/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 11/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 12/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 13/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 14/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 15/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 16/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 17/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 18/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 19/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 20/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 21/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 22/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 23/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 24/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 25/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 26/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 27/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 28/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 29/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 30/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Channel 31/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 00/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 01/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 02/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 03/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 04/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 05/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 06/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 07/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 08/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 09/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 10/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 11/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 12/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 13/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 14/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 15/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 16/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 17/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 18/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 19/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 20/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 21/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 22/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 23/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 24/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 25/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 26/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 27/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 28/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 29/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 30/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Channel 31/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 00/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 01/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 02/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 03/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 04/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 05/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 06/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 07/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 08/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 09/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 10/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 11/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 12/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 13/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 14/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 15/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 16/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 17/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 18/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 19/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 20/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 21/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 22/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 23/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 24/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 25/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 26/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 27/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 28/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 29/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 30/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Channel 31/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 00/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 01/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 02/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 03/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 04/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 05/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 06/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 07/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 08/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 09/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 10/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 11/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 12/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 13/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 14/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 15/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 16/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 17/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 18/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 19/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 20/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 21/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 22/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 23/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 24/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 25/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 26/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 27/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 28/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 29/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 30/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Channel 31/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 00/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 01/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 02/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 03/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 04/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 05/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 06/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 07/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 08/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 09/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 10/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 11/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 12/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 13/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 14/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 15/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 16/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 17/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 18/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 19/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 20/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 21/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 22/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 23/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 24/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 25/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 26/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 27/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 28/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 29/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 30/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Channel 31/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 00/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 01/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 02/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 03/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 04/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 05/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 06/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 07/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 08/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 09/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 10/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 11/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 12/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 13/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 14/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 15/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 16/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 17/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 18/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 19/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 20/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 21/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 22/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 23/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 24/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 25/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 26/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 27/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 28/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 29/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 30/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Channel 31/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9987:10301 [6] NCCL INFO comm 0x7fbd30008fb0 rank 0 nranks 1 cudaDev 6 busId 6b010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9982:10305 [1] NCCL INFO comm 0x7fab40008fb0 rank 0 nranks 1 cudaDev 1 busId 65020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9983:10307 [2] NCCL INFO comm 0x7f4398008fb0 rank 0 nranks 1 cudaDev 2 busId 67010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9984:10306 [3] NCCL INFO comm 0x7fe708008fb0 rank 0 nranks 1 cudaDev 3 busId 67020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9988:10311 [7] NCCL INFO comm 0x7f7eac008fb0 rank 0 nranks 1 cudaDev 7 busId 6b020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9981:10315 [0] NCCL INFO comm 0x7f3fd8008fb0 rank 0 nranks 1 cudaDev 0 busId 65010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9985:10317 [4] NCCL INFO comm 0x7f1174008fb0 rank 0 nranks 1 cudaDev 4 busId 69010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9986:10318 [5] NCCL INFO comm 0x7f1a9c008fb0 rank 0 nranks 1 cudaDev 5 busId 69020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Trees [0] 7/-1/-1->5->4 [1] 7/-1/-1->5->4 [2] 4/-1/-1->5->7 [3] 4/-1/-1->5->7 [4] 6/-1/-1->5->2 [5] 2/-1/-1->5->6 [6] 7/-1/-1->5->4 [7] 7/-1/-1->5->4 [8] 4/-1/-1->5->7 [9] 4/-1/-1->5->7 [10] 6/-1/-1->5->2 [11] 2/-1/-1->5->6
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 00/12 :    0   2   3   1   6   4   5   7
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Trees [0] -1/-1/-1->7->5 [1] -1/-1/-1->7->5 [2] 5/-1/-1->7->0 [3] 5/-1/-1->7->0 [4] 4/-1/-1->7->6 [5] 6/-1/-1->7->4 [6] -1/-1/-1->7->5 [7] -1/-1/-1->7->5 [8] 5/-1/-1->7->0 [9] 5/-1/-1->7->0 [10] 4/-1/-1->7->6 [11] 6/-1/-1->7->4
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 01/12 :    0   2   3   1   6   4   5   7
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 02/12 :    0   7   5   4   6   1   3   2
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 03/12 :    0   7   5   4   6   1   3   2
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 04/12 :    0   1   2   5   6   7   4   3
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 05/12 :    0   3   4   7   6   5   2   1
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 06/12 :    0   2   3   1   6   4   5   7
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 07/12 :    0   2   3   1   6   4   5   7
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 08/12 :    0   7   5   4   6   1   3   2
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 09/12 :    0   7   5   4   6   1   3   2
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 10/12 :    0   1   2   5   6   7   4   3
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 11/12 :    0   3   4   7   6   5   2   1
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 3/-1/-1->2->0 [2] -1/-1/-1->2->3 [3] -1/-1/-1->2->3 [4] 5/-1/-1->2->1 [5] 1/-1/-1->2->5 [6] 3/-1/-1->2->0 [7] 3/-1/-1->2->0 [8] -1/-1/-1->2->3 [9] -1/-1/-1->2->3 [10] 5/-1/-1->2->1 [11] 1/-1/-1->2->5
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Trees [0] 4/-1/-1->6->1 [1] 4/-1/-1->6->1 [2] 1/-1/-1->6->4 [3] 1/-1/-1->6->4 [4] 7/-1/-1->6->5 [5] 5/-1/-1->6->7 [6] 4/-1/-1->6->1 [7] 4/-1/-1->6->1 [8] 1/-1/-1->6->4 [9] 1/-1/-1->6->4 [10] 7/-1/-1->6->5 [11] 5/-1/-1->6->7
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Trees [0] 2/-1/-1->0->-1 [1] 2/-1/-1->0->-1 [2] 7/-1/-1->0->-1 [3] 7/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 3/-1/-1->0->-1 [6] 2/-1/-1->0->-1 [7] 2/-1/-1->0->-1 [8] 7/-1/-1->0->-1 [9] 7/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 3/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Trees [0] 1/-1/-1->3->2 [1] 1/-1/-1->3->2 [2] 2/-1/-1->3->1 [3] 2/-1/-1->3->1 [4] -1/-1/-1->3->4 [5] 4/-1/-1->3->0 [6] 1/-1/-1->3->2 [7] 1/-1/-1->3->2 [8] 2/-1/-1->3->1 [9] 2/-1/-1->3->1 [10] -1/-1/-1->3->4 [11] 4/-1/-1->3->0
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Trees [0] 5/-1/-1->4->6 [1] 5/-1/-1->4->6 [2] 6/-1/-1->4->5 [3] 6/-1/-1->4->5 [4] 3/-1/-1->4->7 [5] 7/-1/-1->4->3 [6] 5/-1/-1->4->6 [7] 5/-1/-1->4->6 [8] 6/-1/-1->4->5 [9] 6/-1/-1->4->5 [10] 3/-1/-1->4->7 [11] 7/-1/-1->4->3
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Trees [0] 6/-1/-1->1->3 [1] 6/-1/-1->1->3 [2] 3/-1/-1->1->6 [3] 3/-1/-1->1->6 [4] 2/-1/-1->1->0 [5] -1/-1/-1->1->2 [6] 6/-1/-1->1->3 [7] 6/-1/-1->1->3 [8] 3/-1/-1->1->6 [9] 3/-1/-1->1->6 [10] 2/-1/-1->1->0 [11] -1/-1/-1->1->2
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 00 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 00 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 04 : 1[65020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 04 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 01 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 04 : 0[65010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 01 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 10 : 1[65020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 10 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 06 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 05 : 3[67020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 04 : 5[69020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 06 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 10 : 0[65010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 06 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 07 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 11 : 3[67020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 10 : 5[69020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 07 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 07 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 02 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 03 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 00 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 02 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 08 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 03 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 01 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 09 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 08 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 06 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 06 : 0[65010] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 09 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 07 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 07 : 0[65010] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 02 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 05 : 4[69010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 04 : 2[67010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 05 : 0[65010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 03 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 11 : 4[69010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 11 : 0[65010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 10 : 2[67010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 08 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 09 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 04 : 3[67020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 05 : 5[69020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 04 : 7[6b020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 10 : 3[67020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 11 : 5[69020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 10 : 7[6b020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 00 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 02 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 02 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 01 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 03 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 03 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 06 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 08 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 06 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 08 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 07 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 09 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 07 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 09 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 00 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 01 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 06 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 02 : 0[65010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 02 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 07 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 03 : 0[65010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 03 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 05 : 2[67010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 08 : 0[65010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 05 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 08 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 11 : 2[67010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 09 : 0[65010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 09 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 11 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 05 : 6[6b010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 05 : 1[65020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 02 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 11 : 1[65020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 11 : 6[6b010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 03 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 04 : 4[69010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 08 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 10 : 4[69010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 09 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 05 : 1[65020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 11 : 1[65020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 05 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 11 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 02 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 03 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 02 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 02 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 08 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 03 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 03 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 09 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 05 : 5[69020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 08 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 08 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 04 : 3[67020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 11 : 5[69020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 09 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 09 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 10 : 3[67020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 06 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 02 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 05 : 2[67010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 00 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 07 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 03 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 11 : 2[67010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 01 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 08 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 06 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 09 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 07 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 00 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 04 : 4[69010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 05 : 3[67020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 01 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 10 : 4[69010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 11 : 3[67020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 06 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 07 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 04 : 5[69020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 05 : 7[6b020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 02 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 10 : 5[69020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 11 : 7[6b020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 03 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 08 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 00 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 09 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 02 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 01 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 03 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 06 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 06 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 07 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 08 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 07 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 09 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 02 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 03 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 00 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 04 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 08 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 01 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 10 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 04 : 2[67010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 09 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 06 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 10 : 2[67010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 07 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 04 : 6[6b010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 04 : 1[65020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 10 : 1[65020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 10 : 6[6b010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 05 : 4[69010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 06 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 11 : 4[69010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 07 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 02 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 02 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 10 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 10 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 02 : 2[67010] -> 4[69010] via P2P/indirect/5[69020]
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 02 : 3[67020] -> 5[69020] via P2P/indirect/4[69010]
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 10 : 2[67010] -> 4[69010] via P2P/indirect/5[69020]
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 10 : 3[67020] -> 5[69020] via P2P/indirect/4[69010]
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 03 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 11 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 03 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 03 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 11 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 03 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 11 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 04 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 04 : 0[65010] -> 4[69010] via P2P/indirect/3[67020]
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 12 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 04 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 12 : 0[65010] -> 4[69010] via P2P/indirect/3[67020]
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 11 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO Channel 12 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 04 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 04 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 04 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020]
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 12 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO Channel 12 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 12 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020]
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 04 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 04 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 12 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 12 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 05 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 05 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 13 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 05 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO Channel 13 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 05 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO Channel 13 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 13 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 06 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO Channel 14 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 06 : 4[69010] -> 2[67010] via P2P/indirect/3[67020]
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 06 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 06 : 5[69020] -> 3[67020] via P2P/indirect/2[67010]
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO Channel 14 : 4[69010] -> 2[67010] via P2P/indirect/3[67020]
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO Channel 14 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO Channel 14 : 5[69020] -> 3[67020] via P2P/indirect/2[67010]
iv-ybpu7pvmiu5m57lh5kdd:9983:10336 [2] NCCL INFO comm 0x7f43980c8e80 rank 2 nranks 8 cudaDev 2 busId 67010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9984:10337 [3] NCCL INFO comm 0x7fe7080c8e80 rank 3 nranks 8 cudaDev 3 busId 67020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9986:10340 [5] NCCL INFO comm 0x7f1a9c0a5d20 rank 5 nranks 8 cudaDev 5 busId 69020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9985:10338 [4] NCCL INFO comm 0x7f11740a5d20 rank 4 nranks 8 cudaDev 4 busId 69010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9981:10332 [0] NCCL INFO comm 0x7f3fd0008fb0 rank 0 nranks 8 cudaDev 0 busId 65010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9988:10333 [7] NCCL INFO comm 0x7f7eac0ad1e0 rank 7 nranks 8 cudaDev 7 busId 6b020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9982:10335 [1] NCCL INFO comm 0x7fab400c8e80 rank 1 nranks 8 cudaDev 1 busId 65020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9987:10334 [6] NCCL INFO comm 0x7fbd300acd60 rank 6 nranks 8 cudaDev 6 busId 6b010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9981:9981 [0] NCCL INFO Launch mode Parallel
 > loading indexed mapping from /dataset/source/dataset/loss_compara_content_sentence_train_indexmap_225280mns_509msl_0.10ssp_1234s.npy
    loaded indexed file in 0.003 seconds
    total number of samples: 226136
 > loading indexed mapping from /dataset/source/dataset/loss_compara_content_sentence_valid_indexmap_10240mns_509msl_0.10ssp_1234s.npy
    loaded indexed file in 0.001 seconds
    total number of samples: 11758
 > loading indexed mapping from /dataset/source/dataset/loss_compara_content_sentence_test_indexmap_10240mns_509msl_0.10ssp_1234s.npy
    loaded indexed file in 0.001 seconds
    total number of samples: 10295
> finished creating BERT datasets ...
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 00/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 01/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 02/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 03/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 04/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 05/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 06/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 07/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 08/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 09/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 10/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 11/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 12/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 13/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 14/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 15/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 16/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 17/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 18/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 19/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 20/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 21/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 22/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 23/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 24/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 25/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 26/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 27/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 28/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 29/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 30/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Channel 31/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 00/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 01/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 02/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 03/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 04/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 05/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 06/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 07/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 08/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 09/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 10/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 11/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 12/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 13/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 14/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 15/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 16/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 17/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 18/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 19/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 20/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 21/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 22/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 23/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 24/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 25/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 26/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 27/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 28/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 29/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 30/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Channel 31/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 00/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 01/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 02/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 03/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 04/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 05/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 06/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 07/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 08/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 09/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 10/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 11/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 12/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 13/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 14/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 15/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 16/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 17/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 18/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 19/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 20/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 21/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 22/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 23/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 24/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 25/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 26/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 27/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 28/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 29/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 30/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Channel 31/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 00/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 01/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 02/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 03/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 04/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 05/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 06/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 07/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 08/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 09/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 10/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 11/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 12/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 13/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 14/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 15/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 16/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 17/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 18/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 19/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 20/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 21/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 22/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 23/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 24/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 25/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 26/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 27/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 28/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 29/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 30/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Channel 31/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 00/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 01/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 02/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 03/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 04/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 05/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 06/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 07/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 08/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 09/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 10/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 11/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 12/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 13/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 14/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 15/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 16/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 17/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 18/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 19/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 20/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 21/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 22/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 23/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 24/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 25/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 26/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 27/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 28/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 29/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 30/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Channel 31/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 00/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 01/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 02/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 03/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 04/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 05/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 06/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 07/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 08/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 09/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 10/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 11/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 12/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 13/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 14/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 15/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 16/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 17/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 18/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 19/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 20/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 21/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 22/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 23/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 24/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 25/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 26/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 27/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 28/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 29/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 30/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Channel 31/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 00/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 01/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 02/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 03/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 04/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 05/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 06/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 07/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 08/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 09/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 10/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 11/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 12/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 13/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 14/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 15/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 16/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 17/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 18/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 19/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 20/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 21/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 22/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 23/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 24/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 25/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 26/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 27/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 28/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 29/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 30/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Channel 31/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 00/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 01/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 02/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 03/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 04/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 05/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 06/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 07/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 08/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 09/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 10/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 11/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 12/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 13/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 14/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 15/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 16/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 17/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 18/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 19/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 20/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 21/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 22/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 23/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 24/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 25/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 26/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 27/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 28/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 29/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 30/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Channel 31/32 :    0
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9986:10364 [5] NCCL INFO comm 0x7f1a58008fb0 rank 0 nranks 1 cudaDev 5 busId 69020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9987:10358 [6] NCCL INFO comm 0x7fbce4008fb0 rank 0 nranks 1 cudaDev 6 busId 6b010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9981:10363 [0] NCCL INFO comm 0x7f3f8c008fb0 rank 0 nranks 1 cudaDev 0 busId 65010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9988:10366 [7] NCCL INFO comm 0x7f7e64008fb0 rank 0 nranks 1 cudaDev 7 busId 6b020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9982:10373 [1] NCCL INFO comm 0x7fab00008fb0 rank 0 nranks 1 cudaDev 1 busId 65020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9984:10362 [3] NCCL INFO comm 0x7fe6c4008fb0 rank 0 nranks 1 cudaDev 3 busId 67020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9983:10376 [2] NCCL INFO comm 0x7f4358008fb0 rank 0 nranks 1 cudaDev 2 busId 67010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9985:10379 [4] NCCL INFO comm 0x7f1130008fb0 rank 0 nranks 1 cudaDev 4 busId 69010 - Init COMPLETE
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[after dataloaders are built] datetime: 2022-07-05 12:10:08 
done with setup ...
time (ms) | model-and-optimizer-setup: 225.03 | train/valid/test-data-iterators-setup: 2273.21
training ...
[before the start of training step] datetime: 2022-07-05 12:10:08 
/dataset/xyn/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at  /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.)
  output = bias_dropout_add_func(
iv-ybpu7pvmiu5m57lh5kdd:9982:12612 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
iv-ybpu7pvmiu5m57lh5kdd:9981:12611 [0] NCCL INFO Channel 00/02 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9982:12612 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9981:12611 [0] NCCL INFO Channel 01/02 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9981:12611 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9981:12611 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9981:12611 [0] NCCL INFO Channel 00 : 0[65010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:12611 [0] NCCL INFO Channel 01 : 0[65010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12612 [1] NCCL INFO Channel 00 : 1[65020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12612 [1] NCCL INFO Channel 01 : 1[65020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:12611 [0] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9981:12611 [0] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9981:12611 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9981:12611 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9982:12612 [1] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9982:12612 [1] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9982:12612 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9982:12612 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9981:12611 [0] NCCL INFO comm 0x7f3ec0008fb0 rank 0 nranks 2 cudaDev 0 busId 65010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9982:12612 [1] NCCL INFO comm 0x7faad4008fb0 rank 1 nranks 2 cudaDev 1 busId 65020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9981:9981 [0] NCCL INFO Launch mode Parallel
/dataset/xyn/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at  /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.)
  output = bias_dropout_add_func(
iv-ybpu7pvmiu5m57lh5kdd:9982:12618 [1] NCCL INFO Channel 00/02 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9983:12619 [2] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
iv-ybpu7pvmiu5m57lh5kdd:9982:12618 [1] NCCL INFO Channel 01/02 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9983:12619 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9982:12618 [1] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9982:12618 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9982:12618 [1] NCCL INFO Channel 00 : 0[65020] -> 1[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12619 [2] NCCL INFO Channel 00 : 1[67010] -> 0[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12618 [1] NCCL INFO Channel 01 : 0[65020] -> 1[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12619 [2] NCCL INFO Channel 01 : 1[67010] -> 0[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12618 [1] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9982:12618 [1] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9982:12618 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9982:12618 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9983:12619 [2] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9983:12619 [2] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9983:12619 [2] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9983:12619 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9982:12618 [1] NCCL INFO comm 0x7faa0c008fb0 rank 0 nranks 2 cudaDev 1 busId 65020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9983:12619 [2] NCCL INFO comm 0x7f4328008fb0 rank 1 nranks 2 cudaDev 2 busId 67010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9982:9982 [1] NCCL INFO Launch mode Parallel
/dataset/xyn/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at  /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.)
  output = bias_dropout_add_func(
iv-ybpu7pvmiu5m57lh5kdd:9984:12639 [3] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0
iv-ybpu7pvmiu5m57lh5kdd:9983:12638 [2] NCCL INFO Channel 00/04 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9983:12638 [2] NCCL INFO Channel 01/04 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9984:12639 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9983:12638 [2] NCCL INFO Channel 02/04 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9983:12638 [2] NCCL INFO Channel 03/04 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9983:12638 [2] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9983:12638 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9983:12638 [2] NCCL INFO Channel 00 : 0[67010] -> 1[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12639 [3] NCCL INFO Channel 00 : 1[67020] -> 0[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12638 [2] NCCL INFO Channel 01 : 0[67010] -> 1[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12639 [3] NCCL INFO Channel 01 : 1[67020] -> 0[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12638 [2] NCCL INFO Channel 02 : 0[67010] -> 1[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12639 [3] NCCL INFO Channel 02 : 1[67020] -> 0[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12638 [2] NCCL INFO Channel 03 : 0[67010] -> 1[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12639 [3] NCCL INFO Channel 03 : 1[67020] -> 0[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12638 [2] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9983:12638 [2] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9983:12638 [2] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9983:12638 [2] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9984:12639 [3] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9984:12639 [3] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9984:12639 [3] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9984:12639 [3] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9983:12638 [2] NCCL INFO comm 0x7f4268008fb0 rank 0 nranks 2 cudaDev 2 busId 67010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9984:12639 [3] NCCL INFO comm 0x7fe694008fb0 rank 1 nranks 2 cudaDev 3 busId 67020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9983:9983 [2] NCCL INFO Launch mode Parallel
/dataset/xyn/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at  /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.)
  output = bias_dropout_add_func(
iv-ybpu7pvmiu5m57lh5kdd:9985:12646 [4] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
iv-ybpu7pvmiu5m57lh5kdd:9984:12645 [3] NCCL INFO Channel 00/02 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9984:12645 [3] NCCL INFO Channel 01/02 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9985:12646 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9984:12645 [3] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9984:12645 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9984:12645 [3] NCCL INFO Channel 00 : 0[67020] -> 1[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12646 [4] NCCL INFO Channel 00 : 1[69010] -> 0[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12645 [3] NCCL INFO Channel 01 : 0[67020] -> 1[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12646 [4] NCCL INFO Channel 01 : 1[69010] -> 0[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12645 [3] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9984:12645 [3] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9984:12645 [3] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9984:12645 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9985:12646 [4] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9985:12646 [4] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9985:12646 [4] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9985:12646 [4] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9984:12645 [3] NCCL INFO comm 0x7fe5d0008fb0 rank 0 nranks 2 cudaDev 3 busId 67020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9985:12646 [4] NCCL INFO comm 0x7f1104008fb0 rank 1 nranks 2 cudaDev 4 busId 69010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9984:9984 [3] NCCL INFO Launch mode Parallel
/dataset/xyn/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at  /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.)
  output = bias_dropout_add_func(
iv-ybpu7pvmiu5m57lh5kdd:9985:12652 [4] NCCL INFO Channel 00/04 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9986:12653 [5] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0
iv-ybpu7pvmiu5m57lh5kdd:9985:12652 [4] NCCL INFO Channel 01/04 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9985:12652 [4] NCCL INFO Channel 02/04 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9986:12653 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9985:12652 [4] NCCL INFO Channel 03/04 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9985:12652 [4] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9985:12652 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9985:12652 [4] NCCL INFO Channel 00 : 0[69010] -> 1[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12652 [4] NCCL INFO Channel 01 : 0[69010] -> 1[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12653 [5] NCCL INFO Channel 00 : 1[69020] -> 0[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12652 [4] NCCL INFO Channel 02 : 0[69010] -> 1[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12653 [5] NCCL INFO Channel 01 : 1[69020] -> 0[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12652 [4] NCCL INFO Channel 03 : 0[69010] -> 1[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12653 [5] NCCL INFO Channel 02 : 1[69020] -> 0[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12653 [5] NCCL INFO Channel 03 : 1[69020] -> 0[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12652 [4] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9985:12652 [4] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9985:12652 [4] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9985:12652 [4] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9986:12653 [5] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9986:12653 [5] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9986:12653 [5] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9986:12653 [5] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9985:12652 [4] NCCL INFO comm 0x7f1040008fb0 rank 0 nranks 2 cudaDev 4 busId 69010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9986:12653 [5] NCCL INFO comm 0x7f1a2c008fb0 rank 1 nranks 2 cudaDev 5 busId 69020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9985:9985 [4] NCCL INFO Launch mode Parallel
/dataset/xyn/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at  /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.)
  output = bias_dropout_add_func(
iv-ybpu7pvmiu5m57lh5kdd:9986:12659 [5] NCCL INFO Channel 00/02 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9987:12660 [6] NCCL INFO Trees [0] 0/-1/-1->1->-1 [1] 0/-1/-1->1->-1
iv-ybpu7pvmiu5m57lh5kdd:9986:12659 [5] NCCL INFO Channel 01/02 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9987:12660 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9986:12659 [5] NCCL INFO Trees [0] -1/-1/-1->0->1 [1] -1/-1/-1->0->1
iv-ybpu7pvmiu5m57lh5kdd:9986:12659 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9987:12660 [6] NCCL INFO Channel 00 : 1[6b010] -> 0[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12660 [6] NCCL INFO Channel 01 : 1[6b010] -> 0[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12659 [5] NCCL INFO Channel 00 : 0[69020] -> 1[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12659 [5] NCCL INFO Channel 01 : 0[69020] -> 1[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12659 [5] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9986:12659 [5] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9986:12659 [5] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9986:12659 [5] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9987:12660 [6] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9987:12660 [6] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9987:12660 [6] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9987:12660 [6] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9986:12659 [5] NCCL INFO comm 0x7f1964008fb0 rank 0 nranks 2 cudaDev 5 busId 69020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9987:12660 [6] NCCL INFO comm 0x7fbcb4008fb0 rank 1 nranks 2 cudaDev 6 busId 6b010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9986:9986 [5] NCCL INFO Launch mode Parallel
/dataset/xyn/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at  /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.)
  output = bias_dropout_add_func(
iv-ybpu7pvmiu5m57lh5kdd:9988:12680 [7] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
iv-ybpu7pvmiu5m57lh5kdd:9987:12679 [6] NCCL INFO Channel 00/02 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9987:12679 [6] NCCL INFO Channel 01/02 :    0   1
iv-ybpu7pvmiu5m57lh5kdd:9988:12680 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9987:12679 [6] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9987:12679 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9988:12680 [7] NCCL INFO Channel 00 : 1[6b020] -> 0[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12679 [6] NCCL INFO Channel 00 : 0[6b010] -> 1[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12680 [7] NCCL INFO Channel 01 : 1[6b020] -> 0[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12679 [6] NCCL INFO Channel 01 : 0[6b010] -> 1[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12679 [6] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9987:12679 [6] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9987:12679 [6] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9987:12679 [6] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9988:12680 [7] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9988:12680 [7] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9988:12680 [7] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9988:12680 [7] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9987:12679 [6] NCCL INFO comm 0x7fbbf4008fb0 rank 0 nranks 2 cudaDev 6 busId 6b010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9988:12680 [7] NCCL INFO comm 0x7f7e30008fb0 rank 1 nranks 2 cudaDev 7 busId 6b020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9987:9987 [6] NCCL INFO Launch mode Parallel
/dataset/xyn/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at  /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.)
  output = bias_dropout_add_func(
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Trees [0] -1/-1/-1->7->5 [1] -1/-1/-1->7->5 [2] 5/-1/-1->7->0 [3] 5/-1/-1->7->0 [4] 4/-1/-1->7->6 [5] 6/-1/-1->7->4 [6] -1/-1/-1->7->5 [7] -1/-1/-1->7->5 [8] 5/-1/-1->7->0 [9] 5/-1/-1->7->0 [10] 4/-1/-1->7->6 [11] 6/-1/-1->7->4
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 00/12 :    0   2   3   1   6   4   5   7
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 01/12 :    0   2   3   1   6   4   5   7
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 02/12 :    0   7   5   4   6   1   3   2
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 03/12 :    0   7   5   4   6   1   3   2
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 04/12 :    0   1   2   5   6   7   4   3
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 05/12 :    0   3   4   7   6   5   2   1
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 06/12 :    0   2   3   1   6   4   5   7
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 07/12 :    0   2   3   1   6   4   5   7
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 08/12 :    0   7   5   4   6   1   3   2
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 09/12 :    0   7   5   4   6   1   3   2
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 10/12 :    0   1   2   5   6   7   4   3
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 11/12 :    0   3   4   7   6   5   2   1
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Trees [0] 2/-1/-1->0->-1 [1] 2/-1/-1->0->-1 [2] 7/-1/-1->0->-1 [3] 7/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 3/-1/-1->0->-1 [6] 2/-1/-1->0->-1 [7] 2/-1/-1->0->-1 [8] 7/-1/-1->0->-1 [9] 7/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 3/-1/-1->0->-1
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 3/-1/-1->2->0 [2] -1/-1/-1->2->3 [3] -1/-1/-1->2->3 [4] 5/-1/-1->2->1 [5] 1/-1/-1->2->5 [6] 3/-1/-1->2->0 [7] 3/-1/-1->2->0 [8] -1/-1/-1->2->3 [9] -1/-1/-1->2->3 [10] 5/-1/-1->2->1 [11] 1/-1/-1->2->5
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Trees [0] 5/-1/-1->4->6 [1] 5/-1/-1->4->6 [2] 6/-1/-1->4->5 [3] 6/-1/-1->4->5 [4] 3/-1/-1->4->7 [5] 7/-1/-1->4->3 [6] 5/-1/-1->4->6 [7] 5/-1/-1->4->6 [8] 6/-1/-1->4->5 [9] 6/-1/-1->4->5 [10] 3/-1/-1->4->7 [11] 7/-1/-1->4->3
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Trees [0] 7/-1/-1->5->4 [1] 7/-1/-1->5->4 [2] 4/-1/-1->5->7 [3] 4/-1/-1->5->7 [4] 6/-1/-1->5->2 [5] 2/-1/-1->5->6 [6] 7/-1/-1->5->4 [7] 7/-1/-1->5->4 [8] 4/-1/-1->5->7 [9] 4/-1/-1->5->7 [10] 6/-1/-1->5->2 [11] 2/-1/-1->5->6
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Trees [0] 6/-1/-1->1->3 [1] 6/-1/-1->1->3 [2] 3/-1/-1->1->6 [3] 3/-1/-1->1->6 [4] 2/-1/-1->1->0 [5] -1/-1/-1->1->2 [6] 6/-1/-1->1->3 [7] 6/-1/-1->1->3 [8] 3/-1/-1->1->6 [9] 3/-1/-1->1->6 [10] 2/-1/-1->1->0 [11] -1/-1/-1->1->2
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Trees [0] 4/-1/-1->6->1 [1] 4/-1/-1->6->1 [2] 1/-1/-1->6->4 [3] 1/-1/-1->6->4 [4] 7/-1/-1->6->5 [5] 5/-1/-1->6->7 [6] 4/-1/-1->6->1 [7] 4/-1/-1->6->1 [8] 1/-1/-1->6->4 [9] 1/-1/-1->6->4 [10] 7/-1/-1->6->5 [11] 5/-1/-1->6->7
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Trees [0] 1/-1/-1->3->2 [1] 1/-1/-1->3->2 [2] 2/-1/-1->3->1 [3] 2/-1/-1->3->1 [4] -1/-1/-1->3->4 [5] 4/-1/-1->3->0 [6] 1/-1/-1->3->2 [7] 1/-1/-1->3->2 [8] 2/-1/-1->3->1 [9] 2/-1/-1->3->1 [10] -1/-1/-1->3->4 [11] 4/-1/-1->3->0
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 00 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 04 : 1[65020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 04 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 00 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 01 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 10 : 1[65020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 10 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 04 : 0[65010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 04 : 5[69020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 01 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 06 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 06 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 10 : 0[65010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 05 : 3[67020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 10 : 5[69020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 06 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 07 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 07 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 11 : 3[67020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 07 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 02 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 03 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 00 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 02 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 08 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 01 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 03 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 09 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 06 : 0[65010] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 06 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 08 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 07 : 0[65010] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 07 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 09 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 04 : 2[67010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 05 : 0[65010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 05 : 4[69010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 02 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 10 : 2[67010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 11 : 0[65010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 11 : 4[69010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 03 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 08 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 09 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 05 : 5[69020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 04 : 3[67020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 04 : 7[6b020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 11 : 5[69020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 10 : 3[67020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 10 : 7[6b020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 00 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 02 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 02 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 01 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 03 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 03 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 06 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 06 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 08 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 08 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 07 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 07 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 09 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 09 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 00 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 01 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 06 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 02 : 0[65010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 02 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 07 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 03 : 0[65010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 03 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 05 : 2[67010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 08 : 0[65010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 05 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 08 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 11 : 2[67010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 09 : 0[65010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 09 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 11 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 05 : 1[65020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 02 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 05 : 6[6b010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 11 : 1[65020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 03 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 11 : 6[6b010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 04 : 4[69010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 08 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 10 : 4[69010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 09 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 05 : 1[65020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 11 : 1[65020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Connected all rings
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 05 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 11 : 6[6b010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 02 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 03 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 02 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 02 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 08 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 03 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 03 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 09 : 7[6b020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 05 : 5[69020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 04 : 3[67020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 08 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 08 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 10 : 3[67020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 09 : 2[67010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 11 : 5[69020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 09 : 4[69010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 06 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 05 : 2[67010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 00 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 07 : 1[65020] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 02 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 11 : 2[67010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 01 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 03 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 06 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 08 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 07 : 4[69010] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 09 : 5[69020] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 00 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 05 : 3[67020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 04 : 4[69010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 01 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 11 : 3[67020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 10 : 4[69010] -> 7[6b020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 06 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 07 : 6[6b010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 02 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 05 : 7[6b020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 04 : 5[69020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 03 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 11 : 7[6b020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 10 : 5[69020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 08 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 09 : 3[67020] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 02 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 00 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 03 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 01 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 08 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 06 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 06 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 09 : 1[65020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 07 : 7[6b020] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 07 : 2[67010] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 02 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 04 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 03 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 10 : 7[6b020] -> 6[6b010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 08 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 00 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 09 : 6[6b010] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 01 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 06 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 04 : 2[67010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 07 : 5[69020] -> 4[69010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 10 : 2[67010] -> 1[65020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 04 : 6[6b010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 04 : 1[65020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 10 : 1[65020] -> 0[65010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 10 : 6[6b010] -> 5[69020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 05 : 4[69010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 06 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 11 : 4[69010] -> 3[67020] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 07 : 3[67020] -> 2[67010] via P2P/IPC
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 02 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Connected all trees
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 10 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 02 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 10 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 02 : 2[67010] -> 4[69010] via P2P/indirect/5[69020]
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 02 : 3[67020] -> 5[69020] via P2P/indirect/4[69010]
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 10 : 2[67010] -> 4[69010] via P2P/indirect/5[69020]
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 10 : 3[67020] -> 5[69020] via P2P/indirect/4[69010]
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 03 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 03 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 03 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 03 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 11 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 11 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 11 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 11 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 04 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 04 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 04 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020]
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 04 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 12 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 12 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 04 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 04 : 0[65010] -> 4[69010] via P2P/indirect/3[67020]
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 04 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 12 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020]
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 04 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO Channel 12 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 12 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 12 : 0[65010] -> 4[69010] via P2P/indirect/3[67020]
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO Channel 12 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 12 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 05 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 05 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 05 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 05 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 13 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010]
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO Channel 13 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO Channel 13 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 13 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020]
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 06 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 06 : 4[69010] -> 2[67010] via P2P/indirect/3[67020]
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 06 : 5[69020] -> 3[67020] via P2P/indirect/2[67010]
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 06 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO Channel 14 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010]
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO Channel 14 : 4[69010] -> 2[67010] via P2P/indirect/3[67020]
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO Channel 14 : 5[69020] -> 3[67020] via P2P/indirect/2[67010]
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO Channel 14 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020]
iv-ybpu7pvmiu5m57lh5kdd:9984:12792 [3] NCCL INFO comm 0x7fe4b0008fb0 rank 3 nranks 8 cudaDev 3 busId 67020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9982:12794 [1] NCCL INFO comm 0x7fa8b8008fb0 rank 1 nranks 8 cudaDev 1 busId 65020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9986:12791 [5] NCCL INFO comm 0x7f1874008fb0 rank 5 nranks 8 cudaDev 5 busId 69020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9981:12789 [0] NCCL INFO comm 0x7f3d44008fb0 rank 0 nranks 8 cudaDev 0 busId 65010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9988:12796 [7] NCCL INFO comm 0x7f7c64008fb0 rank 7 nranks 8 cudaDev 7 busId 6b020 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9987:12795 [6] NCCL INFO comm 0x7fbb18008fb0 rank 6 nranks 8 cudaDev 6 busId 6b010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9985:12790 [4] NCCL INFO comm 0x7f0fd0008fb0 rank 4 nranks 8 cudaDev 4 busId 69010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9983:12793 [2] NCCL INFO comm 0x7f4148008fb0 rank 2 nranks 8 cudaDev 2 busId 67010 - Init COMPLETE
iv-ybpu7pvmiu5m57lh5kdd:9981:9981 [0] NCCL INFO Launch mode Parallel
[Rank 2] (after 100 iterations) memory (MB) | allocated: 1569.52734375 | max allocated: 6913.1162109375 | reserved: 7236.0 | max reserved: 7236.0
[Rank 6] (after 100 iterations) memory (MB) | allocated: 1569.52734375 | max allocated: 5313.1142578125 | reserved: 6148.0 | max reserved: 6148.0[Rank 5] (after 100 iterations) memory (MB) | allocated: 1569.52734375 | max allocated: 5713.11474609375 | reserved: 6564.0 | max reserved: 6564.0

[Rank 1] (after 100 iterations) memory (MB) | allocated: 1569.52734375 | max allocated: 7313.11669921875 | reserved: 8164.0 | max reserved: 8164.0
[Rank 4] (after 100 iterations) memory (MB) | allocated: 1569.52734375 | max allocated: 6113.115234375 | reserved: 6436.0 | max reserved: 6436.0
[Rank 3] (after 100 iterations) memory (MB) | allocated: 1569.52734375 | max allocated: 6513.11572265625 | reserved: 7364.0 | max reserved: 7364.0
 iteration      100/     220 | consumed samples:       102400 | elapsed time per iteration (ms): 12131.4 | learning rate: 8.485E-07 | tpt: 84.4 samples/s | global batch size:  1024 | lm loss: 9.508257E+00 | sop loss: 7.037646E-01 | loss scale: 131072.0 | grad norm: 3.527 | number of skipped iterations:  16 | number of nan iterations:   0 |
[Rank 0] (after 100 iterations) memory (MB) | allocated: 1869.06640625 | max allocated: 8153.25830078125 | reserved: 8482.0 | max reserved: 8482.0
[Rank 7] (after 100 iterations) memory (MB) | allocated: 2027.884765625 | max allocated: 6356.6044921875 | reserved: 7374.0 | max reserved: 7374.0
time (ms) | forward-compute: 2790.68 | forward-recv: 927.91 | backward-compute: 6009.12 | backward-send: 3.17 | backward-send-forward-recv: 87.58 | backward-params-all-reduce: 0.69 | backward-embedding-all-reduce: 2282.94 | optimizer-copy-to-main-grad: 2.30 | optimizer-unscale-and-check-inf: 13.81 | optimizer-clip-main-grad: 1.93 | optimizer-copy-main-to-model-params: 1.16 | optimizer: 22.70 | batch-generator: 27.07
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/07/05 12:30:33.822, Tesla V100-SXM2-32GB, 470.57.02, 30 %, 19 %, 32510 MiB, 21676 MiB, 10834 MiB
2022/07/05 12:30:33.824, Tesla V100-SXM2-32GB, 470.57.02, 30 %, 19 %, 32510 MiB, 21676 MiB, 10834 MiB
2022/07/05 12:30:33.824, Tesla V100-SXM2-32GB, 470.57.02, 30 %, 19 %, 32510 MiB, 21676 MiB, 10834 MiB
2022/07/05 12:30:33.825, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 2 %, 32510 MiB, 22078 MiB, 10432 MiB
2022/07/05 12:30:33.829, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 2 %, 32510 MiB, 22078 MiB, 10432 MiB
2022/07/05 12:30:33.829, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 2 %, 32510 MiB, 22078 MiB, 10432 MiB
2022/07/05 12:30:33.829, Tesla V100-SXM2-32GB, 470.57.02, 90 %, 3 %, 32510 MiB, 23258 MiB, 9252 MiB
2022/07/05 12:30:33.832, Tesla V100-SXM2-32GB, 470.57.02, 90 %, 3 %, 32510 MiB, 23258 MiB, 9252 MiB
2022/07/05 12:30:33.832, Tesla V100-SXM2-32GB, 470.57.02, 90 %, 3 %, 32510 MiB, 23258 MiB, 9252 MiB
2022/07/05 12:30:33.833, Tesla V100-SXM2-32GB, 470.57.02, 39 %, 3 %, 32510 MiB, 22998 MiB, 9512 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/07/05 12:30:33.834, Tesla V100-SXM2-32GB, 470.57.02, 39 %, 3 %, 32510 MiB, 22998 MiB, 9512 MiB
2022/07/05 12:30:33.835, Tesla V100-SXM2-32GB, 470.57.02, 39 %, 3 %, 32510 MiB, 22998 MiB, 9512 MiB
2022/07/05 12:30:33.835, Tesla V100-SXM2-32GB, 470.57.02, 95 %, 3 %, 32510 MiB, 23938 MiB, 8572 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/07/05 12:30:33.835, Tesla V100-SXM2-32GB, 470.57.02, 30 %, 19 %, 32510 MiB, 21676 MiB, 10834 MiB
2022/07/05 12:30:33.836, Tesla V100-SXM2-32GB, 470.57.02, 30 %, 19 %, 32510 MiB, 21676 MiB, 10834 MiB
2022/07/05 12:30:33.837, Tesla V100-SXM2-32GB, 470.57.02, 30 %, 19 %, 32510 MiB, 21676 MiB, 10834 MiB
2022/07/05 12:30:33.838, Tesla V100-SXM2-32GB, 470.57.02, 95 %, 3 %, 32510 MiB, 23938 MiB, 8572 MiB
2022/07/05 12:30:33.839, Tesla V100-SXM2-32GB, 470.57.02, 95 %, 3 %, 32510 MiB, 23938 MiB, 8572 MiB
2022/07/05 12:30:33.839, Tesla V100-SXM2-32GB, 470.57.02, 74 %, 3 %, 32510 MiB, 23738 MiB, 8772 MiB
2022/07/05 12:30:33.839, Tesla V100-SXM2-32GB, 470.57.02, 30 %, 19 %, 32510 MiB, 21676 MiB, 10834 MiB
2022/07/05 12:30:33.840, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 2 %, 32510 MiB, 22078 MiB, 10432 MiB
2022/07/05 12:30:33.842, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 2 %, 32510 MiB, 22078 MiB, 10432 MiB
2022/07/05 12:30:33.843, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 2 %, 32510 MiB, 22078 MiB, 10432 MiB
2022/07/05 12:30:33.845, Tesla V100-SXM2-32GB, 470.57.02, 74 %, 3 %, 32510 MiB, 23738 MiB, 8772 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/07/05 12:30:33.845, Tesla V100-SXM2-32GB, 470.57.02, 74 %, 3 %, 32510 MiB, 23738 MiB, 8772 MiB
2022/07/05 12:30:33.846, Tesla V100-SXM2-32GB, 470.57.02, 80 %, 3 %, 32510 MiB, 23962 MiB, 8548 MiB
2022/07/05 12:30:33.846, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 2 %, 32510 MiB, 22078 MiB, 10432 MiB
2022/07/05 12:30:33.847, Tesla V100-SXM2-32GB, 470.57.02, 90 %, 3 %, 32510 MiB, 23258 MiB, 9252 MiB
2022/07/05 12:30:33.848, Tesla V100-SXM2-32GB, 470.57.02, 90 %, 3 %, 32510 MiB, 23258 MiB, 9252 MiB
2022/07/05 12:30:33.849, Tesla V100-SXM2-32GB, 470.57.02, 90 %, 3 %, 32510 MiB, 23258 MiB, 9252 MiB
2022/07/05 12:30:33.850, Tesla V100-SXM2-32GB, 470.57.02, 80 %, 3 %, 32510 MiB, 23962 MiB, 8548 MiB
2022/07/05 12:30:33.851, Tesla V100-SXM2-32GB, 470.57.02, 80 %, 3 %, 32510 MiB, 23962 MiB, 8548 MiB
2022/07/05 12:30:33.850, Tesla V100-SXM2-32GB, 470.57.02, 30 %, 19 %, 32510 MiB, 21676 MiB, 10834 MiB
2022/07/05 12:30:33.851, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 0 %, 32510 MiB, 22904 MiB, 9606 MiB
2022/07/05 12:30:33.852, Tesla V100-SXM2-32GB, 470.57.02, 90 %, 3 %, 32510 MiB, 23258 MiB, 9252 MiB
2022/07/05 12:30:33.852, Tesla V100-SXM2-32GB, 470.57.02, 39 %, 3 %, 32510 MiB, 22998 MiB, 9512 MiB
2022/07/05 12:30:33.854, Tesla V100-SXM2-32GB, 470.57.02, 39 %, 3 %, 32510 MiB, 22998 MiB, 9512 MiB
2022/07/05 12:30:33.855, Tesla V100-SXM2-32GB, 470.57.02, 39 %, 3 %, 32510 MiB, 22998 MiB, 9512 MiB
2022/07/05 12:30:33.856, Tesla V100-SXM2-32GB, 470.57.02, 10 %, 5 %, 32510 MiB, 22904 MiB, 9606 MiB
2022/07/05 12:30:33.857, Tesla V100-SXM2-32GB, 470.57.02, 10 %, 5 %, 32510 MiB, 22904 MiB, 9606 MiB
2022/07/05 12:30:33.857, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 2 %, 32510 MiB, 22078 MiB, 10432 MiB
2022/07/05 12:30:33.858, Tesla V100-SXM2-32GB, 470.57.02, 39 %, 3 %, 32510 MiB, 22998 MiB, 9512 MiB
2022/07/05 12:30:33.859, Tesla V100-SXM2-32GB, 470.57.02, 95 %, 3 %, 32510 MiB, 23938 MiB, 8572 MiB
2022/07/05 12:30:33.861, Tesla V100-SXM2-32GB, 470.57.02, 95 %, 3 %, 32510 MiB, 23938 MiB, 8572 MiB
2022/07/05 12:30:33.862, Tesla V100-SXM2-32GB, 470.57.02, 95 %, 3 %, 32510 MiB, 23938 MiB, 8572 MiB
2022/07/05 12:30:33.873, Tesla V100-SXM2-32GB, 470.57.02, 90 %, 3 %, 32510 MiB, 23258 MiB, 9252 MiB
2022/07/05 12:30:33.875, Tesla V100-SXM2-32GB, 470.57.02, 95 %, 3 %, 32510 MiB, 23938 MiB, 8572 MiB
2022/07/05 12:30:33.875, Tesla V100-SXM2-32GB, 470.57.02, 74 %, 3 %, 32510 MiB, 23738 MiB, 8772 MiB
2022/07/05 12:30:33.877, Tesla V100-SXM2-32GB, 470.57.02, 74 %, 3 %, 32510 MiB, 23738 MiB, 8772 MiB
2022/07/05 12:30:33.877, Tesla V100-SXM2-32GB, 470.57.02, 74 %, 3 %, 32510 MiB, 23738 MiB, 8772 MiB
2022/07/05 12:30:33.879, Tesla V100-SXM2-32GB, 470.57.02, 39 %, 3 %, 32510 MiB, 22998 MiB, 9512 MiB
2022/07/05 12:30:33.880, Tesla V100-SXM2-32GB, 470.57.02, 74 %, 3 %, 32510 MiB, 23738 MiB, 8772 MiB
2022/07/05 12:30:33.880, Tesla V100-SXM2-32GB, 470.57.02, 80 %, 3 %, 32510 MiB, 23962 MiB, 8548 MiB
2022/07/05 12:30:33.882, Tesla V100-SXM2-32GB, 470.57.02, 80 %, 3 %, 32510 MiB, 23962 MiB, 8548 MiB
2022/07/05 12:30:33.883, Tesla V100-SXM2-32GB, 470.57.02, 80 %, 3 %, 32510 MiB, 23962 MiB, 8548 MiB
2022/07/05 12:30:33.885, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 23938 MiB, 8572 MiB
2022/07/05 12:30:33.886, Tesla V100-SXM2-32GB, 470.57.02, 80 %, 3 %, 32510 MiB, 23962 MiB, 8548 MiB
2022/07/05 12:30:33.886, Tesla V100-SXM2-32GB, 470.57.02, 10 %, 5 %, 32510 MiB, 22904 MiB, 9606 MiB
2022/07/05 12:30:33.888, Tesla V100-SXM2-32GB, 470.57.02, 10 %, 5 %, 32510 MiB, 22904 MiB, 9606 MiB
2022/07/05 12:30:33.888, Tesla V100-SXM2-32GB, 470.57.02, 10 %, 5 %, 32510 MiB, 22904 MiB, 9606 MiB
2022/07/05 12:30:33.893, Tesla V100-SXM2-32GB, 470.57.02, 74 %, 3 %, 32510 MiB, 23738 MiB, 8772 MiB
2022/07/05 12:30:33.894, Tesla V100-SXM2-32GB, 470.57.02, 10 %, 5 %, 32510 MiB, 22904 MiB, 9606 MiB
2022/07/05 12:30:33.902, Tesla V100-SXM2-32GB, 470.57.02, 80 %, 3 %, 32510 MiB, 23962 MiB, 8548 MiB
2022/07/05 12:30:33.904, Tesla V100-SXM2-32GB, 470.57.02, 10 %, 5 %, 32510 MiB, 22904 MiB, 9606 MiB
 iteration      200/     220 | consumed samples:       204800 | elapsed time per iteration (ms): 11971.2 | learning rate: 1.859E-06 | tpt: 85.5 samples/s | global batch size:  1024 | lm loss: 8.912864E+00 | sop loss: 6.938200E-01 | loss scale: 131072.0 | grad norm: 5.838 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward-compute: 2779.07 | forward-recv: 818.12 | backward-compute: 6009.10 | backward-send: 3.17 | backward-send-forward-recv: 55.58 | backward-params-all-reduce: 0.69 | backward-embedding-all-reduce: 2285.52 | optimizer-copy-to-main-grad: 2.21 | optimizer-unscale-and-check-inf: 1.47 | optimizer-clip-main-grad: 2.28 | optimizer-copy-main-to-model-params: 1.39 | optimizer: 11.47 | batch-generator: 26.52
[after training is done] datetime: 2022-07-05 12:54:18 
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 validation loss at the end of training for val data | lm loss value: 8.660615E+00 | lm loss PPL: 5.771083E+03 | sop loss value: 6.919112E-01 | sop loss PPL: 1.997530E+00 | 
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 validation loss at the end of training for test data | lm loss value: 8.619910E+00 | lm loss PPL: 5.540889E+03 | sop loss value: 6.938863E-01 | sop loss PPL: 2.001479E+00 | 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
INFO:torch.distributed.elastic.agent.server.api:[default] worker group successfully finished. Waiting 300 seconds for other agents to finish.
INFO:torch.distributed.elastic.agent.server.api:Local worker group finished (SUCCEEDED). Waiting 300 seconds for other agents to finish
/opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/utils/store.py:70: FutureWarning: This is an experimental API and will be changed in future.
  warnings.warn(
INFO:torch.distributed.elastic.agent.server.api:Done waiting for other agents. Elapsed: 0.00030803680419921875 seconds
{"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 0, "group_rank": 0, "worker_id": "9981", "role": "default", "hostname": "iv-ybpu7pvmiu5m57lh5kdd", "state": "SUCCEEDED", "total_run_time": 2777, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python\", \"local_rank\": [0], \"role_rank\": [0], \"role_world_size\": [8]}", "agent_restarts": 0}}
{"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 1, "group_rank": 0, "worker_id": "9982", "role": "default", "hostname": "iv-ybpu7pvmiu5m57lh5kdd", "state": "SUCCEEDED", "total_run_time": 2777, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python\", \"local_rank\": [1], \"role_rank\": [1], \"role_world_size\": [8]}", "agent_restarts": 0}}
{"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 2, "group_rank": 0, "worker_id": "9983", "role": "default", "hostname": "iv-ybpu7pvmiu5m57lh5kdd", "state": "SUCCEEDED", "total_run_time": 2777, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python\", \"local_rank\": [2], \"role_rank\": [2], \"role_world_size\": [8]}", "agent_restarts": 0}}
{"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 3, "group_rank": 0, "worker_id": "9984", "role": "default", "hostname": "iv-ybpu7pvmiu5m57lh5kdd", "state": "SUCCEEDED", "total_run_time": 2777, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python\", \"local_rank\": [3], \"role_rank\": [3], \"role_world_size\": [8]}", "agent_restarts": 0}}
{"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 4, "group_rank": 0, "worker_id": "9985", "role": "default", "hostname": "iv-ybpu7pvmiu5m57lh5kdd", "state": "SUCCEEDED", "total_run_time": 2777, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python\", \"local_rank\": [4], \"role_rank\": [4], \"role_world_size\": [8]}", "agent_restarts": 0}}
{"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 5, "group_rank": 0, "worker_id": "9986", "role": "default", "hostname": "iv-ybpu7pvmiu5m57lh5kdd", "state": "SUCCEEDED", "total_run_time": 2777, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python\", \"local_rank\": [5], \"role_rank\": [5], \"role_world_size\": [8]}", "agent_restarts": 0}}
{"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 6, "group_rank": 0, "worker_id": "9987", "role": "default", "hostname": "iv-ybpu7pvmiu5m57lh5kdd", "state": "SUCCEEDED", "total_run_time": 2777, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python\", \"local_rank\": [6], \"role_rank\": [6], \"role_world_size\": [8]}", "agent_restarts": 0}}
{"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 7, "group_rank": 0, "worker_id": "9988", "role": "default", "hostname": "iv-ybpu7pvmiu5m57lh5kdd", "state": "SUCCEEDED", "total_run_time": 2777, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python\", \"local_rank\": [7], \"role_rank\": [7], \"role_world_size\": [8]}", "agent_restarts": 0}}
{"name": "torchelastic.worker.status.SUCCEEDED", "source": "AGENT", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": null, "group_rank": 0, "worker_id": null, "role": "default", "hostname": "iv-ybpu7pvmiu5m57lh5kdd", "state": "SUCCEEDED", "total_run_time": 2777, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 1, \"entry_point\": \"python\"}", "agent_restarts": 0}}
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************