The module torch.distributed.launch is deprecated and going to be removed in future.Migrate to torch.distributed.run WARNING:torch.distributed.run:--use_env is deprecated and will be removed in future releases. Please read local_rank from `os.environ('LOCAL_RANK')` instead. INFO:torch.distributed.launcher.api:Starting elastic_operator with launch configs: entrypoint : pretrain_bert.py min_nodes : 2 max_nodes : 2 nproc_per_node : 8 run_id : none rdzv_backend : static rdzv_endpoint : 198.18.8.14:6000 rdzv_configs : {'rank': 1, 'timeout': 900} max_restarts : 3 monitor_interval : 5 log_dir : None metrics_cfg : {} INFO:torch.distributed.elastic.agent.server.local_elastic_agent:log directory set to: /tmp/torchelastic_tp3_dzym/none_tyq_ws4i INFO:torch.distributed.elastic.agent.server.api:[default] starting workers for entrypoint: python INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group /opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/utils/store.py:52: FutureWarning: This is an experimental API and will be changed in future. warnings.warn( INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result: restart_count=0 master_addr=198.18.8.14 master_port=6000 group_rank=1 group_world_size=2 local_ranks=[0, 1, 2, 3, 4, 5, 6, 7] role_ranks=[8, 9, 10, 11, 12, 13, 14, 15] global_ranks=[8, 9, 10, 11, 12, 13, 14, 15] role_world_sizes=[16, 16, 16, 16, 16, 16, 16, 16] global_world_sizes=[16, 16, 16, 16, 16, 16, 16, 16] INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_tp3_dzym/none_tyq_ws4i/attempt_0/0/error.json INFO:torch.distributed.elastic.multiprocessing:Setting worker1 reply file to: /tmp/torchelastic_tp3_dzym/none_tyq_ws4i/attempt_0/1/error.json INFO:torch.distributed.elastic.multiprocessing:Setting worker2 reply file to: /tmp/torchelastic_tp3_dzym/none_tyq_ws4i/attempt_0/2/error.json INFO:torch.distributed.elastic.multiprocessing:Setting worker3 reply file to: /tmp/torchelastic_tp3_dzym/none_tyq_ws4i/attempt_0/3/error.json INFO:torch.distributed.elastic.multiprocessing:Setting worker4 reply file to: /tmp/torchelastic_tp3_dzym/none_tyq_ws4i/attempt_0/4/error.json INFO:torch.distributed.elastic.multiprocessing:Setting worker5 reply file to: /tmp/torchelastic_tp3_dzym/none_tyq_ws4i/attempt_0/5/error.json INFO:torch.distributed.elastic.multiprocessing:Setting worker6 reply file to: /tmp/torchelastic_tp3_dzym/none_tyq_ws4i/attempt_0/6/error.json INFO:torch.distributed.elastic.multiprocessing:Setting worker7 reply file to: /tmp/torchelastic_tp3_dzym/none_tyq_ws4i/attempt_0/7/error.json [W ProcessGroupNCCL.cpp:1671] Rank 9 using best-guess GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. [W ProcessGroupNCCL.cpp:1671] Rank 11 using best-guess GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. [W ProcessGroupNCCL.cpp:1671] Rank 10 using best-guess GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. [W ProcessGroupNCCL.cpp:1671] Rank 14 using best-guess GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. [W ProcessGroupNCCL.cpp:1671] Rank 13 using best-guess GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. [W ProcessGroupNCCL.cpp:1671] Rank 15 using best-guess GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. [W ProcessGroupNCCL.cpp:1671] Rank 8 using best-guess GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. [W ProcessGroupNCCL.cpp:1671] Rank 12 using best-guess GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. iv-ybpu7pvmis5m57pm6ny1:57843:57843 [5] NCCL INFO Bootstrap : Using eth0:192.168.11.142<0> iv-ybpu7pvmis5m57pm6ny1:57839:57839 [1] NCCL INFO Bootstrap : Using eth0:192.168.11.142<0> iv-ybpu7pvmis5m57pm6ny1:57838:57838 [0] NCCL INFO Bootstrap : Using eth0:192.168.11.142<0> iv-ybpu7pvmis5m57pm6ny1:57845:57845 [7] NCCL INFO Bootstrap : Using eth0:192.168.11.142<0> iv-ybpu7pvmis5m57pm6ny1:57843:57843 [5] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-ybpu7pvmis5m57pm6ny1:57843:57843 [5] NCCL INFO P2P plugin IBext iv-ybpu7pvmis5m57pm6ny1:57843:57843 [5] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-ybpu7pvmis5m57pm6ny1:57839:57839 [1] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-ybpu7pvmis5m57pm6ny1:57839:57839 [1] NCCL INFO P2P plugin IBext iv-ybpu7pvmis5m57pm6ny1:57839:57839 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-ybpu7pvmis5m57pm6ny1:57838:57838 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-ybpu7pvmis5m57pm6ny1:57838:57838 [0] NCCL INFO P2P plugin IBext iv-ybpu7pvmis5m57pm6ny1:57838:57838 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-ybpu7pvmis5m57pm6ny1:57842:57842 [4] NCCL INFO Bootstrap : Using eth0:192.168.11.142<0> iv-ybpu7pvmis5m57pm6ny1:57844:57844 [6] NCCL INFO Bootstrap : Using eth0:192.168.11.142<0> iv-ybpu7pvmis5m57pm6ny1:57841:57841 [3] NCCL INFO Bootstrap : Using eth0:192.168.11.142<0> iv-ybpu7pvmis5m57pm6ny1:57845:57845 [7] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-ybpu7pvmis5m57pm6ny1:57845:57845 [7] NCCL INFO P2P plugin IBext iv-ybpu7pvmis5m57pm6ny1:57845:57845 [7] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-ybpu7pvmis5m57pm6ny1:57840:57840 [2] NCCL INFO Bootstrap : Using eth0:192.168.11.142<0> iv-ybpu7pvmis5m57pm6ny1:57844:57844 [6] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-ybpu7pvmis5m57pm6ny1:57842:57842 [4] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-ybpu7pvmis5m57pm6ny1:57842:57842 [4] NCCL INFO P2P plugin IBext iv-ybpu7pvmis5m57pm6ny1:57844:57844 [6] NCCL INFO P2P plugin IBext iv-ybpu7pvmis5m57pm6ny1:57842:57842 [4] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-ybpu7pvmis5m57pm6ny1:57844:57844 [6] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-ybpu7pvmis5m57pm6ny1:57841:57841 [3] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-ybpu7pvmis5m57pm6ny1:57841:57841 [3] NCCL INFO P2P plugin IBext iv-ybpu7pvmis5m57pm6ny1:57841:57841 [3] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-ybpu7pvmis5m57pm6ny1:57840:57840 [2] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-ybpu7pvmis5m57pm6ny1:57840:57840 [2] NCCL INFO P2P plugin IBext iv-ybpu7pvmis5m57pm6ny1:57840:57840 [2] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-ybpu7pvmis5m57pm6ny1:57843:57843 [5] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.142<0> iv-ybpu7pvmis5m57pm6ny1:57843:57843 [5] NCCL INFO Using network IBext iv-ybpu7pvmis5m57pm6ny1:57842:57842 [4] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.142<0> iv-ybpu7pvmis5m57pm6ny1:57842:57842 [4] NCCL INFO Using network IBext iv-ybpu7pvmis5m57pm6ny1:57838:57838 [0] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.142<0> iv-ybpu7pvmis5m57pm6ny1:57838:57838 [0] NCCL INFO Using network IBext iv-ybpu7pvmis5m57pm6ny1:57844:57844 [6] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.142<0> iv-ybpu7pvmis5m57pm6ny1:57844:57844 [6] NCCL INFO Using network IBext iv-ybpu7pvmis5m57pm6ny1:57839:57839 [1] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.142<0> iv-ybpu7pvmis5m57pm6ny1:57839:57839 [1] NCCL INFO Using network IBext iv-ybpu7pvmis5m57pm6ny1:57845:57845 [7] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.142<0> iv-ybpu7pvmis5m57pm6ny1:57845:57845 [7] NCCL INFO Using network IBext iv-ybpu7pvmis5m57pm6ny1:57840:57840 [2] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.142<0> iv-ybpu7pvmis5m57pm6ny1:57840:57840 [2] NCCL INFO Using network IBext iv-ybpu7pvmis5m57pm6ny1:57841:57841 [3] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.142<0> iv-ybpu7pvmis5m57pm6ny1:57841:57841 [3] NCCL INFO Using network IBext iv-ybpu7pvmis5m57pm6ny1:57838:58029 [0] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-ybpu7pvmis5m57pm6ny1:57841:58034 [3] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-ybpu7pvmis5m57pm6ny1:57845:58032 [7] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-ybpu7pvmis5m57pm6ny1:57839:58030 [1] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-ybpu7pvmis5m57pm6ny1:57843:58026 [5] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-ybpu7pvmis5m57pm6ny1:57840:58033 [2] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-ybpu7pvmis5m57pm6ny1:57844:58031 [6] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-ybpu7pvmis5m57pm6ny1:57838:58029 [0] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-ybpu7pvmis5m57pm6ny1:57845:58032 [7] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-ybpu7pvmis5m57pm6ny1:57838:58029 [0] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-ybpu7pvmis5m57pm6ny1:57845:58032 [7] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-ybpu7pvmis5m57pm6ny1:57841:58034 [3] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-ybpu7pvmis5m57pm6ny1:57841:58034 [3] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-ybpu7pvmis5m57pm6ny1:57839:58030 [1] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-ybpu7pvmis5m57pm6ny1:57839:58030 [1] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-ybpu7pvmis5m57pm6ny1:57840:58033 [2] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-ybpu7pvmis5m57pm6ny1:57840:58033 [2] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-ybpu7pvmis5m57pm6ny1:57843:58026 [5] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-ybpu7pvmis5m57pm6ny1:57843:58026 [5] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-ybpu7pvmis5m57pm6ny1:57844:58031 [6] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-ybpu7pvmis5m57pm6ny1:57844:58031 [6] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-ybpu7pvmis5m57pm6ny1:57839:58030 [1] NCCL INFO Trees [0] 14/-1/-1->9->11 [1] 14/-1/-1->9->11 iv-ybpu7pvmis5m57pm6ny1:57838:58029 [0] NCCL INFO Trees [0] 10/-1/-1->8->15 [1] 10/-1/-1->8->15 iv-ybpu7pvmis5m57pm6ny1:57839:58030 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57840:58033 [2] NCCL INFO Trees [0] 11/-1/-1->10->8 [1] 11/-1/-1->10->8 iv-ybpu7pvmis5m57pm6ny1:57838:58029 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57840:58033 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57841:58034 [3] NCCL INFO Trees [0] 9/-1/-1->11->10 [1] 9/-1/-1->11->10 iv-ybpu7pvmis5m57pm6ny1:57841:58034 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57843:58026 [5] NCCL INFO Trees [0] 15/-1/-1->13->12 [1] 15/-1/-1->13->12 iv-ybpu7pvmis5m57pm6ny1:57845:58032 [7] NCCL INFO Trees [0] 8/-1/-1->15->13 [1] 8/-1/-1->15->13 iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO Trees [0] 13/-1/-1->12->4 [1] 13/4/-1->12->-1 iv-ybpu7pvmis5m57pm6ny1:57843:58026 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57844:58031 [6] NCCL INFO Trees [0] -1/-1/-1->14->9 [1] -1/-1/-1->14->9 iv-ybpu7pvmis5m57pm6ny1:57845:58032 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57844:58031 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57838:58029 [0] NCCL INFO Channel 00 : 8[65010] -> 9[65020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58033 [2] NCCL INFO Channel 00 : 10[67010] -> 13[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO Channel 00 : 12[69010] -> 14[6b010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57844:58031 [6] NCCL INFO Channel 00 : 14[6b010] -> 15[6b020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57838:58029 [0] NCCL INFO Channel 01 : 8[65010] -> 9[65020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58033 [2] NCCL INFO Channel 01 : 10[67010] -> 13[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO Channel 01 : 12[69010] -> 14[6b010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57844:58031 [6] NCCL INFO Channel 01 : 14[6b010] -> 15[6b020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57839:58030 [1] NCCL INFO Channel 00 : 9[65020] -> 11[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58026 [5] NCCL INFO Channel 00 : 13[69020] -> 4[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57845:58032 [7] NCCL INFO Channel 00 : 15[6b020] -> 8[65010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57839:58030 [1] NCCL INFO Channel 01 : 9[65020] -> 11[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58026 [5] NCCL INFO Channel 01 : 13[69020] -> 4[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57845:58032 [7] NCCL INFO Channel 01 : 15[6b020] -> 8[65010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57839:58030 [1] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57845:58032 [7] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57838:58029 [0] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57844:58031 [6] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57839:58030 [1] NCCL INFO Channel 00 : 9[65020] -> 14[6b010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58034 [3] NCCL INFO Channel 00 : 11[67020] -> 10[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO Channel 00 : 5[69020] -> 12[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57838:58029 [0] NCCL INFO Channel 00 : 8[65010] -> 10[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57839:58030 [1] NCCL INFO Channel 01 : 9[65020] -> 14[6b010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58034 [3] NCCL INFO Channel 01 : 11[67020] -> 10[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO Channel 01 : 5[69020] -> 12[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57838:58029 [0] NCCL INFO Channel 01 : 8[65010] -> 10[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58033 [2] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57841:58034 [3] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57840:58033 [2] NCCL INFO Channel 00 : 10[67010] -> 11[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58026 [5] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57844:58031 [6] NCCL INFO Channel 00 : 14[6b010] -> 9[65020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58033 [2] NCCL INFO Channel 01 : 10[67010] -> 11[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57844:58031 [6] NCCL INFO Channel 01 : 14[6b010] -> 9[65020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO Channel 00 : 12[69010] -> 13[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57844:58031 [6] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57844:58031 [6] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57844:58031 [6] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO Channel 01 : 12[69010] -> 13[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58034 [3] NCCL INFO Channel 00 : 11[67020] -> 9[65020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58034 [3] NCCL INFO Channel 01 : 11[67020] -> 9[65020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO Channel 00 : 4[69010] -> 12[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57843:58026 [5] NCCL INFO Channel 00 : 13[69020] -> 15[6b020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO Channel 01 : 4[69010] -> 12[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57840:58033 [2] NCCL INFO Channel 00 : 10[67010] -> 8[65010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58026 [5] NCCL INFO Channel 01 : 13[69020] -> 15[6b020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57838:58029 [0] NCCL INFO Channel 00 : 8[65010] -> 15[6b020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO Channel 00 : 12[69010] -> 4[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57839:58030 [1] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57839:58030 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57839:58030 [1] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57841:58034 [3] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57841:58034 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57841:58034 [3] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57840:58033 [2] NCCL INFO Channel 01 : 10[67010] -> 8[65010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57838:58029 [0] NCCL INFO Channel 01 : 8[65010] -> 15[6b020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57839:58030 [1] NCCL INFO Channel 01 : 9[65020] -> 12[69010] via P2P/indirect/14[6b010] iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO Channel 01 : 12[69010] -> 4[69010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57841:58034 [3] NCCL INFO Channel 00 : 11[67020] -> 13[69020] via P2P/indirect/12[69010] iv-ybpu7pvmis5m57pm6ny1:57845:58032 [7] NCCL INFO Channel 00 : 15[6b020] -> 13[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57845:58032 [7] NCCL INFO Channel 01 : 15[6b020] -> 13[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57845:58032 [7] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57845:58032 [7] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57845:58032 [7] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57838:58029 [0] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57838:58029 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57838:58029 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57840:58033 [2] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57840:58033 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57840:58033 [2] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57843:58026 [5] NCCL INFO Channel 00 : 13[69020] -> 12[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58033 [2] NCCL INFO Channel 00 : 10[67010] -> 12[69010] via P2P/indirect/13[69020] iv-ybpu7pvmis5m57pm6ny1:57838:58029 [0] NCCL INFO Channel 00 : 8[65010] -> 12[69010] via P2P/indirect/11[67020] iv-ybpu7pvmis5m57pm6ny1:57843:58026 [5] NCCL INFO Channel 01 : 13[69020] -> 12[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58026 [5] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57843:58026 [5] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57843:58026 [5] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57841:58034 [3] NCCL INFO Channel 01 : 11[67020] -> 14[6b010] via P2P/indirect/9[65020] iv-ybpu7pvmis5m57pm6ny1:57840:58033 [2] NCCL INFO Channel 00 : 10[67010] -> 14[6b010] via P2P/indirect/9[65020] iv-ybpu7pvmis5m57pm6ny1:57841:58034 [3] NCCL INFO Channel 00 : 11[67020] -> 15[6b020] via P2P/indirect/8[65010] iv-ybpu7pvmis5m57pm6ny1:57839:58030 [1] NCCL INFO Channel 00 : 9[65020] -> 13[69020] via P2P/indirect/14[6b010] iv-ybpu7pvmis5m57pm6ny1:57840:58033 [2] NCCL INFO Channel 01 : 10[67010] -> 15[6b020] via P2P/indirect/8[65010] iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO Channel 00 : 12[69010] -> 8[65010] via P2P/indirect/15[6b020] iv-ybpu7pvmis5m57pm6ny1:57839:58030 [1] NCCL INFO Channel 00 : 9[65020] -> 15[6b020] via P2P/indirect/8[65010] iv-ybpu7pvmis5m57pm6ny1:57838:58029 [0] NCCL INFO Channel 01 : 8[65010] -> 13[69020] via P2P/indirect/15[6b020] iv-ybpu7pvmis5m57pm6ny1:57838:58029 [0] NCCL INFO Channel 00 : 8[65010] -> 14[6b010] via P2P/indirect/9[65020] iv-ybpu7pvmis5m57pm6ny1:57843:58026 [5] NCCL INFO Channel 01 : 13[69020] -> 8[65010] via P2P/indirect/15[6b020] iv-ybpu7pvmis5m57pm6ny1:57845:58032 [7] NCCL INFO Channel 00 : 15[6b020] -> 9[65020] via P2P/indirect/14[6b010] iv-ybpu7pvmis5m57pm6ny1:57844:58031 [6] NCCL INFO Channel 00 : 14[6b010] -> 8[65010] via P2P/indirect/15[6b020] iv-ybpu7pvmis5m57pm6ny1:57845:58032 [7] NCCL INFO Channel 01 : 15[6b020] -> 10[67010] via P2P/indirect/8[65010] iv-ybpu7pvmis5m57pm6ny1:57844:58031 [6] NCCL INFO Channel 00 : 14[6b010] -> 10[67010] via P2P/indirect/13[69020] iv-ybpu7pvmis5m57pm6ny1:57845:58032 [7] NCCL INFO Channel 00 : 15[6b020] -> 11[67020] via P2P/indirect/8[65010] iv-ybpu7pvmis5m57pm6ny1:57844:58031 [6] NCCL INFO Channel 01 : 14[6b010] -> 11[67020] via P2P/indirect/9[65020] iv-ybpu7pvmis5m57pm6ny1:57843:58026 [5] NCCL INFO Channel 00 : 13[69020] -> 9[65020] via P2P/indirect/14[6b010] iv-ybpu7pvmis5m57pm6ny1:57843:58026 [5] NCCL INFO Channel 00 : 13[69020] -> 11[67020] via P2P/indirect/10[67010] iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO Channel 01 : 12[69010] -> 9[65020] via P2P/indirect/14[6b010] iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO Channel 00 : 12[69010] -> 10[67010] via P2P/indirect/11[67020] iv-ybpu7pvmis5m57pm6ny1:57843:58026 [5] NCCL INFO comm 0x7fb250008fb0 rank 13 nranks 16 cudaDev 5 busId 69020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57838:58029 [0] NCCL INFO comm 0x7fa280008fb0 rank 8 nranks 16 cudaDev 0 busId 65010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57840:58033 [2] NCCL INFO comm 0x7f4028008fb0 rank 10 nranks 16 cudaDev 2 busId 67010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57842:58028 [4] NCCL INFO comm 0x7f27e0008fb0 rank 12 nranks 16 cudaDev 4 busId 69010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57845:58032 [7] NCCL INFO comm 0x7f2508008fb0 rank 15 nranks 16 cudaDev 7 busId 6b020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57844:58031 [6] NCCL INFO comm 0x7f6c48008fb0 rank 14 nranks 16 cudaDev 6 busId 6b010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57839:58030 [1] NCCL INFO comm 0x7f9640008fb0 rank 9 nranks 16 cudaDev 1 busId 65020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57841:58034 [3] NCCL INFO comm 0x7f90c4008fb0 rank 11 nranks 16 cudaDev 3 busId 67020 - Init COMPLETE > number of parameters on (tensor, pipeline) model parallel rank (0, 2): 75577344 iv-ybpu7pvmis5m57pm6ny1:57843:58096 [5] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-ybpu7pvmis5m57pm6ny1:57843:58096 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57843:58096 [5] NCCL INFO Channel 00 : 0[65020] -> 1[69020] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57842:58098 [4] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-ybpu7pvmis5m57pm6ny1:57842:58098 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57843:58096 [5] NCCL INFO Channel 01 : 0[65020] -> 1[69020] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57844:58101 [6] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-ybpu7pvmis5m57pm6ny1:57844:58101 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57843:58096 [5] NCCL INFO Channel 00 : 1[69020] -> 0[65020] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57842:58098 [4] NCCL INFO Channel 00 : 0[65010] -> 1[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57842:58098 [4] NCCL INFO Channel 01 : 0[65010] -> 1[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57845:58100 [7] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-ybpu7pvmis5m57pm6ny1:57845:58100 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57842:58098 [4] NCCL INFO Channel 00 : 1[69010] -> 0[65010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57843:58096 [5] NCCL INFO Channel 01 : 1[69020] -> 0[65020] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57842:58098 [4] NCCL INFO Channel 01 : 1[69010] -> 0[65010] [send] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57844:58101 [6] NCCL INFO Channel 00 : 0[67010] -> 1[6b010] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57844:58101 [6] NCCL INFO Channel 01 : 0[67010] -> 1[6b010] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57845:58100 [7] NCCL INFO Channel 00 : 0[67020] -> 1[6b020] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57844:58101 [6] NCCL INFO Channel 00 : 1[6b010] -> 0[67010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57845:58100 [7] NCCL INFO Channel 01 : 0[67020] -> 1[6b020] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57844:58101 [6] NCCL INFO Channel 01 : 1[6b010] -> 0[67010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57845:58100 [7] NCCL INFO Channel 00 : 1[6b020] -> 0[67020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57844:58101 [6] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57844:58101 [6] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57844:58101 [6] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57844:58101 [6] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57844:58101 [6] NCCL INFO comm 0x7f6c2c008fb0 rank 1 nranks 2 cudaDev 6 busId 6b010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57845:58100 [7] NCCL INFO Channel 01 : 1[6b020] -> 0[67020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57845:58100 [7] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57845:58100 [7] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57845:58100 [7] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57845:58100 [7] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57845:58100 [7] NCCL INFO comm 0x7f24e8008fb0 rank 1 nranks 2 cudaDev 7 busId 6b020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57843:58096 [5] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57843:58096 [5] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57843:58096 [5] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57843:58096 [5] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57843:58096 [5] NCCL INFO comm 0x7fb22c008fb0 rank 1 nranks 2 cudaDev 5 busId 69020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57842:58098 [4] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57842:58098 [4] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57842:58098 [4] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57842:58098 [4] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57842:58098 [4] NCCL INFO comm 0x7f27ac008fb0 rank 1 nranks 2 cudaDev 4 busId 69010 - Init COMPLETE > number of parameters on (tensor, pipeline) model parallel rank (0, 3): 99461890 NCCL version 2.10.3+cuda11.4 NCCL version 2.10.3+cuda11.4 iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Trees [0] 3/-1/-1->1->0 [1] 0/-1/-1->1->3 [2] 3/-1/-1->1->0 [3] 0/-1/-1->1->3 [4] 3/-1/-1->1->0 [5] 0/-1/-1->1->3 [6] 3/-1/-1->1->0 [7] 0/-1/-1->1->3 iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 00/08 : 0 1 3 2 iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 01/08 : 0 3 1 2 iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 02/08 : 0 2 3 1 iv-ybpu7pvmis5m57pm6ny1:57845:58114 [7] NCCL INFO Trees [0] -1/-1/-1->3->1 [1] 1/-1/-1->3->-1 [2] -1/-1/-1->3->1 [3] 1/-1/-1->3->-1 [4] -1/-1/-1->3->1 [5] 1/-1/-1->3->-1 [6] -1/-1/-1->3->1 [7] 1/-1/-1->3->-1 iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 03/08 : 0 2 1 3 iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 04/08 : 0 1 3 2 iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 05/08 : 0 3 1 2 iv-ybpu7pvmis5m57pm6ny1:57845:58114 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 06/08 : 0 2 3 1 iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 07/08 : 0 2 1 3 iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Trees [0] 1/-1/-1->0->2 [1] 2/-1/-1->0->1 [2] 1/-1/-1->0->2 [3] 2/-1/-1->0->1 [4] 1/-1/-1->0->2 [5] 2/-1/-1->0->1 [6] 1/-1/-1->0->2 [7] 2/-1/-1->0->1 iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57844:58111 [6] NCCL INFO Trees [0] 0/-1/-1->2->-1 [1] -1/-1/-1->2->0 [2] 0/-1/-1->2->-1 [3] -1/-1/-1->2->0 [4] 0/-1/-1->2->-1 [5] -1/-1/-1->2->0 [6] 0/-1/-1->2->-1 [7] -1/-1/-1->2->0 iv-ybpu7pvmis5m57pm6ny1:57844:58111 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Channel 01 : 1[69020] -> 2[6b010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57845:58114 [7] NCCL INFO Channel 03 : 3[6b020] -> 0[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Channel 05 : 1[69020] -> 2[6b010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 00 : 0[69010] -> 1[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57845:58114 [7] NCCL INFO Channel 07 : 3[6b020] -> 0[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 04 : 0[69010] -> 1[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57844:58111 [6] NCCL INFO Channel 02 : 2[6b010] -> 3[6b020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57844:58111 [6] NCCL INFO Channel 06 : 2[6b010] -> 3[6b020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 02 : 0[69010] -> 2[6b010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 03 : 0[69010] -> 2[6b010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Channel 00 : 1[69020] -> 3[6b020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57845:58114 [7] NCCL INFO Channel 01 : 3[6b020] -> 1[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 06 : 0[69010] -> 2[6b010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57844:58111 [6] NCCL INFO Channel 00 : 2[6b010] -> 0[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Channel 03 : 1[69020] -> 3[6b020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 07 : 0[69010] -> 2[6b010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57845:58114 [7] NCCL INFO Channel 02 : 3[6b020] -> 1[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57844:58111 [6] NCCL INFO Channel 01 : 2[6b010] -> 0[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Channel 04 : 1[69020] -> 3[6b020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57845:58114 [7] NCCL INFO Channel 05 : 3[6b020] -> 1[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57844:58111 [6] NCCL INFO Channel 04 : 2[6b010] -> 0[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Channel 07 : 1[69020] -> 3[6b020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57845:58114 [7] NCCL INFO Channel 06 : 3[6b020] -> 1[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57844:58111 [6] NCCL INFO Channel 05 : 2[6b010] -> 0[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Channel 02 : 1[69020] -> 0[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57845:58114 [7] NCCL INFO Channel 00 : 3[6b020] -> 2[6b010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 01 : 0[69010] -> 3[6b020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57844:58111 [6] NCCL INFO Channel 03 : 2[6b010] -> 1[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Channel 06 : 1[69020] -> 0[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57845:58114 [7] NCCL INFO Channel 04 : 3[6b020] -> 2[6b010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57844:58111 [6] NCCL INFO Channel 07 : 2[6b010] -> 1[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 05 : 0[69010] -> 3[6b020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57844:58111 [6] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57845:58114 [7] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 01 : 0[69010] -> 1[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 02 : 0[69010] -> 1[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 03 : 0[69010] -> 1[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 05 : 0[69010] -> 1[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57844:58111 [6] NCCL INFO Channel 02 : 2[6b010] -> 0[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57845:58114 [7] NCCL INFO Channel 00 : 3[6b020] -> 1[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 06 : 0[69010] -> 1[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57844:58111 [6] NCCL INFO Channel 03 : 2[6b010] -> 0[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 07 : 0[69010] -> 1[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57845:58114 [7] NCCL INFO Channel 03 : 3[6b020] -> 1[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57844:58111 [6] NCCL INFO Channel 06 : 2[6b010] -> 0[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57845:58114 [7] NCCL INFO Channel 04 : 3[6b020] -> 1[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57844:58111 [6] NCCL INFO Channel 07 : 2[6b010] -> 0[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57845:58114 [7] NCCL INFO Channel 07 : 3[6b020] -> 1[69020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Channel 01 : 1[69020] -> 3[6b020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 00 : 0[69010] -> 2[6b010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Channel 02 : 1[69020] -> 3[6b020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 01 : 0[69010] -> 2[6b010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Channel 05 : 1[69020] -> 3[6b020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 04 : 0[69010] -> 2[6b010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Channel 06 : 1[69020] -> 3[6b020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Channel 05 : 0[69010] -> 2[6b010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 00/08 : 0 2 3 1 iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 01/08 : 0 2 1 3 iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 02/08 : 0 1 3 2 iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 03/08 : 0 3 1 2 iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 04/08 : 0 2 3 1 iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 05/08 : 0 2 1 3 iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 06/08 : 0 1 3 2 iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 07/08 : 0 3 1 2 iv-ybpu7pvmis5m57pm6ny1:57839:58115 [1] NCCL INFO Trees [0] -1/-1/-1->1->3 [1] 3/-1/-1->1->-1 [2] -1/-1/-1->1->3 [3] 3/-1/-1->1->-1 [4] -1/-1/-1->1->3 [5] 3/-1/-1->1->-1 [6] -1/-1/-1->1->3 [7] 3/-1/-1->1->-1 iv-ybpu7pvmis5m57pm6ny1:57839:58115 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Trees [0] 2/-1/-1->0->-1 [1] -1/-1/-1->0->2 [2] 2/-1/-1->0->-1 [3] -1/-1/-1->0->2 [4] 2/-1/-1->0->-1 [5] -1/-1/-1->0->2 [6] 2/-1/-1->0->-1 [7] -1/-1/-1->0->2 iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 0/-1/-1->2->3 [2] 3/-1/-1->2->0 [3] 0/-1/-1->2->3 [4] 3/-1/-1->2->0 [5] 0/-1/-1->2->3 [6] 3/-1/-1->2->0 [7] 0/-1/-1->2->3 iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Trees [0] 1/-1/-1->3->2 [1] 2/-1/-1->3->1 [2] 1/-1/-1->3->2 [3] 2/-1/-1->3->1 [4] 1/-1/-1->3->2 [5] 2/-1/-1->3->1 [6] 1/-1/-1->3->2 [7] 2/-1/-1->3->1 iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57845:58114 [7] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57845:58114 [7] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57845:58114 [7] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57844:58111 [6] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57844:58111 [6] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57844:58111 [6] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Channel 00 : 1[69020] -> 0[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Channel 01 : 1[69020] -> 0[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Channel 03 : 1[69020] -> 0[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Channel 04 : 1[69020] -> 0[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Channel 05 : 1[69020] -> 0[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Channel 07 : 1[69020] -> 0[69010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57842:58109 [4] NCCL INFO comm 0x7f2770008fb0 rank 0 nranks 4 cudaDev 4 busId 69010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57843:58110 [5] NCCL INFO comm 0x7fb22c0be010 rank 1 nranks 4 cudaDev 5 busId 69020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57845:58114 [7] NCCL INFO comm 0x7f24e80d3010 rank 3 nranks 4 cudaDev 7 busId 6b020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57844:58111 [6] NCCL INFO comm 0x7f6c2c0d3010 rank 2 nranks 4 cudaDev 6 busId 6b010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57842:57842 [4] NCCL INFO Launch mode Parallel iv-ybpu7pvmis5m57pm6ny1:57839:58115 [1] NCCL INFO Channel 03 : 1[65020] -> 2[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Channel 04 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Channel 01 : 3[67020] -> 0[65010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 02 : 0[65010] -> 1[65020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57839:58115 [1] NCCL INFO Channel 07 : 1[65020] -> 2[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Channel 05 : 3[67020] -> 0[65010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 06 : 0[65010] -> 1[65020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Channel 02 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57839:58115 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Channel 03 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Channel 03 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57839:58115 [1] NCCL INFO Channel 02 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Channel 06 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Channel 04 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 04 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57839:58115 [1] NCCL INFO Channel 05 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Channel 07 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Channel 07 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 05 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57839:58115 [1] NCCL INFO Channel 06 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 03 : 0[65010] -> 3[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Channel 02 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Channel 01 : 2[67010] -> 1[65020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57839:58115 [1] NCCL INFO Channel 00 : 1[65020] -> 0[65010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 07 : 0[65010] -> 3[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Channel 05 : 2[67010] -> 1[65020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Channel 06 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57839:58115 [1] NCCL INFO Channel 04 : 1[65020] -> 0[65010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57839:58115 [1] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Channel 02 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Channel 03 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Channel 05 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 02 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57839:58115 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Channel 06 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 03 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57839:58115 [1] NCCL INFO Channel 03 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Channel 07 : 2[67010] -> 3[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 06 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57839:58115 [1] NCCL INFO Channel 04 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Channel 07 : 0[65010] -> 2[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57839:58115 [1] NCCL INFO Channel 07 : 1[65020] -> 3[67020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Channel 02 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Channel 05 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Channel 04 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Channel 06 : 3[67020] -> 1[65020] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Channel 05 : 2[67010] -> 0[65010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57839:58115 [1] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57839:58115 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57839:58115 [1] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Channel 03 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Channel 04 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Channel 05 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Channel 07 : 3[67020] -> 2[67010] via P2P/IPC iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57841:58117 [3] NCCL INFO comm 0x7f9078008fb0 rank 3 nranks 4 cudaDev 3 busId 67020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57840:58116 [2] NCCL INFO comm 0x7f3ff0008fb0 rank 2 nranks 4 cudaDev 2 busId 67010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57838:58113 [0] NCCL INFO comm 0x7fa228008fb0 rank 0 nranks 4 cudaDev 0 busId 65010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57839:58115 [1] NCCL INFO comm 0x7f95ec008fb0 rank 1 nranks 4 cudaDev 1 busId 65020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57838:57838 [0] NCCL INFO Launch mode Parallel iv-ybpu7pvmis5m57pm6ny1:57841:58141 [3] NCCL INFO Trees [0] -1/-1/-1->2->3 [1] -1/-1/-1->2->3 iv-ybpu7pvmis5m57pm6ny1:57845:58131 [7] NCCL INFO Trees [0] 2/-1/-1->3->1 [1] 2/1/-1->3->-1 iv-ybpu7pvmis5m57pm6ny1:57845:58131 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57841:58141 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57844:58132 [6] NCCL INFO Trees [0] 2/-1/-1->3->1 [1] 2/1/-1->3->-1 iv-ybpu7pvmis5m57pm6ny1:57840:58138 [2] NCCL INFO Trees [0] -1/-1/-1->2->3 [1] -1/-1/-1->2->3 iv-ybpu7pvmis5m57pm6ny1:57844:58132 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57840:58138 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57843:58133 [5] NCCL INFO Trees [0] 2/-1/-1->3->1 [1] 2/1/-1->3->-1 iv-ybpu7pvmis5m57pm6ny1:57843:58133 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57839:58140 [1] NCCL INFO Trees [0] -1/-1/-1->2->3 [1] -1/-1/-1->2->3 iv-ybpu7pvmis5m57pm6ny1:57839:58140 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57838:58139 [0] NCCL INFO Trees [0] -1/-1/-1->2->3 [1] -1/-1/-1->2->3 iv-ybpu7pvmis5m57pm6ny1:57842:58130 [4] NCCL INFO Trees [0] 2/-1/-1->3->1 [1] 2/1/-1->3->-1 iv-ybpu7pvmis5m57pm6ny1:57838:58139 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57842:58130 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57841:58141 [3] NCCL INFO Channel 00 : 1[6b020] -> 2[67020] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57840:58138 [2] NCCL INFO Channel 00 : 1[6b010] -> 2[67010] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57841:58141 [3] NCCL INFO Channel 01 : 1[6b020] -> 2[67020] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57839:58140 [1] NCCL INFO Channel 00 : 1[69020] -> 2[65020] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57841:58141 [3] NCCL INFO Channel 00 : 2[67020] -> 3[6b020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57840:58138 [2] NCCL INFO Channel 01 : 1[6b010] -> 2[67010] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57841:58141 [3] NCCL INFO Channel 01 : 2[67020] -> 3[6b020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57838:58139 [0] NCCL INFO Channel 00 : 1[69010] -> 2[65010] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57840:58138 [2] NCCL INFO Channel 00 : 2[67010] -> 3[6b010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57840:58138 [2] NCCL INFO Channel 01 : 2[67010] -> 3[6b010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57839:58140 [1] NCCL INFO Channel 01 : 1[69020] -> 2[65020] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57838:58139 [0] NCCL INFO Channel 01 : 1[69010] -> 2[65010] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57845:58131 [7] NCCL INFO Channel 00 : 3[6b020] -> 0[67020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57838:58139 [0] NCCL INFO Channel 00 : 2[65010] -> 3[69010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57839:58140 [1] NCCL INFO Channel 00 : 2[65020] -> 3[69020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57838:58139 [0] NCCL INFO Channel 01 : 2[65010] -> 3[69010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57839:58140 [1] NCCL INFO Channel 01 : 2[65020] -> 3[69020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57843:58133 [5] NCCL INFO Channel 00 : 3[69020] -> 0[65020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57844:58132 [6] NCCL INFO Channel 00 : 3[6b010] -> 0[67010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57845:58131 [7] NCCL INFO Channel 01 : 3[6b020] -> 0[67020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57842:58130 [4] NCCL INFO Channel 00 : 3[69010] -> 0[65010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57843:58133 [5] NCCL INFO Channel 01 : 3[69020] -> 0[65020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57844:58132 [6] NCCL INFO Channel 01 : 3[6b010] -> 0[67010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57842:58130 [4] NCCL INFO Channel 01 : 3[69010] -> 0[65010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57845:58131 [7] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57843:58133 [5] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57843:58133 [5] NCCL INFO Channel 00 : 1[69020] -> 3[69020] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57843:58133 [5] NCCL INFO Channel 01 : 1[69020] -> 3[69020] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57844:58132 [6] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57845:58131 [7] NCCL INFO Channel 00 : 1[6b020] -> 3[6b020] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57841:58141 [3] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57843:58133 [5] NCCL INFO Channel 00 : 3[69020] -> 1[69020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57842:58130 [4] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57844:58132 [6] NCCL INFO Channel 00 : 1[6b010] -> 3[6b010] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57845:58131 [7] NCCL INFO Channel 01 : 1[6b020] -> 3[6b020] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57842:58130 [4] NCCL INFO Channel 00 : 1[69010] -> 3[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57842:58130 [4] NCCL INFO Channel 01 : 1[69010] -> 3[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57839:58140 [1] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57840:58138 [2] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57843:58133 [5] NCCL INFO Channel 01 : 3[69020] -> 1[69020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57838:58139 [0] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57844:58132 [6] NCCL INFO Channel 01 : 1[6b010] -> 3[6b010] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57845:58131 [7] NCCL INFO Channel 00 : 3[6b020] -> 1[6b020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57842:58130 [4] NCCL INFO Channel 00 : 3[69010] -> 1[69010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57844:58132 [6] NCCL INFO Channel 00 : 3[6b010] -> 1[6b010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57845:58131 [7] NCCL INFO Channel 01 : 3[6b020] -> 1[6b020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57842:58130 [4] NCCL INFO Channel 01 : 3[69010] -> 1[69010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57844:58132 [6] NCCL INFO Channel 01 : 3[6b010] -> 1[6b010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57845:58131 [7] NCCL INFO Channel 00 : 3[6b020] -> 2[67020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57844:58132 [6] NCCL INFO Channel 00 : 3[6b010] -> 2[67010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57845:58131 [7] NCCL INFO Channel 01 : 3[6b020] -> 2[67020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57844:58132 [6] NCCL INFO Channel 01 : 3[6b010] -> 2[67010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57843:58133 [5] NCCL INFO Channel 00 : 3[69020] -> 2[65020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57842:58130 [4] NCCL INFO Channel 00 : 3[69010] -> 2[65010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57841:58141 [3] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57841:58141 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57841:58141 [3] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57843:58133 [5] NCCL INFO Channel 01 : 3[69020] -> 2[65020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57845:58131 [7] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57845:58131 [7] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57845:58131 [7] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57841:58141 [3] NCCL INFO comm 0x7f90781713b0 rank 2 nranks 4 cudaDev 3 busId 67020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57845:58131 [7] NCCL INFO comm 0x7f24e831c340 rank 3 nranks 4 cudaDev 7 busId 6b020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57840:58138 [2] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57840:58138 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57840:58138 [2] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57842:58130 [4] NCCL INFO Channel 01 : 3[69010] -> 2[65010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57844:58132 [6] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57844:58132 [6] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57844:58132 [6] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57840:58138 [2] NCCL INFO comm 0x7f3ff0171800 rank 2 nranks 4 cudaDev 2 busId 67010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57844:58132 [6] NCCL INFO comm 0x7f6c2c31bd50 rank 3 nranks 4 cudaDev 6 busId 6b010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57839:58140 [1] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57839:58140 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57839:58140 [1] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57838:58139 [0] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57838:58139 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57838:58139 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57843:58133 [5] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57843:58133 [5] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57843:58133 [5] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57843:58133 [5] NCCL INFO comm 0x7fb22c319e20 rank 3 nranks 4 cudaDev 5 busId 69020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57842:58130 [4] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57842:58130 [4] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57842:58130 [4] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57838:58139 [0] NCCL INFO comm 0x7fa228160d30 rank 2 nranks 4 cudaDev 0 busId 65010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57842:58130 [4] NCCL INFO comm 0x7f277012bc80 rank 3 nranks 4 cudaDev 4 busId 69010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57839:58140 [1] NCCL INFO comm 0x7f95ec161540 rank 2 nranks 4 cudaDev 1 busId 65020 - Init COMPLETE NCCL version 2.10.3+cuda11.4 NCCL version 2.10.3+cuda11.4 NCCL version 2.10.3+cuda11.4 NCCL version 2.10.3+cuda11.4 NCCL version 2.10.3+cuda11.4 NCCL version 2.10.3+cuda11.4 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 00/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 01/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 02/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 03/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 04/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 05/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 06/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 07/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 08/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 09/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 10/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 11/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 12/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 13/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 14/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 15/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 16/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 17/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 18/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 19/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 20/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 21/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 22/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 23/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 24/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 25/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 26/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 27/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 28/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 29/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 30/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Channel 31/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 00/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 01/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 02/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 03/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 04/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 05/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 06/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 07/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 08/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 09/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 10/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 11/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 12/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 13/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 14/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 15/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 16/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 17/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 18/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 19/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 20/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 21/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 22/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 23/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 24/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 25/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 26/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 27/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 28/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 29/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 30/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Channel 31/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1 iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 00/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 01/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 02/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 03/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 04/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 05/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 06/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 07/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 08/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 09/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 10/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 11/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 12/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 13/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 14/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 15/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 16/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 17/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 18/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 19/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 20/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 21/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 22/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 23/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 24/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 25/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 26/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 27/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 28/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 29/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 30/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Channel 31/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1 iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 00/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 01/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 02/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 03/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 04/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 05/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 06/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 07/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 08/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 09/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 10/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 11/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 12/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 13/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 14/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 15/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 16/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 17/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 18/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 19/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 20/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 21/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 22/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 23/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 24/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 25/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 26/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 27/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 28/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 29/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 30/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Channel 31/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1 iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 00/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 01/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 02/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 03/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 04/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 05/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 06/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 07/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 08/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 09/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 10/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 11/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 12/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 13/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 14/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 15/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 16/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 17/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 18/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 19/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 20/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 21/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 22/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 23/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 24/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 25/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 26/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 27/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 28/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 29/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 30/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Channel 31/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1 iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 00/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 01/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 02/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 03/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 04/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 05/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 06/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 07/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 08/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 09/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 10/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 11/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 12/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 13/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 14/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 15/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 16/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 17/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 18/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 19/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 20/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 21/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 22/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 23/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 24/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 25/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 26/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 27/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 28/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 29/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 30/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Channel 31/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1 iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 00/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 01/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 02/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 03/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 04/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 05/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 06/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 07/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 08/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 09/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 10/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 11/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 12/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 13/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 14/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 15/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 16/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 17/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 18/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 19/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 20/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 21/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 22/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 23/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 24/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 25/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 26/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 27/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 28/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 29/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 30/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Channel 31/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1 iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 00/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 01/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 02/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 03/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 04/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 05/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 06/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 07/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 08/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 09/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 10/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 11/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 12/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 13/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 14/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 15/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 16/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 17/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 18/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 19/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 20/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 21/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 22/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 23/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 24/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 25/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 26/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 27/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 28/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 29/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 30/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Channel 31/32 : 0 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1 iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57843:58164 [5] NCCL INFO comm 0x7fb1c0008fb0 rank 0 nranks 1 cudaDev 5 busId 69020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57839:58161 [1] NCCL INFO comm 0x7f95c4008fb0 rank 0 nranks 1 cudaDev 1 busId 65020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57842:58159 [4] NCCL INFO comm 0x7f2740008fb0 rank 0 nranks 1 cudaDev 4 busId 69010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57840:58165 [2] NCCL INFO comm 0x7f3fc0008fb0 rank 0 nranks 1 cudaDev 2 busId 67010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57838:58171 [0] NCCL INFO comm 0x7fa20c008fb0 rank 0 nranks 1 cudaDev 0 busId 65010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57841:58174 [3] NCCL INFO comm 0x7f904c008fb0 rank 0 nranks 1 cudaDev 3 busId 67020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57844:58172 [6] NCCL INFO comm 0x7f6bc8008fb0 rank 0 nranks 1 cudaDev 6 busId 6b010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57845:58177 [7] NCCL INFO comm 0x7f2484008fb0 rank 0 nranks 1 cudaDev 7 busId 6b020 - Init COMPLETE [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) time (ms) | model-and-optimizer-setup: 454.14 | train/valid/test-data-iterators-setup: 1939.72 iv-ybpu7pvmis5m57pm6ny1:57840:60424 [2] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-ybpu7pvmis5m57pm6ny1:57840:60424 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57840:60424 [2] NCCL INFO Channel 00 : 0[6b010] -> 1[67010] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57840:60424 [2] NCCL INFO Channel 01 : 0[6b010] -> 1[67010] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57840:60424 [2] NCCL INFO Channel 00 : 1[67010] -> 0[6b010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57840:60424 [2] NCCL INFO Channel 01 : 1[67010] -> 0[6b010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57840:60424 [2] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57840:60424 [2] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57840:60424 [2] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57840:60424 [2] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57840:60424 [2] NCCL INFO comm 0x7f3f80008fb0 rank 1 nranks 2 cudaDev 2 busId 67010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57841:60427 [3] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-ybpu7pvmis5m57pm6ny1:57841:60427 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57838:60429 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-ybpu7pvmis5m57pm6ny1:57838:60429 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57839:60431 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-ybpu7pvmis5m57pm6ny1:57839:60431 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57841:60427 [3] NCCL INFO Channel 00 : 0[6b020] -> 1[67020] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57838:60429 [0] NCCL INFO Channel 00 : 0[69010] -> 1[65010] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57841:60427 [3] NCCL INFO Channel 01 : 0[6b020] -> 1[67020] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57838:60429 [0] NCCL INFO Channel 01 : 0[69010] -> 1[65010] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57839:60431 [1] NCCL INFO Channel 00 : 0[69020] -> 1[65020] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57841:60427 [3] NCCL INFO Channel 00 : 1[67020] -> 0[6b020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57838:60429 [0] NCCL INFO Channel 00 : 1[65010] -> 0[69010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57839:60431 [1] NCCL INFO Channel 01 : 0[69020] -> 1[65020] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57841:60427 [3] NCCL INFO Channel 01 : 1[67020] -> 0[6b020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57838:60429 [0] NCCL INFO Channel 01 : 1[65010] -> 0[69010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57839:60431 [1] NCCL INFO Channel 00 : 1[65020] -> 0[69020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57839:60431 [1] NCCL INFO Channel 01 : 1[65020] -> 0[69020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57841:60427 [3] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57841:60427 [3] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57841:60427 [3] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57841:60427 [3] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57841:60427 [3] NCCL INFO comm 0x7f900c008fb0 rank 1 nranks 2 cudaDev 3 busId 67020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57838:60429 [0] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57838:60429 [0] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57838:60429 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57838:60429 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57838:60429 [0] NCCL INFO comm 0x7fa1d4008fb0 rank 1 nranks 2 cudaDev 0 busId 65010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57839:60431 [1] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57839:60431 [1] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57839:60431 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57839:60431 [1] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57839:60431 [1] NCCL INFO comm 0x7f957c008fb0 rank 1 nranks 2 cudaDev 1 busId 65020 - Init COMPLETE /dataset/workspace/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.) output = bias_dropout_add_func( /dataset/workspace/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.) output = bias_dropout_add_func( iv-ybpu7pvmis5m57pm6ny1:57844:60438 [6] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 iv-ybpu7pvmis5m57pm6ny1:57844:60438 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57840:60437 [2] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmis5m57pm6ny1:57840:60437 [2] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmis5m57pm6ny1:57840:60437 [2] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 iv-ybpu7pvmis5m57pm6ny1:57840:60437 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57844:60438 [6] NCCL INFO Channel 00 : 1[6b010] -> 0[67010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57844:60438 [6] NCCL INFO Channel 01 : 1[6b010] -> 0[67010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57840:60437 [2] NCCL INFO Channel 00 : 0[67010] -> 1[6b010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57840:60437 [2] NCCL INFO Channel 01 : 0[67010] -> 1[6b010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57844:60438 [6] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57844:60438 [6] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57844:60438 [6] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57844:60438 [6] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57840:60437 [2] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57840:60437 [2] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57840:60437 [2] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57840:60437 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57844:60438 [6] NCCL INFO comm 0x7f6b8c008fb0 rank 1 nranks 2 cudaDev 6 busId 6b010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57840:60437 [2] NCCL INFO comm 0x7f3e10008fb0 rank 0 nranks 2 cudaDev 2 busId 67010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57840:57840 [2] NCCL INFO Launch mode Parallel /dataset/workspace/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.) output = bias_dropout_add_func( /dataset/workspace/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.) output = bias_dropout_add_func( iv-ybpu7pvmis5m57pm6ny1:57841:60444 [3] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmis5m57pm6ny1:57845:60445 [7] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 iv-ybpu7pvmis5m57pm6ny1:57841:60444 [3] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmis5m57pm6ny1:57845:60445 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57841:60444 [3] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 iv-ybpu7pvmis5m57pm6ny1:57841:60444 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57838:60449 [0] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmis5m57pm6ny1:57842:60450 [4] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 iv-ybpu7pvmis5m57pm6ny1:57838:60449 [0] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmis5m57pm6ny1:57842:60450 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57838:60449 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 iv-ybpu7pvmis5m57pm6ny1:57838:60449 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57841:60444 [3] NCCL INFO Channel 00 : 0[67020] -> 1[6b020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57841:60444 [3] NCCL INFO Channel 01 : 0[67020] -> 1[6b020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57845:60445 [7] NCCL INFO Channel 00 : 1[6b020] -> 0[67020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57845:60445 [7] NCCL INFO Channel 01 : 1[6b020] -> 0[67020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57839:60454 [1] NCCL INFO Channel 00/02 : 0 1 iv-ybpu7pvmis5m57pm6ny1:57843:60455 [5] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 iv-ybpu7pvmis5m57pm6ny1:57839:60454 [1] NCCL INFO Channel 01/02 : 0 1 iv-ybpu7pvmis5m57pm6ny1:57843:60455 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57839:60454 [1] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 iv-ybpu7pvmis5m57pm6ny1:57839:60454 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57845:60445 [7] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57845:60445 [7] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57845:60445 [7] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57845:60445 [7] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57841:60444 [3] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57841:60444 [3] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57841:60444 [3] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57841:60444 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57841:60444 [3] NCCL INFO comm 0x7f8e9c008fb0 rank 0 nranks 2 cudaDev 3 busId 67020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57845:60445 [7] NCCL INFO comm 0x7f2440008fb0 rank 1 nranks 2 cudaDev 7 busId 6b020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57841:57841 [3] NCCL INFO Launch mode Parallel iv-ybpu7pvmis5m57pm6ny1:57838:60449 [0] NCCL INFO Channel 00 : 0[65010] -> 1[69010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57838:60449 [0] NCCL INFO Channel 01 : 0[65010] -> 1[69010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57842:60450 [4] NCCL INFO Channel 00 : 1[69010] -> 0[65010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57842:60450 [4] NCCL INFO Channel 01 : 1[69010] -> 0[65010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57842:60450 [4] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57842:60450 [4] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57842:60450 [4] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57842:60450 [4] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57843:60455 [5] NCCL INFO Channel 00 : 1[69020] -> 0[65020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57838:60449 [0] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57838:60449 [0] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57838:60449 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57838:60449 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57842:60450 [4] NCCL INFO comm 0x7f26fc008fb0 rank 1 nranks 2 cudaDev 4 busId 69010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57838:60449 [0] NCCL INFO comm 0x7fa060008fb0 rank 0 nranks 2 cudaDev 0 busId 65010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57843:60455 [5] NCCL INFO Channel 01 : 1[69020] -> 0[65020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57838:57838 [0] NCCL INFO Launch mode Parallel iv-ybpu7pvmis5m57pm6ny1:57839:60454 [1] NCCL INFO Channel 00 : 0[65020] -> 1[69020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57839:60454 [1] NCCL INFO Channel 01 : 0[65020] -> 1[69020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57843:60455 [5] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57843:60455 [5] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57843:60455 [5] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57843:60455 [5] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57839:60454 [1] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57839:60454 [1] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57839:60454 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57839:60454 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57843:60455 [5] NCCL INFO comm 0x7fb180008fb0 rank 1 nranks 2 cudaDev 5 busId 69020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57839:60454 [1] NCCL INFO comm 0x7f9414008fb0 rank 0 nranks 2 cudaDev 1 busId 65020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57839:57839 [1] NCCL INFO Launch mode Parallel /dataset/workspace/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.) output = bias_dropout_add_func( /dataset/workspace/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.) output = bias_dropout_add_func( /dataset/workspace/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.) output = bias_dropout_add_func( /dataset/workspace/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.) output = bias_dropout_add_func( iv-ybpu7pvmis5m57pm6ny1:57839:60568 [1] NCCL INFO Trees [0] -1/-1/-1->2->3 [1] -1/-1/-1->2->3 iv-ybpu7pvmis5m57pm6ny1:57843:60571 [5] NCCL INFO Trees [0] 2/-1/-1->3->1 [1] 2/1/-1->3->-1 iv-ybpu7pvmis5m57pm6ny1:57839:60568 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57843:60571 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57844:60572 [6] NCCL INFO Trees [0] 2/-1/-1->3->1 [1] 2/1/-1->3->-1 iv-ybpu7pvmis5m57pm6ny1:57840:60567 [2] NCCL INFO Trees [0] -1/-1/-1->2->3 [1] -1/-1/-1->2->3 iv-ybpu7pvmis5m57pm6ny1:57840:60567 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57844:60572 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57845:60573 [7] NCCL INFO Trees [0] 2/-1/-1->3->1 [1] 2/1/-1->3->-1 iv-ybpu7pvmis5m57pm6ny1:57841:60570 [3] NCCL INFO Trees [0] -1/-1/-1->2->3 [1] -1/-1/-1->2->3 iv-ybpu7pvmis5m57pm6ny1:57845:60573 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57841:60570 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57838:60569 [0] NCCL INFO Trees [0] -1/-1/-1->2->3 [1] -1/-1/-1->2->3 iv-ybpu7pvmis5m57pm6ny1:57842:60574 [4] NCCL INFO Trees [0] 2/-1/-1->3->1 [1] 2/1/-1->3->-1 iv-ybpu7pvmis5m57pm6ny1:57838:60569 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-ybpu7pvmis5m57pm6ny1:57842:60574 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-ybpu7pvmis5m57pm6ny1:57839:60568 [1] NCCL INFO Channel 00 : 1[69020] -> 2[65020] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57839:60568 [1] NCCL INFO Channel 01 : 1[69020] -> 2[65020] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57840:60567 [2] NCCL INFO Channel 00 : 1[6b010] -> 2[67010] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57839:60568 [1] NCCL INFO Channel 00 : 2[65020] -> 3[69020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57839:60568 [1] NCCL INFO Channel 01 : 2[65020] -> 3[69020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57838:60569 [0] NCCL INFO Channel 00 : 1[69010] -> 2[65010] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57841:60570 [3] NCCL INFO Channel 00 : 1[6b020] -> 2[67020] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57843:60571 [5] NCCL INFO Channel 00 : 3[69020] -> 0[65020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57838:60569 [0] NCCL INFO Channel 01 : 1[69010] -> 2[65010] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57840:60567 [2] NCCL INFO Channel 01 : 1[6b010] -> 2[67010] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57838:60569 [0] NCCL INFO Channel 00 : 2[65010] -> 3[69010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57838:60569 [0] NCCL INFO Channel 01 : 2[65010] -> 3[69010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57840:60567 [2] NCCL INFO Channel 00 : 2[67010] -> 3[6b010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57841:60570 [3] NCCL INFO Channel 01 : 1[6b020] -> 2[67020] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57840:60567 [2] NCCL INFO Channel 01 : 2[67010] -> 3[6b010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57841:60570 [3] NCCL INFO Channel 00 : 2[67020] -> 3[6b020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57841:60570 [3] NCCL INFO Channel 01 : 2[67020] -> 3[6b020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57843:60571 [5] NCCL INFO Channel 01 : 3[69020] -> 0[65020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57844:60572 [6] NCCL INFO Channel 00 : 3[6b010] -> 0[67010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57842:60574 [4] NCCL INFO Channel 00 : 3[69010] -> 0[65010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57845:60573 [7] NCCL INFO Channel 00 : 3[6b020] -> 0[67020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57844:60572 [6] NCCL INFO Channel 01 : 3[6b010] -> 0[67010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57843:60571 [5] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57843:60571 [5] NCCL INFO Channel 00 : 1[69020] -> 3[69020] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57843:60571 [5] NCCL INFO Channel 01 : 1[69020] -> 3[69020] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57842:60574 [4] NCCL INFO Channel 01 : 3[69010] -> 0[65010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57845:60573 [7] NCCL INFO Channel 01 : 3[6b020] -> 0[67020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57844:60572 [6] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57843:60571 [5] NCCL INFO Channel 00 : 3[69020] -> 1[69020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57839:60568 [1] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57844:60572 [6] NCCL INFO Channel 00 : 1[6b010] -> 3[6b010] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57843:60571 [5] NCCL INFO Channel 01 : 3[69020] -> 1[69020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57842:60574 [4] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57842:60574 [4] NCCL INFO Channel 00 : 1[69010] -> 3[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57845:60573 [7] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57842:60574 [4] NCCL INFO Channel 01 : 1[69010] -> 3[69010] [receive] via NET/IBext/0/GDRDMA iv-ybpu7pvmis5m57pm6ny1:57844:60572 [6] NCCL INFO Channel 01 : 1[6b010] -> 3[6b010] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57840:60567 [2] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57845:60573 [7] NCCL INFO Channel 00 : 1[6b020] -> 3[6b020] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57842:60574 [4] NCCL INFO Channel 00 : 3[69010] -> 1[69010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57838:60569 [0] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57844:60572 [6] NCCL INFO Channel 00 : 3[6b010] -> 1[6b010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57841:60570 [3] NCCL INFO Connected all rings iv-ybpu7pvmis5m57pm6ny1:57845:60573 [7] NCCL INFO Channel 01 : 1[6b020] -> 3[6b020] [receive] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57842:60574 [4] NCCL INFO Channel 01 : 3[69010] -> 1[69010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57844:60572 [6] NCCL INFO Channel 01 : 3[6b010] -> 1[6b010] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57845:60573 [7] NCCL INFO Channel 00 : 3[6b020] -> 1[6b020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57845:60573 [7] NCCL INFO Channel 01 : 3[6b020] -> 1[6b020] [send] via NET/IBext/0 iv-ybpu7pvmis5m57pm6ny1:57844:60572 [6] NCCL INFO Channel 00 : 3[6b010] -> 2[67010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57845:60573 [7] NCCL INFO Channel 00 : 3[6b020] -> 2[67020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57844:60572 [6] NCCL INFO Channel 01 : 3[6b010] -> 2[67010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57843:60571 [5] NCCL INFO Channel 00 : 3[69020] -> 2[65020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57845:60573 [7] NCCL INFO Channel 01 : 3[6b020] -> 2[67020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57842:60574 [4] NCCL INFO Channel 00 : 3[69010] -> 2[65010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57843:60571 [5] NCCL INFO Channel 01 : 3[69020] -> 2[65020] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57840:60567 [2] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57840:60567 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57840:60567 [2] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57842:60574 [4] NCCL INFO Channel 01 : 3[69010] -> 2[65010] via direct shared memory iv-ybpu7pvmis5m57pm6ny1:57844:60572 [6] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57844:60572 [6] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57844:60572 [6] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57844:60572 [6] NCCL INFO comm 0x7f680c008fb0 rank 3 nranks 4 cudaDev 6 busId 6b010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57841:60570 [3] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57840:60567 [2] NCCL INFO comm 0x7f3c98008fb0 rank 2 nranks 4 cudaDev 2 busId 67010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57841:60570 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57841:60570 [3] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57845:60573 [7] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57845:60573 [7] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57845:60573 [7] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57845:60573 [7] NCCL INFO comm 0x7f20c0008fb0 rank 3 nranks 4 cudaDev 7 busId 6b020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57841:60570 [3] NCCL INFO comm 0x7f8e5c008fb0 rank 2 nranks 4 cudaDev 3 busId 67020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57839:60568 [1] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57839:60568 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57839:60568 [1] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57838:60569 [0] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57838:60569 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57838:60569 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57843:60571 [5] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57843:60571 [5] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57843:60571 [5] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57843:60571 [5] NCCL INFO comm 0x7fae00008fb0 rank 3 nranks 4 cudaDev 5 busId 69020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57839:60568 [1] NCCL INFO comm 0x7f92a0008fb0 rank 2 nranks 4 cudaDev 1 busId 65020 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57842:60574 [4] NCCL INFO Connected all trees iv-ybpu7pvmis5m57pm6ny1:57842:60574 [4] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-ybpu7pvmis5m57pm6ny1:57842:60574 [4] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-ybpu7pvmis5m57pm6ny1:57838:60569 [0] NCCL INFO comm 0x7f9eec008fb0 rank 2 nranks 4 cudaDev 0 busId 65010 - Init COMPLETE iv-ybpu7pvmis5m57pm6ny1:57842:60574 [4] NCCL INFO comm 0x7f2378008fb0 rank 3 nranks 4 cudaDev 4 busId 69010 - Init COMPLETE [Rank 8] (after 100 iterations) memory (MB) | allocated: 1697.52734375 | max allocated: 10049.6142578125 | reserved: 11542.0 | max reserved: 11542.0 iteration 100/ 210 | consumed samples: 409600 | elapsed time per iteration (ms): 12043.3 | tpt: 340.1 samples/s | global batch size: 4096 | lm loss: 9.577758E+00 | sop loss: 7.028186E-01 | loss scale: 262144.0 | grad norm: 3.133 | number of skipped iterations: 15 | number of nan iterations: 0 | [Rank 12] (after 100 iterations) memory (MB) | allocated: 2156.884765625 | max allocated: 11193.2265625 | reserved: 14432.0 | max reserved: 14432.0 time (ms) | forward-compute: 2861.30 | forward-recv: 834.23 | backward-compute: 6045.15 | backward-send: 18.39 | backward-send-forward-recv: 188.69 | backward-params-all-reduce: 15.37 | backward-embedding-all-reduce: 2057.72 | optimizer-copy-to-main-grad: 2.72 | optimizer-unscale-and-check-inf: 8.48 | optimizer-clip-main-grad: 2.06 | optimizer-copy-main-to-model-params: 1.20 | optimizer: 18.09 | batch-generator: 19.40 timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/06/15 12:36:29.566, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 0 %, 32510 MiB, 19584 MiB, 12926 MiB 2022/06/15 12:36:29.567, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 0 %, 32510 MiB, 19584 MiB, 12926 MiB timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/06/15 12:36:29.568, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 0 %, 32510 MiB, 19584 MiB, 12926 MiB 2022/06/15 12:36:29.570, Tesla V100-SXM2-32GB, 470.57.02, 47 %, 3 %, 32510 MiB, 19578 MiB, 12932 MiB 2022/06/15 12:36:29.570, Tesla V100-SXM2-32GB, 470.57.02, 47 %, 3 %, 32510 MiB, 19578 MiB, 12932 MiB timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/06/15 12:36:29.571, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 0 %, 32510 MiB, 19584 MiB, 12926 MiB 2022/06/15 12:36:29.571, Tesla V100-SXM2-32GB, 470.57.02, 47 %, 3 %, 32510 MiB, 19578 MiB, 12932 MiB timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/06/15 12:36:29.573, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 0 %, 32510 MiB, 19560 MiB, 12950 MiB timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/06/15 12:36:29.573, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 0 %, 32510 MiB, 19560 MiB, 12950 MiB 2022/06/15 12:36:29.574, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 0 %, 32510 MiB, 19584 MiB, 12926 MiB 2022/06/15 12:36:29.574, Tesla V100-SXM2-32GB, 470.57.02, 47 %, 3 %, 32510 MiB, 19578 MiB, 12932 MiB 2022/06/15 12:36:29.574, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 0 %, 32510 MiB, 19560 MiB, 12950 MiB 2022/06/15 12:36:29.574, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 0 %, 32510 MiB, 19584 MiB, 12926 MiB 2022/06/15 12:36:29.575, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 0 %, 32510 MiB, 19584 MiB, 12926 MiB 2022/06/15 12:36:29.577, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 3 %, 32510 MiB, 19552 MiB, 12958 MiB 2022/06/15 12:36:29.578, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 3 %, 32510 MiB, 19552 MiB, 12958 MiB 2022/06/15 12:36:29.578, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 0 %, 32510 MiB, 19584 MiB, 12926 MiB 2022/06/15 12:36:29.579, Tesla V100-SXM2-32GB, 470.57.02, 47 %, 3 %, 32510 MiB, 19578 MiB, 12932 MiB 2022/06/15 12:36:29.579, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 0 %, 32510 MiB, 19560 MiB, 12950 MiB 2022/06/15 12:36:29.580, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 3 %, 32510 MiB, 19552 MiB, 12958 MiB 2022/06/15 12:36:29.580, Tesla V100-SXM2-32GB, 470.57.02, 47 %, 3 %, 32510 MiB, 19578 MiB, 12932 MiB 2022/06/15 12:36:29.581, Tesla V100-SXM2-32GB, 470.57.02, 47 %, 3 %, 32510 MiB, 19578 MiB, 12932 MiB 2022/06/15 12:36:29.583, Tesla V100-SXM2-32GB, 470.57.02, 71 %, 6 %, 32510 MiB, 16550 MiB, 15960 MiB 2022/06/15 12:36:29.584, Tesla V100-SXM2-32GB, 470.57.02, 71 %, 6 %, 32510 MiB, 16550 MiB, 15960 MiB 2022/06/15 12:36:29.585, Tesla V100-SXM2-32GB, 470.57.02, 47 %, 3 %, 32510 MiB, 19578 MiB, 12932 MiB 2022/06/15 12:36:29.586, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 0 %, 32510 MiB, 19560 MiB, 12950 MiB 2022/06/15 12:36:29.586, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 3 %, 32510 MiB, 19552 MiB, 12958 MiB 2022/06/15 12:36:29.586, Tesla V100-SXM2-32GB, 470.57.02, 71 %, 6 %, 32510 MiB, 16550 MiB, 15960 MiB 2022/06/15 12:36:29.587, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 0 %, 32510 MiB, 19560 MiB, 12950 MiB 2022/06/15 12:36:29.588, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 0 %, 32510 MiB, 19560 MiB, 12950 MiB 2022/06/15 12:36:29.589, Tesla V100-SXM2-32GB, 470.57.02, 99 %, 1 %, 32510 MiB, 16538 MiB, 15972 MiB 2022/06/15 12:36:29.591, Tesla V100-SXM2-32GB, 470.57.02, 99 %, 1 %, 32510 MiB, 16538 MiB, 15972 MiB 2022/06/15 12:36:29.591, Tesla V100-SXM2-32GB, 470.57.02, 100 %, 0 %, 32510 MiB, 19560 MiB, 12950 MiB 2022/06/15 12:36:29.593, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 3 %, 32510 MiB, 19552 MiB, 12958 MiB 2022/06/15 12:36:29.593, Tesla V100-SXM2-32GB, 470.57.02, 71 %, 6 %, 32510 MiB, 16550 MiB, 15960 MiB 2022/06/15 12:36:29.593, Tesla V100-SXM2-32GB, 470.57.02, 99 %, 1 %, 32510 MiB, 16538 MiB, 15972 MiB 2022/06/15 12:36:29.594, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 3 %, 32510 MiB, 19552 MiB, 12958 MiB 2022/06/15 12:36:29.594, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 3 %, 32510 MiB, 19552 MiB, 12958 MiB 2022/06/15 12:36:29.596, Tesla V100-SXM2-32GB, 470.57.02, 94 %, 5 %, 32510 MiB, 16696 MiB, 15814 MiB 2022/06/15 12:36:29.598, Tesla V100-SXM2-32GB, 470.57.02, 94 %, 5 %, 32510 MiB, 16696 MiB, 15814 MiB 2022/06/15 12:36:29.598, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 3 %, 32510 MiB, 19552 MiB, 12958 MiB 2022/06/15 12:36:29.599, Tesla V100-SXM2-32GB, 470.57.02, 71 %, 6 %, 32510 MiB, 16550 MiB, 15960 MiB 2022/06/15 12:36:29.599, Tesla V100-SXM2-32GB, 470.57.02, 99 %, 1 %, 32510 MiB, 16538 MiB, 15972 MiB 2022/06/15 12:36:29.600, Tesla V100-SXM2-32GB, 470.57.02, 94 %, 5 %, 32510 MiB, 16696 MiB, 15814 MiB 2022/06/15 12:36:29.600, Tesla V100-SXM2-32GB, 470.57.02, 71 %, 6 %, 32510 MiB, 16550 MiB, 15960 MiB 2022/06/15 12:36:29.601, Tesla V100-SXM2-32GB, 470.57.02, 71 %, 6 %, 32510 MiB, 16550 MiB, 15960 MiB 2022/06/15 12:36:29.603, Tesla V100-SXM2-32GB, 470.57.02, 20 %, 5 %, 32510 MiB, 16688 MiB, 15822 MiB 2022/06/15 12:36:29.604, Tesla V100-SXM2-32GB, 470.57.02, 20 %, 5 %, 32510 MiB, 16688 MiB, 15822 MiB 2022/06/15 12:36:29.605, Tesla V100-SXM2-32GB, 470.57.02, 71 %, 6 %, 32510 MiB, 16550 MiB, 15960 MiB 2022/06/15 12:36:29.606, Tesla V100-SXM2-32GB, 470.57.02, 99 %, 1 %, 32510 MiB, 16538 MiB, 15972 MiB 2022/06/15 12:36:29.606, Tesla V100-SXM2-32GB, 470.57.02, 94 %, 5 %, 32510 MiB, 16696 MiB, 15814 MiB 2022/06/15 12:36:29.606, Tesla V100-SXM2-32GB, 470.57.02, 20 %, 5 %, 32510 MiB, 16688 MiB, 15822 MiB 2022/06/15 12:36:29.607, Tesla V100-SXM2-32GB, 470.57.02, 99 %, 1 %, 32510 MiB, 16538 MiB, 15972 MiB 2022/06/15 12:36:29.608, Tesla V100-SXM2-32GB, 470.57.02, 99 %, 1 %, 32510 MiB, 16538 MiB, 15972 MiB 2022/06/15 12:36:29.611, Tesla V100-SXM2-32GB, 470.57.02, 99 %, 1 %, 32510 MiB, 16538 MiB, 15972 MiB 2022/06/15 12:36:29.612, Tesla V100-SXM2-32GB, 470.57.02, 94 %, 5 %, 32510 MiB, 16696 MiB, 15814 MiB 2022/06/15 12:36:29.613, Tesla V100-SXM2-32GB, 470.57.02, 20 %, 5 %, 32510 MiB, 16688 MiB, 15822 MiB 2022/06/15 12:36:29.614, Tesla V100-SXM2-32GB, 470.57.02, 94 %, 5 %, 32510 MiB, 16696 MiB, 15814 MiB 2022/06/15 12:36:29.614, Tesla V100-SXM2-32GB, 470.57.02, 94 %, 5 %, 32510 MiB, 16696 MiB, 15814 MiB 2022/06/15 12:36:29.617, Tesla V100-SXM2-32GB, 470.57.02, 94 %, 5 %, 32510 MiB, 16696 MiB, 15814 MiB 2022/06/15 12:36:29.618, Tesla V100-SXM2-32GB, 470.57.02, 20 %, 5 %, 32510 MiB, 16688 MiB, 15822 MiB 2022/06/15 12:36:29.619, Tesla V100-SXM2-32GB, 470.57.02, 20 %, 5 %, 32510 MiB, 16688 MiB, 15822 MiB 2022/06/15 12:36:29.620, Tesla V100-SXM2-32GB, 470.57.02, 20 %, 5 %, 32510 MiB, 16688 MiB, 15822 MiB 2022/06/15 12:36:29.623, Tesla V100-SXM2-32GB, 470.57.02, 20 %, 5 %, 32510 MiB, 16688 MiB, 15822 MiB iteration 200/ 210 | consumed samples: 819200 | elapsed time per iteration (ms): 11946.5 | tpt: 342.9 samples/s | global batch size: 4096 | lm loss: 8.931987E+00 | sop loss: 6.935722E-01 | loss scale: 262144.0 | grad norm: 3.905 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward-compute: 2843.47 | forward-recv: 760.76 | backward-compute: 6045.38 | backward-send: 19.06 | backward-send-forward-recv: 180.67 | backward-params-all-reduce: 15.78 | backward-embedding-all-reduce: 2062.68 | optimizer-copy-to-main-grad: 2.76 | optimizer-unscale-and-check-inf: 1.80 | optimizer-clip-main-grad: 2.45 | optimizer-copy-main-to-model-params: 1.42 | optimizer: 12.63 | batch-generator: 19.22 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ validation loss at the end of training for val data | lm loss value: 8.670823E+00 | lm loss PPL: 5.830296E+03 | sop loss value: 6.932384E-01 | sop loss PPL: 2.000183E+00 | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- validation loss at the end of training for test data | lm loss value: 8.629493E+00 | lm loss PPL: 5.594240E+03 | sop loss value: 6.943765E-01 | sop loss PPL: 2.002460E+00 | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- INFO:torch.distributed.elastic.agent.server.api:[default] worker group successfully finished. Waiting 300 seconds for other agents to finish. INFO:torch.distributed.elastic.agent.server.api:Local worker group finished (SUCCEEDED). Waiting 300 seconds for other agents to finish /opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/utils/store.py:70: FutureWarning: This is an experimental API and will be changed in future. warnings.warn( INFO:torch.distributed.elastic.agent.server.api:Done waiting for other agents. Elapsed: 0.0005855560302734375 seconds {"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 8, "group_rank": 1, "worker_id": "57838", "role": "default", "hostname": "iv-ybpu7pvmis5m57pm6ny1", "state": "SUCCEEDED", "total_run_time": 2642, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 2, \"entry_point\": \"python\", \"local_rank\": [0], \"role_rank\": [8], \"role_world_size\": [16]}", "agent_restarts": 0}} {"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 9, "group_rank": 1, "worker_id": "57839", "role": "default", "hostname": "iv-ybpu7pvmis5m57pm6ny1", "state": "SUCCEEDED", "total_run_time": 2642, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 2, \"entry_point\": \"python\", \"local_rank\": [1], \"role_rank\": [9], \"role_world_size\": [16]}", "agent_restarts": 0}} {"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 10, "group_rank": 1, "worker_id": "57840", "role": "default", "hostname": "iv-ybpu7pvmis5m57pm6ny1", "state": "SUCCEEDED", "total_run_time": 2642, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 2, \"entry_point\": \"python\", \"local_rank\": [2], \"role_rank\": [10], \"role_world_size\": [16]}", "agent_restarts": 0}} {"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 11, "group_rank": 1, "worker_id": "57841", "role": "default", "hostname": "iv-ybpu7pvmis5m57pm6ny1", "state": "SUCCEEDED", "total_run_time": 2642, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 2, \"entry_point\": \"python\", \"local_rank\": [3], \"role_rank\": [11], \"role_world_size\": [16]}", "agent_restarts": 0}} {"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 12, "group_rank": 1, "worker_id": "57842", "role": "default", "hostname": "iv-ybpu7pvmis5m57pm6ny1", "state": "SUCCEEDED", "total_run_time": 2642, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 2, \"entry_point\": \"python\", \"local_rank\": [4], \"role_rank\": [12], \"role_world_size\": [16]}", "agent_restarts": 0}} {"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 13, "group_rank": 1, "worker_id": "57843", "role": "default", "hostname": "iv-ybpu7pvmis5m57pm6ny1", "state": "SUCCEEDED", "total_run_time": 2642, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 2, \"entry_point\": \"python\", \"local_rank\": [5], \"role_rank\": [13], \"role_world_size\": [16]}", "agent_restarts": 0}} {"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 14, "group_rank": 1, "worker_id": "57844", "role": "default", "hostname": "iv-ybpu7pvmis5m57pm6ny1", "state": "SUCCEEDED", "total_run_time": 2642, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 2, \"entry_point\": \"python\", \"local_rank\": [6], \"role_rank\": [14], \"role_world_size\": [16]}", "agent_restarts": 0}} {"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 15, "group_rank": 1, "worker_id": "57845", "role": "default", "hostname": "iv-ybpu7pvmis5m57pm6ny1", "state": "SUCCEEDED", "total_run_time": 2642, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 2, \"entry_point\": \"python\", \"local_rank\": [7], \"role_rank\": [15], \"role_world_size\": [16]}", "agent_restarts": 0}} {"name": "torchelastic.worker.status.SUCCEEDED", "source": "AGENT", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": null, "group_rank": 1, "worker_id": null, "role": "default", "hostname": "iv-ybpu7pvmis5m57pm6ny1", "state": "SUCCEEDED", "total_run_time": 2642, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 2, \"entry_point\": \"python\"}", "agent_restarts": 0}} ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. *****************************************