The module torch.distributed.launch is deprecated and going to be removed in future.Migrate to torch.distributed.run WARNING:torch.distributed.run:--use_env is deprecated and will be removed in future releases. Please read local_rank from `os.environ('LOCAL_RANK')` instead. INFO:torch.distributed.launcher.api:Starting elastic_operator with launch configs: entrypoint : pretrain_bert.py min_nodes : 4 max_nodes : 4 nproc_per_node : 8 run_id : none rdzv_backend : static rdzv_endpoint : 198.18.8.30:6000 rdzv_configs : {'rank': 3, 'timeout': 900} max_restarts : 3 monitor_interval : 5 log_dir : None metrics_cfg : {} INFO:torch.distributed.elastic.agent.server.local_elastic_agent:log directory set to: /tmp/torchelastic_ir3byeal/none_3yb_qrcn INFO:torch.distributed.elastic.agent.server.api:[default] starting workers for entrypoint: python INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group /opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/utils/store.py:52: FutureWarning: This is an experimental API and will be changed in future. warnings.warn( INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result: restart_count=0 master_addr=198.18.8.30 master_port=6000 group_rank=3 group_world_size=4 local_ranks=[0, 1, 2, 3, 4, 5, 6, 7] role_ranks=[24, 25, 26, 27, 28, 29, 30, 31] global_ranks=[24, 25, 26, 27, 28, 29, 30, 31] role_world_sizes=[32, 32, 32, 32, 32, 32, 32, 32] global_world_sizes=[32, 32, 32, 32, 32, 32, 32, 32] INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_ir3byeal/none_3yb_qrcn/attempt_0/0/error.json INFO:torch.distributed.elastic.multiprocessing:Setting worker1 reply file to: /tmp/torchelastic_ir3byeal/none_3yb_qrcn/attempt_0/1/error.json INFO:torch.distributed.elastic.multiprocessing:Setting worker2 reply file to: /tmp/torchelastic_ir3byeal/none_3yb_qrcn/attempt_0/2/error.json INFO:torch.distributed.elastic.multiprocessing:Setting worker3 reply file to: /tmp/torchelastic_ir3byeal/none_3yb_qrcn/attempt_0/3/error.json INFO:torch.distributed.elastic.multiprocessing:Setting worker4 reply file to: /tmp/torchelastic_ir3byeal/none_3yb_qrcn/attempt_0/4/error.json INFO:torch.distributed.elastic.multiprocessing:Setting worker5 reply file to: /tmp/torchelastic_ir3byeal/none_3yb_qrcn/attempt_0/5/error.json INFO:torch.distributed.elastic.multiprocessing:Setting worker6 reply file to: /tmp/torchelastic_ir3byeal/none_3yb_qrcn/attempt_0/6/error.json INFO:torch.distributed.elastic.multiprocessing:Setting worker7 reply file to: /tmp/torchelastic_ir3byeal/none_3yb_qrcn/attempt_0/7/error.json [W ProcessGroupNCCL.cpp:1671] Rank 31 using best-guess GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. [W ProcessGroupNCCL.cpp:1671] Rank 29 using best-guess GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. [W ProcessGroupNCCL.cpp:1671] Rank 25 using best-guess GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. [W ProcessGroupNCCL.cpp:1671] Rank 26 using best-guess GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. [W ProcessGroupNCCL.cpp:1671] Rank 27 using best-guess GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. [W ProcessGroupNCCL.cpp:1671] Rank 24 using best-guess GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. [W ProcessGroupNCCL.cpp:1671] Rank 30 using best-guess GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. [W ProcessGroupNCCL.cpp:1671] Rank 28 using best-guess GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device. iv-2udaavw4l02thdv8lcrl:3373:3373 [0] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:3379:3379 [6] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:3376:3376 [3] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:3380:3380 [7] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:3373:3373 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-2udaavw4l02thdv8lcrl:3373:3373 [0] NCCL INFO P2P plugin IBext iv-2udaavw4l02thdv8lcrl:3373:3373 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-2udaavw4l02thdv8lcrl:3379:3379 [6] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-2udaavw4l02thdv8lcrl:3379:3379 [6] NCCL INFO P2P plugin IBext iv-2udaavw4l02thdv8lcrl:3379:3379 [6] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-2udaavw4l02thdv8lcrl:3376:3376 [3] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-2udaavw4l02thdv8lcrl:3376:3376 [3] NCCL INFO P2P plugin IBext iv-2udaavw4l02thdv8lcrl:3376:3376 [3] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-2udaavw4l02thdv8lcrl:3374:3374 [1] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:3377:3377 [4] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:3380:3380 [7] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-2udaavw4l02thdv8lcrl:3380:3380 [7] NCCL INFO P2P plugin IBext iv-2udaavw4l02thdv8lcrl:3380:3380 [7] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-2udaavw4l02thdv8lcrl:3375:3375 [2] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:3378:3378 [5] NCCL INFO Bootstrap : Using eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:3374:3374 [1] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-2udaavw4l02thdv8lcrl:3374:3374 [1] NCCL INFO P2P plugin IBext iv-2udaavw4l02thdv8lcrl:3374:3374 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-2udaavw4l02thdv8lcrl:3377:3377 [4] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-2udaavw4l02thdv8lcrl:3377:3377 [4] NCCL INFO P2P plugin IBext iv-2udaavw4l02thdv8lcrl:3377:3377 [4] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-2udaavw4l02thdv8lcrl:3375:3375 [2] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-2udaavw4l02thdv8lcrl:3375:3375 [2] NCCL INFO P2P plugin IBext iv-2udaavw4l02thdv8lcrl:3375:3375 [2] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-2udaavw4l02thdv8lcrl:3378:3378 [5] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so iv-2udaavw4l02thdv8lcrl:3378:3378 [5] NCCL INFO P2P plugin IBext iv-2udaavw4l02thdv8lcrl:3378:3378 [5] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. iv-2udaavw4l02thdv8lcrl:3373:3373 [0] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:3373:3373 [0] NCCL INFO Using network IBext iv-2udaavw4l02thdv8lcrl:3376:3376 [3] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:3376:3376 [3] NCCL INFO Using network IBext iv-2udaavw4l02thdv8lcrl:3377:3377 [4] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:3377:3377 [4] NCCL INFO Using network IBext iv-2udaavw4l02thdv8lcrl:3379:3379 [6] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:3379:3379 [6] NCCL INFO Using network IBext iv-2udaavw4l02thdv8lcrl:3374:3374 [1] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:3374:3374 [1] NCCL INFO Using network IBext iv-2udaavw4l02thdv8lcrl:3378:3378 [5] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:3378:3378 [5] NCCL INFO Using network IBext iv-2udaavw4l02thdv8lcrl:3380:3380 [7] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:3375:3375 [2] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB eth0:192.168.11.42<0> iv-2udaavw4l02thdv8lcrl:3375:3375 [2] NCCL INFO Using network IBext iv-2udaavw4l02thdv8lcrl:3380:3380 [7] NCCL INFO Using network IBext iv-2udaavw4l02thdv8lcrl:3376:3571 [3] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-2udaavw4l02thdv8lcrl:3379:3573 [6] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-2udaavw4l02thdv8lcrl:3380:3577 [7] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-2udaavw4l02thdv8lcrl:3378:3575 [5] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-2udaavw4l02thdv8lcrl:3374:3574 [1] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-2udaavw4l02thdv8lcrl:3373:3570 [0] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-2udaavw4l02thdv8lcrl:3375:3576 [2] NCCL INFO NCCL_IB_GID_INDEX set by environment to 3. iv-2udaavw4l02thdv8lcrl:3380:3577 [7] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-2udaavw4l02thdv8lcrl:3376:3571 [3] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-2udaavw4l02thdv8lcrl:3379:3573 [6] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-2udaavw4l02thdv8lcrl:3374:3574 [1] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-2udaavw4l02thdv8lcrl:3378:3575 [5] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-2udaavw4l02thdv8lcrl:3376:3571 [3] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-2udaavw4l02thdv8lcrl:3374:3574 [1] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-2udaavw4l02thdv8lcrl:3379:3573 [6] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-2udaavw4l02thdv8lcrl:3380:3577 [7] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-2udaavw4l02thdv8lcrl:3378:3575 [5] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-2udaavw4l02thdv8lcrl:3375:3576 [2] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-2udaavw4l02thdv8lcrl:3375:3576 [2] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-2udaavw4l02thdv8lcrl:3373:3570 [0] NCCL INFO NCCL_IB_TIMEOUT set by environment to 23. iv-2udaavw4l02thdv8lcrl:3373:3570 [0] NCCL INFO NCCL_IB_RETRY_CNT set by environment to 7. iv-2udaavw4l02thdv8lcrl:3373:3570 [0] NCCL INFO Trees [0] 26/-1/-1->24->31 [1] 26/-1/-1->24->31 iv-2udaavw4l02thdv8lcrl:3373:3570 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3375:3576 [2] NCCL INFO Trees [0] 27/-1/-1->26->24 [1] 27/-1/-1->26->24 iv-2udaavw4l02thdv8lcrl:3374:3574 [1] NCCL INFO Trees [0] 30/-1/-1->25->27 [1] 30/-1/-1->25->27 iv-2udaavw4l02thdv8lcrl:3375:3576 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3374:3574 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO Trees [0] 29/-1/-1->28->20 [1] 29/12/-1->28->-1 iv-2udaavw4l02thdv8lcrl:3376:3571 [3] NCCL INFO Trees [0] 25/-1/-1->27->26 [1] 25/-1/-1->27->26 iv-2udaavw4l02thdv8lcrl:3376:3571 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3378:3575 [5] NCCL INFO Trees [0] 31/-1/-1->29->28 [1] 31/-1/-1->29->28 iv-2udaavw4l02thdv8lcrl:3379:3573 [6] NCCL INFO Trees [0] -1/-1/-1->30->25 [1] -1/-1/-1->30->25 iv-2udaavw4l02thdv8lcrl:3380:3577 [7] NCCL INFO Trees [0] 24/-1/-1->31->29 [1] 24/-1/-1->31->29 iv-2udaavw4l02thdv8lcrl:3379:3573 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3378:3575 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3380:3577 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3375:3576 [2] NCCL INFO Channel 00 : 26[67010] -> 29[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3576 [2] NCCL INFO Channel 01 : 26[67010] -> 29[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3573 [6] NCCL INFO Channel 00 : 30[6b010] -> 31[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3373:3570 [0] NCCL INFO Channel 00 : 24[65010] -> 25[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO Channel 00 : 28[69010] -> 30[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3573 [6] NCCL INFO Channel 01 : 30[6b010] -> 31[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3373:3570 [0] NCCL INFO Channel 01 : 24[65010] -> 25[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO Channel 01 : 28[69010] -> 30[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3574 [1] NCCL INFO Channel 00 : 25[65020] -> 27[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3575 [5] NCCL INFO Channel 00 : 29[69020] -> 4[69010] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3380:3577 [7] NCCL INFO Channel 00 : 31[6b020] -> 24[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3574 [1] NCCL INFO Channel 01 : 25[65020] -> 27[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3575 [5] NCCL INFO Channel 01 : 29[69020] -> 4[69010] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3380:3577 [7] NCCL INFO Channel 01 : 31[6b020] -> 24[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3574 [1] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3379:3573 [6] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3373:3570 [0] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3380:3577 [7] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3374:3574 [1] NCCL INFO Channel 00 : 25[65020] -> 30[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3571 [3] NCCL INFO Channel 00 : 27[67020] -> 26[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO Channel 00 : 21[69020] -> 28[69010] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3373:3570 [0] NCCL INFO Channel 00 : 24[65010] -> 26[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3574 [1] NCCL INFO Channel 01 : 25[65020] -> 30[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3571 [3] NCCL INFO Channel 01 : 27[67020] -> 26[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO Channel 01 : 21[69020] -> 28[69010] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3373:3570 [0] NCCL INFO Channel 01 : 24[65010] -> 26[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3571 [3] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3375:3576 [2] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3375:3576 [2] NCCL INFO Channel 00 : 26[67010] -> 27[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3573 [6] NCCL INFO Channel 00 : 30[6b010] -> 25[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3576 [2] NCCL INFO Channel 01 : 26[67010] -> 27[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3575 [5] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3379:3573 [6] NCCL INFO Channel 01 : 30[6b010] -> 25[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3379:3573 [6] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3379:3573 [6] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3379:3573 [6] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO Channel 00 : 28[69010] -> 29[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3571 [3] NCCL INFO Channel 00 : 27[67020] -> 25[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO Channel 01 : 28[69010] -> 29[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3571 [3] NCCL INFO Channel 01 : 27[67020] -> 25[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3575 [5] NCCL INFO Channel 00 : 29[69020] -> 31[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO Channel 00 : 20[69010] -> 28[69010] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3375:3576 [2] NCCL INFO Channel 00 : 26[67010] -> 24[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3373:3570 [0] NCCL INFO Channel 00 : 24[65010] -> 31[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3575 [5] NCCL INFO Channel 01 : 29[69020] -> 31[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3576 [2] NCCL INFO Channel 01 : 26[67010] -> 24[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3574 [1] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3374:3574 [1] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3374:3574 [1] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3373:3570 [0] NCCL INFO Channel 01 : 24[65010] -> 31[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3571 [3] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3376:3571 [3] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3376:3571 [3] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3374:3574 [1] NCCL INFO Channel 01 : 25[65020] -> 28[69010] via P2P/indirect/30[6b010] iv-2udaavw4l02thdv8lcrl:3376:3571 [3] NCCL INFO Channel 00 : 27[67020] -> 29[69020] via P2P/indirect/28[69010] iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO Channel 01 : 12[69010] -> 28[69010] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO Channel 01 : 28[69010] -> 12[69010] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3380:3577 [7] NCCL INFO Channel 00 : 31[6b020] -> 29[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3577 [7] NCCL INFO Channel 01 : 31[6b020] -> 29[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3577 [7] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3380:3577 [7] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3380:3577 [7] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3378:3575 [5] NCCL INFO Channel 00 : 29[69020] -> 28[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3576 [2] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3375:3576 [2] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3375:3576 [2] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3373:3570 [0] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3373:3570 [0] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3373:3570 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3373:3570 [0] NCCL INFO Channel 00 : 24[65010] -> 28[69010] via P2P/indirect/27[67020] iv-2udaavw4l02thdv8lcrl:3378:3575 [5] NCCL INFO Channel 01 : 29[69020] -> 28[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3576 [2] NCCL INFO Channel 00 : 26[67010] -> 28[69010] via P2P/indirect/29[69020] iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO Channel 00 : 28[69010] -> 20[69010] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3378:3575 [5] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3378:3575 [5] NCCL INFO threadThresholds 8/8/64 | 256/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3378:3575 [5] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3375:3576 [2] NCCL INFO Channel 00 : 26[67010] -> 30[6b010] via P2P/indirect/25[65020] iv-2udaavw4l02thdv8lcrl:3376:3571 [3] NCCL INFO Channel 01 : 27[67020] -> 30[6b010] via P2P/indirect/25[65020] iv-2udaavw4l02thdv8lcrl:3374:3574 [1] NCCL INFO Channel 00 : 25[65020] -> 29[69020] via P2P/indirect/30[6b010] iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO Channel 00 : 28[69010] -> 24[65010] via P2P/indirect/31[6b020] iv-2udaavw4l02thdv8lcrl:3376:3571 [3] NCCL INFO Channel 00 : 27[67020] -> 31[6b020] via P2P/indirect/24[65010] iv-2udaavw4l02thdv8lcrl:3373:3570 [0] NCCL INFO Channel 01 : 24[65010] -> 29[69020] via P2P/indirect/31[6b020] iv-2udaavw4l02thdv8lcrl:3374:3574 [1] NCCL INFO Channel 00 : 25[65020] -> 31[6b020] via P2P/indirect/24[65010] iv-2udaavw4l02thdv8lcrl:3375:3576 [2] NCCL INFO Channel 01 : 26[67010] -> 31[6b020] via P2P/indirect/24[65010] iv-2udaavw4l02thdv8lcrl:3378:3575 [5] NCCL INFO Channel 01 : 29[69020] -> 24[65010] via P2P/indirect/31[6b020] iv-2udaavw4l02thdv8lcrl:3373:3570 [0] NCCL INFO Channel 00 : 24[65010] -> 30[6b010] via P2P/indirect/25[65020] iv-2udaavw4l02thdv8lcrl:3379:3573 [6] NCCL INFO Channel 00 : 30[6b010] -> 24[65010] via P2P/indirect/31[6b020] iv-2udaavw4l02thdv8lcrl:3380:3577 [7] NCCL INFO Channel 00 : 31[6b020] -> 25[65020] via P2P/indirect/30[6b010] iv-2udaavw4l02thdv8lcrl:3379:3573 [6] NCCL INFO Channel 00 : 30[6b010] -> 26[67010] via P2P/indirect/29[69020] iv-2udaavw4l02thdv8lcrl:3380:3577 [7] NCCL INFO Channel 01 : 31[6b020] -> 26[67010] via P2P/indirect/24[65010] iv-2udaavw4l02thdv8lcrl:3378:3575 [5] NCCL INFO Channel 00 : 29[69020] -> 25[65020] via P2P/indirect/30[6b010] iv-2udaavw4l02thdv8lcrl:3378:3575 [5] NCCL INFO Channel 00 : 29[69020] -> 27[67020] via P2P/indirect/26[67010] iv-2udaavw4l02thdv8lcrl:3380:3577 [7] NCCL INFO Channel 00 : 31[6b020] -> 27[67020] via P2P/indirect/24[65010] iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO Channel 01 : 28[69010] -> 25[65020] via P2P/indirect/30[6b010] iv-2udaavw4l02thdv8lcrl:3379:3573 [6] NCCL INFO Channel 01 : 30[6b010] -> 27[67020] via P2P/indirect/25[65020] iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO Channel 00 : 28[69010] -> 26[67010] via P2P/indirect/27[67020] iv-2udaavw4l02thdv8lcrl:3375:3576 [2] NCCL INFO comm 0x7f1c7c008fb0 rank 26 nranks 32 cudaDev 2 busId 67010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3377:3572 [4] NCCL INFO comm 0x7fd478008fb0 rank 28 nranks 32 cudaDev 4 busId 69010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3376:3571 [3] NCCL INFO comm 0x7fc6b0008fb0 rank 27 nranks 32 cudaDev 3 busId 67020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3373:3570 [0] NCCL INFO comm 0x7f4c60008fb0 rank 24 nranks 32 cudaDev 0 busId 65010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3378:3575 [5] NCCL INFO comm 0x7f5ee0008fb0 rank 29 nranks 32 cudaDev 5 busId 69020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3374:3574 [1] NCCL INFO comm 0x7f7ef8008fb0 rank 25 nranks 32 cudaDev 1 busId 65020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3380:3577 [7] NCCL INFO comm 0x7f50bc008fb0 rank 31 nranks 32 cudaDev 7 busId 6b020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3379:3573 [6] NCCL INFO comm 0x7faea8008fb0 rank 30 nranks 32 cudaDev 6 busId 6b010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3379:3627 [6] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-2udaavw4l02thdv8lcrl:3379:3627 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3379:3627 [6] NCCL INFO Channel 00 : 0[6b010] -> 1[6b010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3379:3627 [6] NCCL INFO Channel 01 : 0[6b010] -> 1[6b010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3377:3628 [4] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-2udaavw4l02thdv8lcrl:3377:3628 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3380:3631 [7] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-2udaavw4l02thdv8lcrl:3380:3631 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3377:3628 [4] NCCL INFO Channel 00 : 0[69010] -> 1[69010] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3377:3628 [4] NCCL INFO Channel 01 : 0[69010] -> 1[69010] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3376:3635 [3] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-2udaavw4l02thdv8lcrl:3376:3635 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3377:3628 [4] NCCL INFO Channel 00 : 1[69010] -> 0[69010] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3374:3634 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-2udaavw4l02thdv8lcrl:3374:3634 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3377:3628 [4] NCCL INFO Channel 01 : 1[69010] -> 0[69010] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3379:3627 [6] NCCL INFO Channel 00 : 1[6b010] -> 0[6b010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:3633 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-2udaavw4l02thdv8lcrl:3373:3633 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3375:3639 [2] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-2udaavw4l02thdv8lcrl:3375:3639 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3378:3641 [5] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-2udaavw4l02thdv8lcrl:3378:3641 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3380:3631 [7] NCCL INFO Channel 00 : 0[6b020] -> 1[6b020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3379:3627 [6] NCCL INFO Channel 01 : 1[6b010] -> 0[6b010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3376:3635 [3] NCCL INFO Channel 00 : 0[67020] -> 1[67020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:3634 [1] NCCL INFO Channel 00 : 0[65020] -> 1[65020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3376:3635 [3] NCCL INFO Channel 01 : 0[67020] -> 1[67020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:3633 [0] NCCL INFO Channel 00 : 0[65010] -> 1[65010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:3639 [2] NCCL INFO Channel 00 : 0[67010] -> 1[67010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3378:3641 [5] NCCL INFO Channel 00 : 0[69020] -> 1[69020] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3380:3631 [7] NCCL INFO Channel 01 : 0[6b020] -> 1[6b020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:3634 [1] NCCL INFO Channel 01 : 0[65020] -> 1[65020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3378:3641 [5] NCCL INFO Channel 01 : 0[69020] -> 1[69020] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3378:3641 [5] NCCL INFO Channel 00 : 1[69020] -> 0[69020] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3376:3635 [3] NCCL INFO Channel 00 : 1[67020] -> 0[67020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:3639 [2] NCCL INFO Channel 01 : 0[67010] -> 1[67010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:3633 [0] NCCL INFO Channel 01 : 0[65010] -> 1[65010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3379:3627 [6] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3379:3627 [6] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3379:3627 [6] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3379:3627 [6] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3379:3627 [6] NCCL INFO comm 0x7fae7c008fb0 rank 1 nranks 2 cudaDev 6 busId 6b010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3380:3631 [7] NCCL INFO Channel 00 : 1[6b020] -> 0[6b020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:3634 [1] NCCL INFO Channel 00 : 1[65020] -> 0[65020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3376:3635 [3] NCCL INFO Channel 01 : 1[67020] -> 0[67020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:3639 [2] NCCL INFO Channel 00 : 1[67010] -> 0[67010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:3633 [0] NCCL INFO Channel 00 : 1[65010] -> 0[65010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3378:3641 [5] NCCL INFO Channel 01 : 1[69020] -> 0[69020] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3380:3631 [7] NCCL INFO Channel 01 : 1[6b020] -> 0[6b020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:3634 [1] NCCL INFO Channel 01 : 1[65020] -> 0[65020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:3639 [2] NCCL INFO Channel 01 : 1[67010] -> 0[67010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:3633 [0] NCCL INFO Channel 01 : 1[65010] -> 0[65010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3376:3635 [3] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3376:3635 [3] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3376:3635 [3] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3376:3635 [3] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3376:3635 [3] NCCL INFO comm 0x7fc684008fb0 rank 1 nranks 2 cudaDev 3 busId 67020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3380:3631 [7] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3380:3631 [7] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3380:3631 [7] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3380:3631 [7] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3380:3631 [7] NCCL INFO comm 0x7f5084008fb0 rank 1 nranks 2 cudaDev 7 busId 6b020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3375:3639 [2] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3375:3639 [2] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3375:3639 [2] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3375:3639 [2] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3375:3639 [2] NCCL INFO comm 0x7f1c50008fb0 rank 1 nranks 2 cudaDev 2 busId 67010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3373:3633 [0] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3373:3633 [0] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3373:3633 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3373:3633 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3373:3633 [0] NCCL INFO comm 0x7f4c34008fb0 rank 1 nranks 2 cudaDev 0 busId 65010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3374:3634 [1] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3374:3634 [1] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3374:3634 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3374:3634 [1] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3374:3634 [1] NCCL INFO comm 0x7f7ec8008fb0 rank 1 nranks 2 cudaDev 1 busId 65020 - Init COMPLETE > number of parameters on (tensor, pipeline) model parallel rank (0, 3): 99461890 iv-2udaavw4l02thdv8lcrl:3377:3628 [4] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3377:3628 [4] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3377:3628 [4] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3377:3628 [4] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3377:3628 [4] NCCL INFO comm 0x7fd42c008fb0 rank 1 nranks 2 cudaDev 4 busId 69010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3378:3641 [5] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3378:3641 [5] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3378:3641 [5] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3378:3641 [5] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3378:3641 [5] NCCL INFO comm 0x7f5eb4008fb0 rank 1 nranks 2 cudaDev 5 busId 69020 - Init COMPLETE NCCL version 2.10.3+cuda11.4 iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Trees [0] 6/-1/-1->1->3 [1] 6/-1/-1->1->3 [2] 3/-1/-1->1->6 [3] 3/-1/-1->1->6 [4] 2/-1/-1->1->0 [5] -1/-1/-1->1->2 [6] 6/-1/-1->1->3 [7] 6/-1/-1->1->3 [8] 3/-1/-1->1->6 [9] 3/-1/-1->1->6 [10] 2/-1/-1->1->0 [11] -1/-1/-1->1->2 iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Trees [0] 1/-1/-1->3->2 [1] 1/-1/-1->3->2 [2] 2/-1/-1->3->1 [3] 2/-1/-1->3->1 [4] -1/-1/-1->3->4 [5] 4/-1/-1->3->0 [6] 1/-1/-1->3->2 [7] 1/-1/-1->3->2 [8] 2/-1/-1->3->1 [9] 2/-1/-1->3->1 [10] -1/-1/-1->3->4 [11] 4/-1/-1->3->0 iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Trees [0] 5/-1/-1->4->6 [1] 5/-1/-1->4->6 [2] 6/-1/-1->4->5 [3] 6/-1/-1->4->5 [4] 3/-1/-1->4->7 [5] 7/-1/-1->4->3 [6] 5/-1/-1->4->6 [7] 5/-1/-1->4->6 [8] 6/-1/-1->4->5 [9] 6/-1/-1->4->5 [10] 3/-1/-1->4->7 [11] 7/-1/-1->4->3 iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Trees [0] 3/-1/-1->2->0 [1] 3/-1/-1->2->0 [2] -1/-1/-1->2->3 [3] -1/-1/-1->2->3 [4] 5/-1/-1->2->1 [5] 1/-1/-1->2->5 [6] 3/-1/-1->2->0 [7] 3/-1/-1->2->0 [8] -1/-1/-1->2->3 [9] -1/-1/-1->2->3 [10] 5/-1/-1->2->1 [11] 1/-1/-1->2->5 iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Trees [0] 7/-1/-1->5->4 [1] 7/-1/-1->5->4 [2] 4/-1/-1->5->7 [3] 4/-1/-1->5->7 [4] 6/-1/-1->5->2 [5] 2/-1/-1->5->6 [6] 7/-1/-1->5->4 [7] 7/-1/-1->5->4 [8] 4/-1/-1->5->7 [9] 4/-1/-1->5->7 [10] 6/-1/-1->5->2 [11] 2/-1/-1->5->6 iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Trees [0] 4/-1/-1->6->1 [1] 4/-1/-1->6->1 [2] 1/-1/-1->6->4 [3] 1/-1/-1->6->4 [4] 7/-1/-1->6->5 [5] 5/-1/-1->6->7 [6] 4/-1/-1->6->1 [7] 4/-1/-1->6->1 [8] 1/-1/-1->6->4 [9] 1/-1/-1->6->4 [10] 7/-1/-1->6->5 [11] 5/-1/-1->6->7 iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 00/12 : 0 2 3 1 6 4 5 7 iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 01/12 : 0 2 3 1 6 4 5 7 iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 02/12 : 0 7 5 4 6 1 3 2 iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Trees [0] -1/-1/-1->7->5 [1] -1/-1/-1->7->5 [2] 5/-1/-1->7->0 [3] 5/-1/-1->7->0 [4] 4/-1/-1->7->6 [5] 6/-1/-1->7->4 [6] -1/-1/-1->7->5 [7] -1/-1/-1->7->5 [8] 5/-1/-1->7->0 [9] 5/-1/-1->7->0 [10] 4/-1/-1->7->6 [11] 6/-1/-1->7->4 iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 03/12 : 0 7 5 4 6 1 3 2 iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 04/12 : 0 1 2 5 6 7 4 3 iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 05/12 : 0 3 4 7 6 5 2 1 iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 06/12 : 0 2 3 1 6 4 5 7 iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 07/12 : 0 2 3 1 6 4 5 7 iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 08/12 : 0 7 5 4 6 1 3 2 iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 09/12 : 0 7 5 4 6 1 3 2 iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 10/12 : 0 1 2 5 6 7 4 3 iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 11/12 : 0 3 4 7 6 5 2 1 iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Trees [0] 2/-1/-1->0->-1 [1] 2/-1/-1->0->-1 [2] 7/-1/-1->0->-1 [3] 7/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 3/-1/-1->0->-1 [6] 2/-1/-1->0->-1 [7] 2/-1/-1->0->-1 [8] 7/-1/-1->0->-1 [9] 7/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 3/-1/-1->0->-1 iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 04 : 1[65020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 00 : 4[69010] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 04 : 6[6b010] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 00 : 7[6b020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 10 : 1[65020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 01 : 4[69010] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 10 : 6[6b010] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 01 : 7[6b020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 06 : 4[69010] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 05 : 3[67020] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 04 : 5[69020] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 06 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 04 : 0[65010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 06 : 7[6b020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 07 : 4[69010] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 11 : 3[67020] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 10 : 5[69020] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 07 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 07 : 7[6b020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 10 : 0[65010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 02 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 02 : 4[69010] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 00 : 5[69020] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 03 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 00 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 03 : 4[69010] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 01 : 5[69020] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 08 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 01 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 08 : 4[69010] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 06 : 5[69020] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 09 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 06 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 09 : 4[69010] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 07 : 5[69020] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 07 : 0[65010] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 02 : 6[6b010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 05 : 4[69010] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 04 : 2[67010] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 05 : 0[65010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 03 : 6[6b010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 11 : 4[69010] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 11 : 0[65010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 10 : 2[67010] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 08 : 6[6b010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 09 : 6[6b010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 04 : 3[67020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 05 : 5[69020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 04 : 7[6b020] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 10 : 3[67020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 11 : 5[69020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 10 : 7[6b020] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 00 : 1[65020] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 00 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 02 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 02 : 7[6b020] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 01 : 1[65020] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 01 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 03 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 03 : 7[6b020] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 06 : 1[65020] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 06 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 08 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 08 : 7[6b020] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 07 : 1[65020] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 07 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 09 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 09 : 7[6b020] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 00 : 6[6b010] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 01 : 6[6b010] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 06 : 6[6b010] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 02 : 5[69020] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 02 : 0[65010] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 07 : 6[6b010] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 03 : 0[65010] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 03 : 5[69020] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 05 : 7[6b020] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 08 : 0[65010] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 08 : 5[69020] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 05 : 2[67010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 11 : 7[6b020] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 09 : 0[65010] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 09 : 5[69020] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 11 : 2[67010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 05 : 1[65020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 02 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 05 : 6[6b010] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 03 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 11 : 1[65020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 11 : 6[6b010] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 04 : 4[69010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 08 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 10 : 4[69010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 09 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 05 : 1[65020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 11 : 1[65020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 05 : 6[6b010] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 02 : 7[6b020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 11 : 6[6b010] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 03 : 7[6b020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 02 : 4[69010] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 02 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 08 : 7[6b020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 03 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 03 : 4[69010] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 09 : 7[6b020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 05 : 5[69020] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 04 : 3[67020] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 08 : 4[69010] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 08 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 10 : 3[67020] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 11 : 5[69020] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 09 : 4[69010] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 09 : 2[67010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 00 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 01 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 06 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 00 : 4[69010] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 02 : 5[69020] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 05 : 2[67010] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 07 : 1[65020] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 01 : 4[69010] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 03 : 5[69020] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 11 : 2[67010] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 06 : 4[69010] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 08 : 5[69020] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 07 : 4[69010] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 09 : 5[69020] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 05 : 3[67020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 00 : 6[6b010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 04 : 4[69010] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 11 : 3[67020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 01 : 6[6b010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 10 : 4[69010] -> 7[6b020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 06 : 6[6b010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 07 : 6[6b010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 02 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 05 : 7[6b020] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 04 : 5[69020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 03 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 11 : 7[6b020] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 10 : 5[69020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 08 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 09 : 3[67020] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 02 : 1[65020] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 00 : 7[6b020] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 00 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 03 : 1[65020] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 01 : 7[6b020] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 01 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 08 : 1[65020] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 06 : 7[6b020] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 06 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 09 : 1[65020] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 07 : 7[6b020] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 07 : 2[67010] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 02 : 6[6b010] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 04 : 7[6b020] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 03 : 6[6b010] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 10 : 7[6b020] -> 6[6b010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 08 : 6[6b010] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 00 : 5[69020] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 09 : 6[6b010] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 01 : 5[69020] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 06 : 5[69020] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 04 : 2[67010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 07 : 5[69020] -> 4[69010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 10 : 2[67010] -> 1[65020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 04 : 6[6b010] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 04 : 1[65020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 01 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 10 : 6[6b010] -> 5[69020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 10 : 1[65020] -> 0[65010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 05 : 4[69010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 06 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 11 : 4[69010] -> 3[67020] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 07 : 3[67020] -> 2[67010] via P2P/IPC iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 02 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010] iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 10 : 7[6b020] -> 1[65020] via P2P/indirect/6[6b010] iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 02 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020] iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 10 : 6[6b010] -> 0[65010] via P2P/indirect/7[6b020] iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 02 : 2[67010] -> 4[69010] via P2P/indirect/5[69020] iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 02 : 3[67020] -> 5[69020] via P2P/indirect/4[69010] iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 10 : 2[67010] -> 4[69010] via P2P/indirect/5[69020] iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 03 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010] iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 10 : 3[67020] -> 5[69020] via P2P/indirect/4[69010] iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 03 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010] iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 11 : 1[65020] -> 4[69010] via P2P/indirect/6[6b010] iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 03 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020] iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 03 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020] iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 11 : 7[6b020] -> 2[67010] via P2P/indirect/0[65010] iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 11 : 3[67020] -> 6[6b010] via P2P/indirect/1[65020] iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 11 : 5[69020] -> 0[65010] via P2P/indirect/7[6b020] iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 04 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020] iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 04 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020] iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 12 : 2[67010] -> 6[6b010] via P2P/indirect/1[65020] iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 12 : 4[69010] -> 0[65010] via P2P/indirect/7[6b020] iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 04 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010] iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 04 : 0[65010] -> 4[69010] via P2P/indirect/3[67020] iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 04 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010] iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 04 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020] iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 04 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010] iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 04 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010] iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 12 : 0[65010] -> 4[69010] via P2P/indirect/3[67020] iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO Channel 12 : 3[67020] -> 7[6b020] via P2P/indirect/0[65010] iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 12 : 1[65020] -> 5[69020] via P2P/indirect/6[6b010] iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 12 : 6[6b010] -> 2[67010] via P2P/indirect/5[69020] iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 12 : 5[69020] -> 1[65020] via P2P/indirect/6[6b010] iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO Channel 12 : 7[6b020] -> 3[67020] via P2P/indirect/0[65010] iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 05 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010] iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 05 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010] iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 13 : 4[69010] -> 1[65020] via P2P/indirect/6[6b010] iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO Channel 13 : 2[67010] -> 7[6b020] via P2P/indirect/0[65010] iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 05 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020] iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 05 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020] iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO Channel 13 : 6[6b010] -> 3[67020] via P2P/indirect/1[65020] iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 13 : 0[65010] -> 5[69020] via P2P/indirect/7[6b020] iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 06 : 4[69010] -> 2[67010] via P2P/indirect/3[67020] iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 06 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010] iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 06 : 5[69020] -> 3[67020] via P2P/indirect/2[67010] iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 06 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020] iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO Channel 14 : 4[69010] -> 2[67010] via P2P/indirect/3[67020] iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO Channel 14 : 5[69020] -> 3[67020] via P2P/indirect/2[67010] iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO Channel 14 : 1[65020] -> 7[6b020] via P2P/indirect/0[65010] iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO Channel 14 : 0[65010] -> 6[6b010] via P2P/indirect/1[65020] iv-2udaavw4l02thdv8lcrl:3378:3659 [5] NCCL INFO comm 0x7f5eb40be010 rank 5 nranks 8 cudaDev 5 busId 69020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3380:3658 [7] NCCL INFO comm 0x7f50840d3010 rank 7 nranks 8 cudaDev 7 busId 6b020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3374:3654 [1] NCCL INFO comm 0x7f7ec80f6010 rank 1 nranks 8 cudaDev 1 busId 65020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3376:3656 [3] NCCL INFO comm 0x7fc6840f6010 rank 3 nranks 8 cudaDev 3 busId 67020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3377:3655 [4] NCCL INFO comm 0x7fd42c0be010 rank 4 nranks 8 cudaDev 4 busId 69010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3373:3652 [0] NCCL INFO comm 0x7f4bf8008fb0 rank 0 nranks 8 cudaDev 0 busId 65010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3379:3657 [6] NCCL INFO comm 0x7fae7c0d3010 rank 6 nranks 8 cudaDev 6 busId 6b010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3375:3653 [2] NCCL INFO comm 0x7f1c500f6010 rank 2 nranks 8 cudaDev 2 busId 67010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3373:3373 [0] NCCL INFO Launch mode Parallel iv-2udaavw4l02thdv8lcrl:3379:3676 [6] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 1/-1/-1->3->-1 iv-2udaavw4l02thdv8lcrl:3379:3676 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3374:3679 [1] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 1/-1/-1->3->-1 iv-2udaavw4l02thdv8lcrl:3374:3679 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3373:3681 [0] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 1/-1/-1->3->-1 iv-2udaavw4l02thdv8lcrl:3373:3681 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3379:3676 [6] NCCL INFO Channel 00 : 2[6b010] -> 3[6b010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:3678 [2] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 1/-1/-1->3->-1 iv-2udaavw4l02thdv8lcrl:3375:3678 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3374:3679 [1] NCCL INFO Channel 00 : 2[65020] -> 3[65020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3377:3683 [4] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 1/-1/-1->3->-1 iv-2udaavw4l02thdv8lcrl:3377:3683 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3378:3680 [5] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 1/-1/-1->3->-1 iv-2udaavw4l02thdv8lcrl:3378:3680 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3380:3677 [7] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 1/-1/-1->3->-1 iv-2udaavw4l02thdv8lcrl:3380:3677 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3376:3682 [3] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 1/-1/-1->3->-1 iv-2udaavw4l02thdv8lcrl:3376:3682 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3377:3683 [4] NCCL INFO Channel 00 : 2[69010] -> 3[69010] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3378:3680 [5] NCCL INFO Channel 00 : 2[69020] -> 3[69020] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3377:3683 [4] NCCL INFO Channel 01 : 2[69010] -> 3[69010] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3378:3680 [5] NCCL INFO Channel 01 : 2[69020] -> 3[69020] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3377:3683 [4] NCCL INFO Channel 00 : 3[69010] -> 0[69010] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3373:3681 [0] NCCL INFO Channel 00 : 2[65010] -> 3[65010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:3679 [1] NCCL INFO Channel 01 : 2[65020] -> 3[65020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3378:3680 [5] NCCL INFO Channel 00 : 3[69020] -> 0[69020] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3379:3676 [6] NCCL INFO Channel 01 : 2[6b010] -> 3[6b010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3377:3683 [4] NCCL INFO Channel 01 : 3[69010] -> 0[69010] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3378:3680 [5] NCCL INFO Channel 01 : 3[69020] -> 0[69020] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3375:3678 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:3677 [7] NCCL INFO Channel 00 : 2[6b020] -> 3[6b020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:3681 [0] NCCL INFO Channel 01 : 2[65010] -> 3[65010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:3679 [1] NCCL INFO Channel 00 : 3[65020] -> 0[65020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3379:3676 [6] NCCL INFO Channel 00 : 3[6b010] -> 0[6b010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:3677 [7] NCCL INFO Channel 01 : 2[6b020] -> 3[6b020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3376:3682 [3] NCCL INFO Channel 00 : 2[67020] -> 3[67020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:3678 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:3677 [7] NCCL INFO Channel 00 : 3[6b020] -> 0[6b020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:3681 [0] NCCL INFO Channel 00 : 3[65010] -> 0[65010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:3679 [1] NCCL INFO Channel 01 : 3[65020] -> 0[65020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3379:3676 [6] NCCL INFO Channel 01 : 3[6b010] -> 0[6b010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3376:3682 [3] NCCL INFO Channel 01 : 2[67020] -> 3[67020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:3678 [2] NCCL INFO Channel 00 : 3[67010] -> 0[67010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:3679 [1] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3379:3676 [6] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3373:3681 [0] NCCL INFO Channel 01 : 3[65010] -> 0[65010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:3677 [7] NCCL INFO Channel 01 : 3[6b020] -> 0[6b020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:3678 [2] NCCL INFO Channel 01 : 3[67010] -> 0[67010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3376:3682 [3] NCCL INFO Channel 00 : 3[67020] -> 0[67020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:3679 [1] NCCL INFO Channel 01 : 1[65020] -> 3[65020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:3677 [7] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3379:3676 [6] NCCL INFO Channel 01 : 1[6b010] -> 3[6b010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:3681 [0] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3376:3682 [3] NCCL INFO Channel 01 : 3[67020] -> 0[67020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:3679 [1] NCCL INFO Channel 01 : 3[65020] -> 1[65020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3376:3682 [3] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3380:3677 [7] NCCL INFO Channel 01 : 1[6b020] -> 3[6b020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:3678 [2] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3379:3676 [6] NCCL INFO Channel 01 : 3[6b010] -> 1[6b010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:3681 [0] NCCL INFO Channel 01 : 1[65010] -> 3[65010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3376:3682 [3] NCCL INFO Channel 01 : 1[67020] -> 3[67020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:3679 [1] NCCL INFO Channel 00 : 3[65020] -> 2[65020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:3677 [7] NCCL INFO Channel 01 : 3[6b020] -> 1[6b020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:3678 [2] NCCL INFO Channel 01 : 1[67010] -> 3[67010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:3681 [0] NCCL INFO Channel 01 : 3[65010] -> 1[65010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3379:3676 [6] NCCL INFO Channel 00 : 3[6b010] -> 2[6b010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3376:3682 [3] NCCL INFO Channel 01 : 3[67020] -> 1[67020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:3678 [2] NCCL INFO Channel 01 : 3[67010] -> 1[67010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:3677 [7] NCCL INFO Channel 00 : 3[6b020] -> 2[6b020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:3681 [0] NCCL INFO Channel 00 : 3[65010] -> 2[65010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:3679 [1] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3374:3679 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3374:3679 [1] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3374:3679 [1] NCCL INFO comm 0x7f7ec85093e0 rank 3 nranks 4 cudaDev 1 busId 65020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3379:3676 [6] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3379:3676 [6] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3379:3676 [6] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3379:3676 [6] NCCL INFO comm 0x7fae7c4e6d70 rank 3 nranks 4 cudaDev 6 busId 6b010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3377:3683 [4] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3376:3682 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:3677 [7] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3380:3677 [7] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3380:3677 [7] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3380:3677 [7] NCCL INFO comm 0x7f50844e4440 rank 3 nranks 4 cudaDev 7 busId 6b020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3373:3681 [0] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3373:3681 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3373:3681 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3373:3681 [0] NCCL INFO comm 0x7f4bf82d6bd0 rank 3 nranks 4 cudaDev 0 busId 65010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3375:3678 [2] NCCL INFO Channel 00 : 3[67010] -> 2[67010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:3678 [2] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3375:3678 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3375:3678 [2] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3375:3678 [2] NCCL INFO comm 0x7f1c50505f90 rank 3 nranks 4 cudaDev 2 busId 67010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3377:3683 [4] NCCL INFO Channel 01 : 1[69010] -> 3[69010] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3378:3680 [5] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3377:3683 [4] NCCL INFO Channel 01 : 3[69010] -> 1[69010] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3378:3680 [5] NCCL INFO Channel 01 : 1[69020] -> 3[69020] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3378:3680 [5] NCCL INFO Channel 01 : 3[69020] -> 1[69020] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3376:3682 [3] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3376:3682 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3376:3682 [3] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3376:3682 [3] NCCL INFO comm 0x7fc684509240 rank 3 nranks 4 cudaDev 3 busId 67020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3377:3683 [4] NCCL INFO Channel 00 : 3[69010] -> 2[69010] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3378:3680 [5] NCCL INFO Channel 00 : 3[69020] -> 2[69020] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3378:3680 [5] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3378:3680 [5] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3378:3680 [5] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3378:3680 [5] NCCL INFO comm 0x7f5eb44d56c0 rank 3 nranks 4 cudaDev 5 busId 69020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3377:3683 [4] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3377:3683 [4] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3377:3683 [4] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3377:3683 [4] NCCL INFO comm 0x7fd42c4d5610 rank 3 nranks 4 cudaDev 4 busId 69010 - Init COMPLETE NCCL version 2.10.3+cuda11.4 NCCL version 2.10.3+cuda11.4 NCCL version 2.10.3+cuda11.4 NCCL version 2.10.3+cuda11.4 NCCL version 2.10.3+cuda11.4 NCCL version 2.10.3+cuda11.4 NCCL version 2.10.3+cuda11.4 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 00/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 01/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 02/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 03/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 04/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 05/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 06/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 07/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 08/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 09/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 10/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 11/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 12/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 13/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 14/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 15/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 16/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 17/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 18/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 19/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 20/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 21/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 22/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 23/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 24/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 25/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 26/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 27/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 28/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 29/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 30/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Channel 31/32 : 0 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 00/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 01/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 02/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 03/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 04/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 05/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 06/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 07/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 08/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 09/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 10/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 11/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 12/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 13/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 14/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 15/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 16/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 17/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 18/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 19/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 20/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 21/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 22/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 23/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 24/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 25/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 26/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 27/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 28/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 29/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 30/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Channel 31/32 : 0 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1 iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 00/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 01/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 02/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 03/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 04/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 05/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 06/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 07/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 08/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 09/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 10/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 11/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 12/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 13/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 14/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 15/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 16/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 17/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 18/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 19/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 20/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 21/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 22/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 23/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 24/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 25/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 26/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 27/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 28/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 29/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 30/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Channel 31/32 : 0 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1 iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 00/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 01/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 02/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 03/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 04/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 05/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 06/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 07/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 08/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 09/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 10/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 11/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 12/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 13/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 14/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 15/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 16/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 17/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 18/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 19/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 20/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 21/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 22/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 23/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 24/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 25/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 26/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 27/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 28/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 29/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 30/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Channel 31/32 : 0 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1 iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 00/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 01/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 02/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 03/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 04/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 05/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 06/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 07/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 08/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 09/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 10/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 11/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 12/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 13/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 14/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 15/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 16/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 17/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 18/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 19/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 20/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 21/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 22/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 23/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 24/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 25/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 26/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 27/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 28/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 29/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 30/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Channel 31/32 : 0 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1 iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 00/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 01/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 02/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 03/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 04/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 05/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 06/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 07/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 08/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 09/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 10/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 11/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 12/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 13/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 14/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 15/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 16/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 17/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 18/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 19/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 20/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 21/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 22/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 23/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 24/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 25/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 26/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 27/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 28/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 29/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 30/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Channel 31/32 : 0 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1 iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 00/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 01/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 02/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 03/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 04/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 05/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 06/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 07/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 08/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 09/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 10/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 11/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 12/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 13/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 14/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 15/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 16/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 17/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 18/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 19/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 20/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 21/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 22/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 23/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 24/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 25/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 26/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 27/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 28/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 29/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 30/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Channel 31/32 : 0 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1 iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 00/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 01/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 02/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 03/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 04/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 05/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 06/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 07/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 08/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 09/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 10/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 11/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 12/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 13/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 14/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 15/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 16/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 17/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 18/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 19/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 20/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 21/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 22/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 23/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 24/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 25/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 26/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 27/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 28/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 29/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 30/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Channel 31/32 : 0 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1 iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3376:3705 [3] NCCL INFO comm 0x7fc5fc008fb0 rank 0 nranks 1 cudaDev 3 busId 67020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3373:3704 [0] NCCL INFO comm 0x7f4bbc008fb0 rank 0 nranks 1 cudaDev 0 busId 65010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3377:3702 [4] NCCL INFO comm 0x7fd3ac008fb0 rank 0 nranks 1 cudaDev 4 busId 69010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3379:3710 [6] NCCL INFO comm 0x7fadf0008fb0 rank 0 nranks 1 cudaDev 6 busId 6b010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3374:3713 [1] NCCL INFO comm 0x7f7e48008fb0 rank 0 nranks 1 cudaDev 1 busId 65020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3378:3714 [5] NCCL INFO comm 0x7f5e38008fb0 rank 0 nranks 1 cudaDev 5 busId 69020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3380:3721 [7] NCCL INFO comm 0x7f4ffc008fb0 rank 0 nranks 1 cudaDev 7 busId 6b020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3375:3720 [2] NCCL INFO comm 0x7f1bd8008fb0 rank 0 nranks 1 cudaDev 2 busId 67010 - Init COMPLETE [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:99] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) time (ms) | model-and-optimizer-setup: 603.25 | train/valid/test-data-iterators-setup: 2641.91 iv-2udaavw4l02thdv8lcrl:3375:5966 [2] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-2udaavw4l02thdv8lcrl:3375:5966 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3375:5966 [2] NCCL INFO Channel 00 : 0[67010] -> 1[67010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:5966 [2] NCCL INFO Channel 01 : 0[67010] -> 1[67010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:5966 [2] NCCL INFO Channel 00 : 1[67010] -> 0[67010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:5968 [7] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-2udaavw4l02thdv8lcrl:3380:5968 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3378:5970 [5] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-2udaavw4l02thdv8lcrl:3378:5970 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3375:5966 [2] NCCL INFO Channel 01 : 1[67010] -> 0[67010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3379:5980 [6] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-2udaavw4l02thdv8lcrl:3378:5970 [5] NCCL INFO Channel 00 : 0[69020] -> 1[69020] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3379:5980 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3374:5972 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-2udaavw4l02thdv8lcrl:3374:5972 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3378:5970 [5] NCCL INFO Channel 01 : 0[69020] -> 1[69020] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3378:5970 [5] NCCL INFO Channel 00 : 1[69020] -> 0[69020] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3376:5973 [3] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-2udaavw4l02thdv8lcrl:3376:5973 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3378:5970 [5] NCCL INFO Channel 01 : 1[69020] -> 0[69020] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3377:5976 [4] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-2udaavw4l02thdv8lcrl:3377:5976 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3373:5978 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 iv-2udaavw4l02thdv8lcrl:3373:5978 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3377:5976 [4] NCCL INFO Channel 00 : 0[69010] -> 1[69010] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3377:5976 [4] NCCL INFO Channel 01 : 0[69010] -> 1[69010] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3380:5968 [7] NCCL INFO Channel 00 : 0[6b020] -> 1[6b020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3377:5976 [4] NCCL INFO Channel 00 : 1[69010] -> 0[69010] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3377:5976 [4] NCCL INFO Channel 01 : 1[69010] -> 0[69010] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3374:5972 [1] NCCL INFO Channel 00 : 0[65020] -> 1[65020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3379:5980 [6] NCCL INFO Channel 00 : 0[6b010] -> 1[6b010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3376:5973 [3] NCCL INFO Channel 00 : 0[67020] -> 1[67020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3379:5980 [6] NCCL INFO Channel 01 : 0[6b010] -> 1[6b010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:5968 [7] NCCL INFO Channel 01 : 0[6b020] -> 1[6b020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:5972 [1] NCCL INFO Channel 01 : 0[65020] -> 1[65020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3376:5973 [3] NCCL INFO Channel 01 : 0[67020] -> 1[67020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:5966 [2] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3375:5966 [2] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3375:5966 [2] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3375:5966 [2] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3375:5966 [2] NCCL INFO comm 0x7f1b98008fb0 rank 1 nranks 2 cudaDev 2 busId 67010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3373:5978 [0] NCCL INFO Channel 00 : 0[65010] -> 1[65010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3379:5980 [6] NCCL INFO Channel 00 : 1[6b010] -> 0[6b010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:5968 [7] NCCL INFO Channel 00 : 1[6b020] -> 0[6b020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:5972 [1] NCCL INFO Channel 00 : 1[65020] -> 0[65020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3376:5973 [3] NCCL INFO Channel 00 : 1[67020] -> 0[67020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:5978 [0] NCCL INFO Channel 01 : 0[65010] -> 1[65010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3379:5980 [6] NCCL INFO Channel 01 : 1[6b010] -> 0[6b010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:5968 [7] NCCL INFO Channel 01 : 1[6b020] -> 0[6b020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:5972 [1] NCCL INFO Channel 01 : 1[65020] -> 0[65020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3376:5973 [3] NCCL INFO Channel 01 : 1[67020] -> 0[67020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3379:5980 [6] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3379:5980 [6] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3379:5980 [6] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3379:5980 [6] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3379:5980 [6] NCCL INFO comm 0x7fadb4008fb0 rank 1 nranks 2 cudaDev 6 busId 6b010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3374:5972 [1] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3374:5972 [1] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3374:5972 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3374:5972 [1] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3374:5972 [1] NCCL INFO comm 0x7f7e08008fb0 rank 1 nranks 2 cudaDev 1 busId 65020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3380:5968 [7] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3380:5968 [7] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3380:5968 [7] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3380:5968 [7] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3380:5968 [7] NCCL INFO comm 0x7f4fc8008fb0 rank 1 nranks 2 cudaDev 7 busId 6b020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3373:5978 [0] NCCL INFO Channel 00 : 1[65010] -> 0[65010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3376:5973 [3] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3376:5973 [3] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3376:5973 [3] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3376:5973 [3] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3376:5973 [3] NCCL INFO comm 0x7fc5c0008fb0 rank 1 nranks 2 cudaDev 3 busId 67020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3373:5978 [0] NCCL INFO Channel 01 : 1[65010] -> 0[65010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:5978 [0] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3373:5978 [0] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3373:5978 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3373:5978 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3373:5978 [0] NCCL INFO comm 0x7f4b7c008fb0 rank 1 nranks 2 cudaDev 0 busId 65010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3378:5970 [5] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3378:5970 [5] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3378:5970 [5] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3378:5970 [5] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3378:5970 [5] NCCL INFO comm 0x7f5df8008fb0 rank 1 nranks 2 cudaDev 5 busId 69020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3377:5976 [4] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3377:5976 [4] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3377:5976 [4] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3377:5976 [4] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3377:5976 [4] NCCL INFO comm 0x7fd370008fb0 rank 1 nranks 2 cudaDev 4 busId 69010 - Init COMPLETE /dataset/xyn/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.) output = bias_dropout_add_func( /dataset/xyn/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.) output = bias_dropout_add_func( /dataset/xyn/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.) output = bias_dropout_add_func( /dataset/xyn/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.) output = bias_dropout_add_func( /dataset/xyn/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.) output = bias_dropout_add_func( /dataset/xyn/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.) output = bias_dropout_add_func( /dataset/xyn/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.) output = bias_dropout_add_func( /dataset/xyn/Megatron-LM/megatron/model/transformer.py:536: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/LegacyTypeDispatch.h:74.) output = bias_dropout_add_func( iv-2udaavw4l02thdv8lcrl:3378:6094 [5] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 1/-1/-1->3->-1 iv-2udaavw4l02thdv8lcrl:3378:6094 [5] NCCL INFO Setting affinity for GPU 5 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3378:6094 [5] NCCL INFO Channel 00 : 2[69020] -> 3[69020] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3378:6094 [5] NCCL INFO Channel 01 : 2[69020] -> 3[69020] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3373:6098 [0] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 1/-1/-1->3->-1 iv-2udaavw4l02thdv8lcrl:3373:6098 [0] NCCL INFO Setting affinity for GPU 0 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3378:6094 [5] NCCL INFO Channel 00 : 3[69020] -> 0[69020] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3374:6095 [1] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 1/-1/-1->3->-1 iv-2udaavw4l02thdv8lcrl:3374:6095 [1] NCCL INFO Setting affinity for GPU 1 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3378:6094 [5] NCCL INFO Channel 01 : 3[69020] -> 0[69020] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3380:6097 [7] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 1/-1/-1->3->-1 iv-2udaavw4l02thdv8lcrl:3380:6097 [7] NCCL INFO Setting affinity for GPU 7 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3377:6093 [4] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 1/-1/-1->3->-1 iv-2udaavw4l02thdv8lcrl:3377:6093 [4] NCCL INFO Setting affinity for GPU 4 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3379:6100 [6] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 1/-1/-1->3->-1 iv-2udaavw4l02thdv8lcrl:3379:6100 [6] NCCL INFO Setting affinity for GPU 6 to 0fffff,fffffc00,00000000 iv-2udaavw4l02thdv8lcrl:3375:6099 [2] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 1/-1/-1->3->-1 iv-2udaavw4l02thdv8lcrl:3375:6099 [2] NCCL INFO Setting affinity for GPU 2 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3377:6093 [4] NCCL INFO Channel 00 : 2[69010] -> 3[69010] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3376:6096 [3] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 1/-1/-1->3->-1 iv-2udaavw4l02thdv8lcrl:3376:6096 [3] NCCL INFO Setting affinity for GPU 3 to 03ff,ffffffff iv-2udaavw4l02thdv8lcrl:3377:6093 [4] NCCL INFO Channel 01 : 2[69010] -> 3[69010] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3374:6095 [1] NCCL INFO Channel 00 : 2[65020] -> 3[65020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3377:6093 [4] NCCL INFO Channel 00 : 3[69010] -> 0[69010] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3379:6100 [6] NCCL INFO Channel 00 : 2[6b010] -> 3[6b010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:6097 [7] NCCL INFO Channel 00 : 2[6b020] -> 3[6b020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:6098 [0] NCCL INFO Channel 00 : 2[65010] -> 3[65010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:6099 [2] NCCL INFO Channel 00 : 2[67010] -> 3[67010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:6095 [1] NCCL INFO Channel 01 : 2[65020] -> 3[65020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3379:6100 [6] NCCL INFO Channel 01 : 2[6b010] -> 3[6b010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:6097 [7] NCCL INFO Channel 01 : 2[6b020] -> 3[6b020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3377:6093 [4] NCCL INFO Channel 01 : 3[69010] -> 0[69010] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3376:6096 [3] NCCL INFO Channel 00 : 2[67020] -> 3[67020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:6095 [1] NCCL INFO Channel 00 : 3[65020] -> 0[65020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:6099 [2] NCCL INFO Channel 01 : 2[67010] -> 3[67010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:6098 [0] NCCL INFO Channel 01 : 2[65010] -> 3[65010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3379:6100 [6] NCCL INFO Channel 00 : 3[6b010] -> 0[6b010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:6097 [7] NCCL INFO Channel 00 : 3[6b020] -> 0[6b020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3376:6096 [3] NCCL INFO Channel 01 : 2[67020] -> 3[67020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:6095 [1] NCCL INFO Channel 01 : 3[65020] -> 0[65020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:6099 [2] NCCL INFO Channel 00 : 3[67010] -> 0[67010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:6098 [0] NCCL INFO Channel 00 : 3[65010] -> 0[65010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:6095 [1] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3379:6100 [6] NCCL INFO Channel 01 : 3[6b010] -> 0[6b010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:6097 [7] NCCL INFO Channel 01 : 3[6b020] -> 0[6b020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3376:6096 [3] NCCL INFO Channel 00 : 3[67020] -> 0[67020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:6098 [0] NCCL INFO Channel 01 : 3[65010] -> 0[65010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:6099 [2] NCCL INFO Channel 01 : 3[67010] -> 0[67010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:6095 [1] NCCL INFO Channel 01 : 1[65020] -> 3[65020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3379:6100 [6] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3380:6097 [7] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3375:6099 [2] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3373:6098 [0] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3376:6096 [3] NCCL INFO Channel 01 : 3[67020] -> 0[67020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3376:6096 [3] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3379:6100 [6] NCCL INFO Channel 01 : 1[6b010] -> 3[6b010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:6095 [1] NCCL INFO Channel 01 : 3[65020] -> 1[65020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:6097 [7] NCCL INFO Channel 01 : 1[6b020] -> 3[6b020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:6099 [2] NCCL INFO Channel 01 : 1[67010] -> 3[67010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:6098 [0] NCCL INFO Channel 01 : 1[65010] -> 3[65010] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3379:6100 [6] NCCL INFO Channel 01 : 3[6b010] -> 1[6b010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3376:6096 [3] NCCL INFO Channel 01 : 1[67020] -> 3[67020] [receive] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:6095 [1] NCCL INFO Channel 00 : 3[65020] -> 2[65020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:6097 [7] NCCL INFO Channel 01 : 3[6b020] -> 1[6b020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:6099 [2] NCCL INFO Channel 01 : 3[67010] -> 1[67010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:6098 [0] NCCL INFO Channel 01 : 3[65010] -> 1[65010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3374:6095 [1] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3374:6095 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3374:6095 [1] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3374:6095 [1] NCCL INFO comm 0x7f7a8c008fb0 rank 3 nranks 4 cudaDev 1 busId 65020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3376:6096 [3] NCCL INFO Channel 01 : 3[67020] -> 1[67020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3378:6094 [5] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3379:6100 [6] NCCL INFO Channel 00 : 3[6b010] -> 2[6b010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:6097 [7] NCCL INFO Channel 00 : 3[6b020] -> 2[6b020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3375:6099 [2] NCCL INFO Channel 00 : 3[67010] -> 2[67010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3373:6098 [0] NCCL INFO Channel 00 : 3[65010] -> 2[65010] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3378:6094 [5] NCCL INFO Channel 01 : 1[69020] -> 3[69020] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3378:6094 [5] NCCL INFO Channel 01 : 3[69020] -> 1[69020] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3376:6096 [3] NCCL INFO Channel 00 : 3[67020] -> 2[67020] [send] via NET/IBext/0 iv-2udaavw4l02thdv8lcrl:3380:6097 [7] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3380:6097 [7] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3380:6097 [7] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3380:6097 [7] NCCL INFO comm 0x7f4c44008fb0 rank 3 nranks 4 cudaDev 7 busId 6b020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3373:6098 [0] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3373:6098 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3373:6098 [0] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3373:6098 [0] NCCL INFO comm 0x7f47f8008fb0 rank 3 nranks 4 cudaDev 0 busId 65010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3375:6099 [2] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3375:6099 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3375:6099 [2] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3375:6099 [2] NCCL INFO comm 0x7f1818008fb0 rank 3 nranks 4 cudaDev 2 busId 67010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3379:6100 [6] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3379:6100 [6] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3379:6100 [6] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3379:6100 [6] NCCL INFO comm 0x7faa3c008fb0 rank 3 nranks 4 cudaDev 6 busId 6b010 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3377:6093 [4] NCCL INFO Connected all rings iv-2udaavw4l02thdv8lcrl:3376:6096 [3] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3376:6096 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3376:6096 [3] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3376:6096 [3] NCCL INFO comm 0x7fc244008fb0 rank 3 nranks 4 cudaDev 3 busId 67020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3377:6093 [4] NCCL INFO Channel 01 : 1[69010] -> 3[69010] [receive] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3377:6093 [4] NCCL INFO Channel 01 : 3[69010] -> 1[69010] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3378:6094 [5] NCCL INFO Channel 00 : 3[69020] -> 2[69020] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3377:6093 [4] NCCL INFO Channel 00 : 3[69010] -> 2[69010] [send] via NET/IBext/0/GDRDMA iv-2udaavw4l02thdv8lcrl:3378:6094 [5] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3378:6094 [5] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3378:6094 [5] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3378:6094 [5] NCCL INFO comm 0x7f5a78008fb0 rank 3 nranks 4 cudaDev 5 busId 69020 - Init COMPLETE iv-2udaavw4l02thdv8lcrl:3377:6093 [4] NCCL INFO Connected all trees iv-2udaavw4l02thdv8lcrl:3377:6093 [4] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512 iv-2udaavw4l02thdv8lcrl:3377:6093 [4] NCCL INFO 2 coll channels, 2 p2p channels, 1 p2p channels per peer iv-2udaavw4l02thdv8lcrl:3377:6093 [4] NCCL INFO comm 0x7fcff0008fb0 rank 3 nranks 4 cudaDev 4 busId 69010 - Init COMPLETE iteration 100/ 220 | consumed samples: 819200 | elapsed time per iteration (ms): 12700.1 | learning rate: 8.586E-07 | tpt: 645.0 samples/s | global batch size: 8192 | lm loss: 9.575354E+00 | sop loss: 7.024770E-01 | loss scale: 262144.0 | grad norm: 2.028 | number of skipped iterations: 15 | number of nan iterations: 0 | [Rank 24] (after 100 iterations) memory (MB) | allocated: 2156.884765625 | max allocated: 11193.2265625 | reserved: 14432.0 | max reserved: 14432.0 time (ms) | forward-compute: 2869.01 | forward-recv: 991.36 | backward-compute: 6081.93 | backward-send: 67.48 | backward-send-forward-recv: 531.26 | backward-params-all-reduce: 11.38 | backward-embedding-all-reduce: 2124.78 | optimizer-copy-to-main-grad: 2.36 | optimizer-unscale-and-check-inf: 9.43 | optimizer-clip-main-grad: 1.97 | optimizer-copy-main-to-model-params: 1.19 | optimizer: 18.57 | batch-generator: 18.34 timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/07/06 15:46:19.250, Tesla V100-SXM2-32GB, 470.57.02, 77 %, 5 %, 32510 MiB, 16452 MiB, 16058 MiB 2022/07/06 15:46:19.253, Tesla V100-SXM2-32GB, 470.57.02, 67 %, 5 %, 32510 MiB, 16466 MiB, 16044 MiB 2022/07/06 15:46:19.255, Tesla V100-SXM2-32GB, 470.57.02, 37 %, 5 %, 32510 MiB, 16612 MiB, 15898 MiB 2022/07/06 15:46:19.256, Tesla V100-SXM2-32GB, 470.57.02, 36 %, 5 %, 32510 MiB, 16560 MiB, 15950 MiB 2022/07/06 15:46:19.257, Tesla V100-SXM2-32GB, 470.57.02, 98 %, 2 %, 32510 MiB, 16318 MiB, 16192 MiB 2022/07/06 15:46:19.258, Tesla V100-SXM2-32GB, 470.57.02, 60 %, 6 %, 32510 MiB, 16280 MiB, 16230 MiB timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/07/06 15:46:19.259, Tesla V100-SXM2-32GB, 470.57.02, 98 %, 2 %, 32510 MiB, 16432 MiB, 16078 MiB 2022/07/06 15:46:19.261, Tesla V100-SXM2-32GB, 470.57.02, 77 %, 5 %, 32510 MiB, 16452 MiB, 16058 MiB 2022/07/06 15:46:19.262, Tesla V100-SXM2-32GB, 470.57.02, 77 %, 5 %, 32510 MiB, 16452 MiB, 16058 MiB 2022/07/06 15:46:19.263, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 5 %, 32510 MiB, 16488 MiB, 16022 MiB 2022/07/06 15:46:19.265, Tesla V100-SXM2-32GB, 470.57.02, 67 %, 5 %, 32510 MiB, 16466 MiB, 16044 MiB timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/07/06 15:46:19.266, Tesla V100-SXM2-32GB, 470.57.02, 67 %, 5 %, 32510 MiB, 16466 MiB, 16044 MiB 2022/07/06 15:46:19.267, Tesla V100-SXM2-32GB, 470.57.02, 37 %, 5 %, 32510 MiB, 16612 MiB, 15898 MiB timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/07/06 15:46:19.267, Tesla V100-SXM2-32GB, 470.57.02, 77 %, 5 %, 32510 MiB, 16452 MiB, 16058 MiB 2022/07/06 15:46:19.268, Tesla V100-SXM2-32GB, 470.57.02, 37 %, 5 %, 32510 MiB, 16612 MiB, 15898 MiB timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/07/06 15:46:19.269, Tesla V100-SXM2-32GB, 470.57.02, 36 %, 5 %, 32510 MiB, 16560 MiB, 15950 MiB 2022/07/06 15:46:19.270, Tesla V100-SXM2-32GB, 470.57.02, 77 %, 5 %, 32510 MiB, 16452 MiB, 16058 MiB 2022/07/06 15:46:19.271, Tesla V100-SXM2-32GB, 470.57.02, 67 %, 5 %, 32510 MiB, 16466 MiB, 16044 MiB 2022/07/06 15:46:19.271, Tesla V100-SXM2-32GB, 470.57.02, 36 %, 5 %, 32510 MiB, 16560 MiB, 15950 MiB 2022/07/06 15:46:19.272, Tesla V100-SXM2-32GB, 470.57.02, 77 %, 5 %, 32510 MiB, 16452 MiB, 16058 MiB 2022/07/06 15:46:19.274, Tesla V100-SXM2-32GB, 470.57.02, 98 %, 2 %, 32510 MiB, 16318 MiB, 16192 MiB 2022/07/06 15:46:19.275, Tesla V100-SXM2-32GB, 470.57.02, 67 %, 5 %, 32510 MiB, 16466 MiB, 16044 MiB 2022/07/06 15:46:19.276, Tesla V100-SXM2-32GB, 470.57.02, 37 %, 5 %, 32510 MiB, 16612 MiB, 15898 MiB 2022/07/06 15:46:19.276, Tesla V100-SXM2-32GB, 470.57.02, 98 %, 2 %, 32510 MiB, 16318 MiB, 16192 MiB timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/07/06 15:46:19.277, Tesla V100-SXM2-32GB, 470.57.02, 67 %, 5 %, 32510 MiB, 16466 MiB, 16044 MiB timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/07/06 15:46:19.279, Tesla V100-SXM2-32GB, 470.57.02, 60 %, 6 %, 32510 MiB, 16280 MiB, 16230 MiB 2022/07/06 15:46:19.279, Tesla V100-SXM2-32GB, 470.57.02, 37 %, 5 %, 32510 MiB, 16612 MiB, 15898 MiB 2022/07/06 15:46:19.282, Tesla V100-SXM2-32GB, 470.57.02, 36 %, 5 %, 32510 MiB, 16560 MiB, 15950 MiB 2022/07/06 15:46:19.283, Tesla V100-SXM2-32GB, 470.57.02, 60 %, 6 %, 32510 MiB, 16280 MiB, 16230 MiB 2022/07/06 15:46:19.283, Tesla V100-SXM2-32GB, 470.57.02, 77 %, 5 %, 32510 MiB, 16452 MiB, 16058 MiB 2022/07/06 15:46:19.284, Tesla V100-SXM2-32GB, 470.57.02, 37 %, 5 %, 32510 MiB, 16612 MiB, 15898 MiB 2022/07/06 15:46:19.285, Tesla V100-SXM2-32GB, 470.57.02, 77 %, 5 %, 32510 MiB, 16452 MiB, 16058 MiB 2022/07/06 15:46:19.285, Tesla V100-SXM2-32GB, 470.57.02, 98 %, 2 %, 32510 MiB, 16432 MiB, 16078 MiB 2022/07/06 15:46:19.286, Tesla V100-SXM2-32GB, 470.57.02, 36 %, 5 %, 32510 MiB, 16560 MiB, 15950 MiB 2022/07/06 15:46:19.287, Tesla V100-SXM2-32GB, 470.57.02, 98 %, 2 %, 32510 MiB, 16318 MiB, 16192 MiB 2022/07/06 15:46:19.287, Tesla V100-SXM2-32GB, 470.57.02, 98 %, 2 %, 32510 MiB, 16432 MiB, 16078 MiB 2022/07/06 15:46:19.288, Tesla V100-SXM2-32GB, 470.57.02, 67 %, 5 %, 32510 MiB, 16466 MiB, 16044 MiB 2022/07/06 15:46:19.289, Tesla V100-SXM2-32GB, 470.57.02, 36 %, 5 %, 32510 MiB, 16560 MiB, 15950 MiB 2022/07/06 15:46:19.290, Tesla V100-SXM2-32GB, 470.57.02, 67 %, 5 %, 32510 MiB, 16466 MiB, 16044 MiB 2022/07/06 15:46:19.291, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 5 %, 32510 MiB, 16488 MiB, 16022 MiB 2022/07/06 15:46:19.292, Tesla V100-SXM2-32GB, 470.57.02, 98 %, 2 %, 32510 MiB, 16318 MiB, 16192 MiB 2022/07/06 15:46:19.292, Tesla V100-SXM2-32GB, 470.57.02, 60 %, 6 %, 32510 MiB, 16280 MiB, 16230 MiB 2022/07/06 15:46:19.293, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 5 %, 32510 MiB, 16488 MiB, 16022 MiB 2022/07/06 15:46:19.294, Tesla V100-SXM2-32GB, 470.57.02, 37 %, 5 %, 32510 MiB, 16612 MiB, 15898 MiB 2022/07/06 15:46:19.294, Tesla V100-SXM2-32GB, 470.57.02, 98 %, 2 %, 32510 MiB, 16318 MiB, 16192 MiB 2022/07/06 15:46:19.296, Tesla V100-SXM2-32GB, 470.57.02, 37 %, 5 %, 32510 MiB, 16612 MiB, 15898 MiB 2022/07/06 15:46:19.298, Tesla V100-SXM2-32GB, 470.57.02, 60 %, 6 %, 32510 MiB, 16280 MiB, 16230 MiB 2022/07/06 15:46:19.298, Tesla V100-SXM2-32GB, 470.57.02, 98 %, 2 %, 32510 MiB, 16432 MiB, 16078 MiB 2022/07/06 15:46:19.300, Tesla V100-SXM2-32GB, 470.57.02, 36 %, 5 %, 32510 MiB, 16560 MiB, 15950 MiB 2022/07/06 15:46:19.300, Tesla V100-SXM2-32GB, 470.57.02, 60 %, 6 %, 32510 MiB, 16280 MiB, 16230 MiB 2022/07/06 15:46:19.301, Tesla V100-SXM2-32GB, 470.57.02, 36 %, 5 %, 32510 MiB, 16560 MiB, 15950 MiB 2022/07/06 15:46:19.303, Tesla V100-SXM2-32GB, 470.57.02, 98 %, 2 %, 32510 MiB, 16432 MiB, 16078 MiB 2022/07/06 15:46:19.303, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 5 %, 32510 MiB, 16488 MiB, 16022 MiB 2022/07/06 15:46:19.304, Tesla V100-SXM2-32GB, 470.57.02, 98 %, 2 %, 32510 MiB, 16318 MiB, 16192 MiB 2022/07/06 15:46:19.305, Tesla V100-SXM2-32GB, 470.57.02, 98 %, 2 %, 32510 MiB, 16432 MiB, 16078 MiB 2022/07/06 15:46:19.306, Tesla V100-SXM2-32GB, 470.57.02, 4 %, 4 %, 32510 MiB, 16318 MiB, 16192 MiB 2022/07/06 15:46:19.307, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 5 %, 32510 MiB, 16488 MiB, 16022 MiB 2022/07/06 15:46:19.309, Tesla V100-SXM2-32GB, 470.57.02, 60 %, 6 %, 32510 MiB, 16280 MiB, 16230 MiB 2022/07/06 15:46:19.309, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 5 %, 32510 MiB, 16488 MiB, 16022 MiB 2022/07/06 15:46:19.311, Tesla V100-SXM2-32GB, 470.57.02, 60 %, 6 %, 32510 MiB, 16280 MiB, 16230 MiB 2022/07/06 15:46:19.313, Tesla V100-SXM2-32GB, 470.57.02, 3 %, 3 %, 32510 MiB, 16432 MiB, 16078 MiB 2022/07/06 15:46:19.316, Tesla V100-SXM2-32GB, 470.57.02, 3 %, 3 %, 32510 MiB, 16432 MiB, 16078 MiB 2022/07/06 15:46:19.319, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 5 %, 32510 MiB, 16488 MiB, 16022 MiB 2022/07/06 15:46:19.320, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 5 %, 32510 MiB, 16488 MiB, 16022 MiB iteration 200/ 220 | consumed samples: 1638400 | elapsed time per iteration (ms): 12589.7 | learning rate: 1.869E-06 | tpt: 650.7 samples/s | global batch size: 8192 | lm loss: 8.916858E+00 | sop loss: 6.931251E-01 | loss scale: 262144.0 | grad norm: 2.799 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward-compute: 2848.19 | forward-recv: 914.55 | backward-compute: 6078.84 | backward-send: 67.23 | backward-send-forward-recv: 525.38 | backward-params-all-reduce: 11.27 | backward-embedding-all-reduce: 2126.13 | optimizer-copy-to-main-grad: 2.40 | optimizer-unscale-and-check-inf: 1.77 | optimizer-clip-main-grad: 2.35 | optimizer-copy-main-to-model-params: 1.38 | optimizer: 12.08 | batch-generator: 17.99 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ validation loss at the end of training for val data | lm loss value: 8.636298E+00 | lm loss PPL: 5.632441E+03 | sop loss value: 6.921773E-01 | sop loss PPL: 1.998061E+00 | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- validation loss at the end of training for test data | lm loss value: 8.595807E+00 | lm loss PPL: 5.408933E+03 | sop loss value: 6.932754E-01 | sop loss PPL: 2.000256E+00 | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- INFO:torch.distributed.elastic.agent.server.api:[default] worker group successfully finished. Waiting 300 seconds for other agents to finish. INFO:torch.distributed.elastic.agent.server.api:Local worker group finished (SUCCEEDED). Waiting 300 seconds for other agents to finish /opt/conda/lib/python3.8/site-packages/torch/distributed/elastic/utils/store.py:70: FutureWarning: This is an experimental API and will be changed in future. warnings.warn( INFO:torch.distributed.elastic.agent.server.api:Done waiting for other agents. Elapsed: 0.11380386352539062 seconds {"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 24, "group_rank": 3, "worker_id": "3373", "role": "default", "hostname": "iv-2udaavw4l02thdv8lcrl", "state": "SUCCEEDED", "total_run_time": 2914, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 4, \"entry_point\": \"python\", \"local_rank\": [0], \"role_rank\": [24], \"role_world_size\": [32]}", "agent_restarts": 0}} {"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 25, "group_rank": 3, "worker_id": "3374", "role": "default", "hostname": "iv-2udaavw4l02thdv8lcrl", "state": "SUCCEEDED", "total_run_time": 2914, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 4, \"entry_point\": \"python\", \"local_rank\": [1], \"role_rank\": [25], \"role_world_size\": [32]}", "agent_restarts": 0}} {"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 26, "group_rank": 3, "worker_id": "3375", "role": "default", "hostname": "iv-2udaavw4l02thdv8lcrl", "state": "SUCCEEDED", "total_run_time": 2914, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 4, \"entry_point\": \"python\", \"local_rank\": [2], \"role_rank\": [26], \"role_world_size\": [32]}", "agent_restarts": 0}} {"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 27, "group_rank": 3, "worker_id": "3376", "role": "default", "hostname": "iv-2udaavw4l02thdv8lcrl", "state": "SUCCEEDED", "total_run_time": 2914, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 4, \"entry_point\": \"python\", \"local_rank\": [3], \"role_rank\": [27], \"role_world_size\": [32]}", "agent_restarts": 0}} {"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 28, "group_rank": 3, "worker_id": "3377", "role": "default", "hostname": "iv-2udaavw4l02thdv8lcrl", "state": "SUCCEEDED", "total_run_time": 2914, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 4, \"entry_point\": \"python\", \"local_rank\": [4], \"role_rank\": [28], \"role_world_size\": [32]}", "agent_restarts": 0}} {"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 29, "group_rank": 3, "worker_id": "3378", "role": "default", "hostname": "iv-2udaavw4l02thdv8lcrl", "state": "SUCCEEDED", "total_run_time": 2914, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 4, \"entry_point\": \"python\", \"local_rank\": [5], \"role_rank\": [29], \"role_world_size\": [32]}", "agent_restarts": 0}} {"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 30, "group_rank": 3, "worker_id": "3379", "role": "default", "hostname": "iv-2udaavw4l02thdv8lcrl", "state": "SUCCEEDED", "total_run_time": 2914, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 4, \"entry_point\": \"python\", \"local_rank\": [6], \"role_rank\": [30], \"role_world_size\": [32]}", "agent_restarts": 0}} {"name": "torchelastic.worker.status.SUCCEEDED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 31, "group_rank": 3, "worker_id": "3380", "role": "default", "hostname": "iv-2udaavw4l02thdv8lcrl", "state": "SUCCEEDED", "total_run_time": 2914, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 4, \"entry_point\": \"python\", \"local_rank\": [7], \"role_rank\": [31], \"role_world_size\": [32]}", "agent_restarts": 0}} {"name": "torchelastic.worker.status.SUCCEEDED", "source": "AGENT", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": null, "group_rank": 3, "worker_id": null, "role": "default", "hostname": "iv-2udaavw4l02thdv8lcrl", "state": "SUCCEEDED", "total_run_time": 2914, "rdzv_backend": "static", "raw_error": null, "metadata": "{\"group_world_size\": 4, \"entry_point\": \"python\"}", "agent_restarts": 0}} ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. *****************************************