loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
loaded library: loaded library: loaded library: loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1/usr/lib/x86_64-linux-gnu/libibverbs.so.1
loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1/usr/lib/x86_64-linux-gnu/libibverbs.so.1


loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
/usr/lib/x86_64-linux-gnu/libibverbs.so.1
loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
W20220430 02:51:47.348515  3605 rpc_client.cpp:190] LoadServer 10.7.5.234 Failed at 0 times error_code 14 error_message failed to connect to all addresses
W20220430 02:51:47.349668  3604 rpc_client.cpp:190] LoadServer 10.7.5.234 Failed at 0 times error_code 14 error_message failed to connect to all addresses
W20220430 02:51:47.350481  3608 rpc_client.cpp:190] LoadServer 10.7.5.234 Failed at 0 times error_code 14 error_message failed to connect to all addresses
W20220430 02:51:47.349608  3607 rpc_client.cpp:190] LoadServer 10.7.5.234 Failed at 0 times error_code 14 error_message failed to connect to all addresses
------------------------ arguments ------------------------
  batch_size ...................................... 131072
  batch_size_per_proc ............................. 16384
  data_dir ........................................ /dataset/f9f659c5/wdl_ofrecord
  data_part_name_suffix_length .................... 5
  data_part_num ................................... 256
  dataset_format .................................. ofrecord
  ddp ............................................. False
  deep_dropout_rate ............................... 0.5
  deep_embedding_vec_size ......................... 16
  deep_vocab_size ................................. 2322444
  eval_after_training ............................. False
  eval_batchs ..................................... 20
  eval_interval ................................... 0
  execution_mode .................................. graph
  hidden_size ..................................... 1024
  hidden_units_num ................................ 2
  learning_rate ................................... 0.001
  loss_print_every_n_iter ......................... 100
  max_iter ........................................ 1100
  model_load_dir .................................. 
  model_save_dir .................................. ./checkpoint
  num_deep_sparse_fields .......................... 26
  num_dense_fields ................................ 13
  num_wide_sparse_fields .......................... 2
  save_initial_model .............................. False
  save_model_after_each_eval ...................... False
  test_name ....................................... noname_test
  wide_vocab_size ................................. 2322444
-------------------- end of arguments ---------------------
[rank:0] iter: 100/1100, loss: 0.0618603229522705, latency(ms): 196.2159679830074310 | 2022-04-30 02:52:18.621
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/30 02:52:18.750, Tesla V100-SXM2-32GB, 470.57.02, 87 %, 35 %, 32510 MiB, 29709 MiB, 2801 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/30 02:52:18.756, Tesla V100-SXM2-32GB, 470.57.02, 95 %, 37 %, 32510 MiB, 29665 MiB, 2845 MiB
2022/04/30 02:52:18.758, Tesla V100-SXM2-32GB, 470.57.02, 87 %, 35 %, 32510 MiB, 29709 MiB, 2801 MiB
2022/04/30 02:52:18.762, Tesla V100-SXM2-32GB, 470.57.02, 31 %, 14 %, 32510 MiB, 29817 MiB, 2693 MiB
2022/04/30 02:52:18.762, Tesla V100-SXM2-32GB, 470.57.02, 87 %, 35 %, 32510 MiB, 29709 MiB, 2801 MiB
2022/04/30 02:52:18.767, Tesla V100-SXM2-32GB, 470.57.02, 95 %, 37 %, 32510 MiB, 29665 MiB, 2845 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/30 02:52:18.771, Tesla V100-SXM2-32GB, 470.57.02, 59 %, 24 %, 32510 MiB, 29833 MiB, 2677 MiB
2022/04/30 02:52:18.771, Tesla V100-SXM2-32GB, 470.57.02, 95 %, 37 %, 32510 MiB, 29665 MiB, 2845 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/30 02:52:18.775, Tesla V100-SXM2-32GB, 470.57.02, 31 %, 14 %, 32510 MiB, 29817 MiB, 2693 MiB
2022/04/30 02:52:18.775, Tesla V100-SXM2-32GB, 470.57.02, 87 %, 35 %, 32510 MiB, 29709 MiB, 2801 MiB
2022/04/30 02:52:18.779, Tesla V100-SXM2-32GB, 470.57.02, 85 %, 35 %, 32510 MiB, 29849 MiB, 2661 MiB
2022/04/30 02:52:18.778, Tesla V100-SXM2-32GB, 470.57.02, 87 %, 35 %, 32510 MiB, 29709 MiB, 2801 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/30 02:52:18.779, Tesla V100-SXM2-32GB, 470.57.02, 31 %, 14 %, 32510 MiB, 29817 MiB, 2693 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/30 02:52:18.782, Tesla V100-SXM2-32GB, 470.57.02, 87 %, 35 %, 32510 MiB, 29709 MiB, 2801 MiB
2022/04/30 02:52:18.782, Tesla V100-SXM2-32GB, 470.57.02, 59 %, 24 %, 32510 MiB, 29833 MiB, 2677 MiB
2022/04/30 02:52:18.784, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29665 MiB, 2845 MiB
2022/04/30 02:52:18.786, Tesla V100-SXM2-32GB, 470.57.02, 29 %, 11 %, 32510 MiB, 29649 MiB, 2861 MiB
2022/04/30 02:52:18.787, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29665 MiB, 2845 MiB
2022/04/30 02:52:18.787, Tesla V100-SXM2-32GB, 470.57.02, 59 %, 24 %, 32510 MiB, 29833 MiB, 2677 MiB
2022/04/30 02:52:18.787, Tesla V100-SXM2-32GB, 470.57.02, 87 %, 35 %, 32510 MiB, 29709 MiB, 2801 MiB
2022/04/30 02:52:18.788, Tesla V100-SXM2-32GB, 470.57.02, 87 %, 35 %, 32510 MiB, 29709 MiB, 2801 MiB
2022/04/30 02:52:18.791, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29665 MiB, 2845 MiB
2022/04/30 02:52:18.791, Tesla V100-SXM2-32GB, 470.57.02, 85 %, 35 %, 32510 MiB, 29849 MiB, 2661 MiB
2022/04/30 02:52:18.793, Tesla V100-SXM2-32GB, 470.57.02, 31 %, 14 %, 32510 MiB, 29817 MiB, 2693 MiB
2022/04/30 02:52:18.796, Tesla V100-SXM2-32GB, 470.57.02, 77 %, 30 %, 32510 MiB, 29537 MiB, 2973 MiB
2022/04/30 02:52:18.796, Tesla V100-SXM2-32GB, 470.57.02, 31 %, 14 %, 32510 MiB, 29817 MiB, 2693 MiB
2022/04/30 02:52:18.796, Tesla V100-SXM2-32GB, 470.57.02, 85 %, 35 %, 32510 MiB, 29849 MiB, 2661 MiB
2022/04/30 02:52:18.797, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29665 MiB, 2845 MiB
2022/04/30 02:52:18.799, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29665 MiB, 2845 MiB
2022/04/30 02:52:18.800, Tesla V100-SXM2-32GB, 470.57.02, 31 %, 14 %, 32510 MiB, 29817 MiB, 2693 MiB
2022/04/30 02:52:18.801, Tesla V100-SXM2-32GB, 470.57.02, 29 %, 11 %, 32510 MiB, 29649 MiB, 2861 MiB
2022/04/30 02:52:18.802, Tesla V100-SXM2-32GB, 470.57.02, 59 %, 24 %, 32510 MiB, 29833 MiB, 2677 MiB
2022/04/30 02:52:18.806, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29697 MiB, 2813 MiB
2022/04/30 02:52:18.806, Tesla V100-SXM2-32GB, 470.57.02, 59 %, 24 %, 32510 MiB, 29833 MiB, 2677 MiB
2022/04/30 02:52:18.806, Tesla V100-SXM2-32GB, 470.57.02, 29 %, 11 %, 32510 MiB, 29649 MiB, 2861 MiB
2022/04/30 02:52:18.807, Tesla V100-SXM2-32GB, 470.57.02, 31 %, 14 %, 32510 MiB, 29817 MiB, 2693 MiB
2022/04/30 02:52:18.809, Tesla V100-SXM2-32GB, 470.57.02, 31 %, 14 %, 32510 MiB, 29817 MiB, 2693 MiB
2022/04/30 02:52:18.810, Tesla V100-SXM2-32GB, 470.57.02, 59 %, 24 %, 32510 MiB, 29833 MiB, 2677 MiB
2022/04/30 02:52:18.811, Tesla V100-SXM2-32GB, 470.57.02, 77 %, 30 %, 32510 MiB, 29537 MiB, 2973 MiB
2022/04/30 02:52:18.812, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29849 MiB, 2661 MiB
2022/04/30 02:52:18.816, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29849 MiB, 2661 MiB
2022/04/30 02:52:18.816, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29537 MiB, 2973 MiB
2022/04/30 02:52:18.817, Tesla V100-SXM2-32GB, 470.57.02, 59 %, 24 %, 32510 MiB, 29833 MiB, 2677 MiB
2022/04/30 02:52:18.819, Tesla V100-SXM2-32GB, 470.57.02, 59 %, 24 %, 32510 MiB, 29833 MiB, 2677 MiB
2022/04/30 02:52:18.821, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29849 MiB, 2661 MiB
2022/04/30 02:52:18.821, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29697 MiB, 2813 MiB
2022/04/30 02:52:18.822, Tesla V100-SXM2-32GB, 470.57.02, 29 %, 11 %, 32510 MiB, 29649 MiB, 2861 MiB
2022/04/30 02:52:18.826, Tesla V100-SXM2-32GB, 470.57.02, 29 %, 11 %, 32510 MiB, 29649 MiB, 2861 MiB
2022/04/30 02:52:18.826, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29697 MiB, 2813 MiB
2022/04/30 02:52:18.827, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29849 MiB, 2661 MiB
2022/04/30 02:52:18.828, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29849 MiB, 2661 MiB
2022/04/30 02:52:18.830, Tesla V100-SXM2-32GB, 470.57.02, 29 %, 11 %, 32510 MiB, 29649 MiB, 2861 MiB
2022/04/30 02:52:18.832, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29537 MiB, 2973 MiB
2022/04/30 02:52:18.836, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29537 MiB, 2973 MiB
2022/04/30 02:52:18.837, Tesla V100-SXM2-32GB, 470.57.02, 29 %, 11 %, 32510 MiB, 29649 MiB, 2861 MiB
2022/04/30 02:52:18.838, Tesla V100-SXM2-32GB, 470.57.02, 29 %, 11 %, 32510 MiB, 29649 MiB, 2861 MiB
2022/04/30 02:52:18.840, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29537 MiB, 2973 MiB
2022/04/30 02:52:18.842, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29697 MiB, 2813 MiB
2022/04/30 02:52:18.845, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29697 MiB, 2813 MiB
2022/04/30 02:52:18.846, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29537 MiB, 2973 MiB
2022/04/30 02:52:18.847, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29537 MiB, 2973 MiB
2022/04/30 02:52:18.849, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29697 MiB, 2813 MiB
2022/04/30 02:52:18.860, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29697 MiB, 2813 MiB
2022/04/30 02:52:18.861, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29697 MiB, 2813 MiB
[rank:0] iter: 200/1100, loss: 0.0581246316432953, latency(ms): 24.1689876466989517 | 2022-04-30 02:52:21.038
[rank:0] iter: 300/1100, loss: 0.0573087818920612, latency(ms): 21.9243365898728371 | 2022-04-30 02:52:23.231
[rank:0] iter: 400/1100, loss: 0.0562789477407932, latency(ms): 22.2745340317487717 | 2022-04-30 02:52:25.458
[rank:0] iter: 500/1100, loss: 0.0559753030538559, latency(ms): 21.6060904413461685 | 2022-04-30 02:52:27.619
[rank:0] iter: 600/1100, loss: 0.0555363334715366, latency(ms): 21.7894067615270615 | 2022-04-30 02:52:29.798
[rank:0] iter: 700/1100, loss: 0.0546374432742596, latency(ms): 21.4926633983850479 | 2022-04-30 02:52:31.947
[rank:0] iter: 800/1100, loss: 0.0547551438212395, latency(ms): 21.1008892953395844 | 2022-04-30 02:52:34.057
[rank:0] iter: 900/1100, loss: 0.0537434332072735, latency(ms): 20.8582769334316254 | 2022-04-30 02:52:36.143
[rank:0] iter: 1000/1100, loss: 0.0533392429351807, latency(ms): 20.7719608396291733 | 2022-04-30 02:52:38.220
[rank:0] iter: 1100/1100, loss: 0.0533936060965061, latency(ms): 20.8201499655842781 | 2022-04-30 02:52:40.302