loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
loaded library: loaded library: loaded library: loaded library: loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1/usr/lib/x86_64-linux-gnu/libibverbs.so.1/usr/lib/x86_64-linux-gnu/libibverbs.so.1

/usr/lib/x86_64-linux-gnu/libibverbs.so.1/usr/lib/x86_64-linux-gnu/libibverbs.so.1


loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
W20220428 10:54:28.393051  7640 rpc_client.cpp:190] LoadServer 10.7.106.156 Failed at 0 times error_code 14 error_message failed to connect to all addresses
------------------------ arguments ------------------------
  batch_size ...................................... 131072
  batch_size_per_proc ............................. 16384
  data_dir ........................................ /dataset/f9f659c5/wdl_ofrecord
  data_part_name_suffix_length .................... 5
  data_part_num ................................... 256
  dataset_format .................................. ofrecord
  ddp ............................................. True
  deep_dropout_rate ............................... 0.5
  deep_embedding_vec_size ......................... 16
  deep_vocab_size ................................. 2322444
  eval_after_training ............................. False
  eval_batchs ..................................... 20
  eval_interval ................................... 0
  execution_mode .................................. eager
  hidden_size ..................................... 1024
  hidden_units_num ................................ 2
  learning_rate ................................... 0.001
  loss_print_every_n_iter ......................... 100
  max_iter ........................................ 1100
  model_load_dir .................................. 
  model_save_dir .................................. ./checkpoint
  num_deep_sparse_fields .......................... 26
  num_dense_fields ................................ 13
  num_wide_sparse_fields .......................... 2
  save_initial_model .............................. False
  save_model_after_each_eval ...................... False
  test_name ....................................... noname_test
  wide_vocab_size ................................. 2322444
-------------------- end of arguments ---------------------
[rank:0] iter: 100/1100, loss: 0.5039828419685364, latency(ms): 105.8108877390623093 | 2022-04-28 10:54:50.846
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/28 10:54:50.952, Tesla V100-SXM2-32GB, 470.57.02, 23 %, 13 %, 32510 MiB, 29888 MiB, 2622 MiB
2022/04/28 10:54:50.957, Tesla V100-SXM2-32GB, 470.57.02, 23 %, 13 %, 32510 MiB, 29888 MiB, 2622 MiB
2022/04/28 10:54:50.957, Tesla V100-SXM2-32GB, 470.57.02, 48 %, 17 %, 32510 MiB, 29908 MiB, 2602 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/28 10:54:50.961, Tesla V100-SXM2-32GB, 470.57.02, 48 %, 17 %, 32510 MiB, 29908 MiB, 2602 MiB
2022/04/28 10:54:50.962, Tesla V100-SXM2-32GB, 470.57.02, 64 %, 27 %, 32510 MiB, 29984 MiB, 2526 MiB
2022/04/28 10:54:50.962, Tesla V100-SXM2-32GB, 470.57.02, 23 %, 13 %, 32510 MiB, 29888 MiB, 2622 MiB
2022/04/28 10:54:50.968, Tesla V100-SXM2-32GB, 470.57.02, 64 %, 27 %, 32510 MiB, 29984 MiB, 2526 MiB
2022/04/28 10:54:50.968, Tesla V100-SXM2-32GB, 470.57.02, 53 %, 26 %, 32510 MiB, 29960 MiB, 2550 MiB
2022/04/28 10:54:50.970, Tesla V100-SXM2-32GB, 470.57.02, 48 %, 17 %, 32510 MiB, 29908 MiB, 2602 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/28 10:54:50.975, Tesla V100-SXM2-32GB, 470.57.02, 53 %, 26 %, 32510 MiB, 29960 MiB, 2550 MiB
2022/04/28 10:54:50.975, Tesla V100-SXM2-32GB, 470.57.02, 34 %, 9 %, 32510 MiB, 29968 MiB, 2542 MiB
2022/04/28 10:54:50.977, Tesla V100-SXM2-32GB, 470.57.02, 64 %, 27 %, 32510 MiB, 29984 MiB, 2526 MiB
2022/04/28 10:54:50.980, Tesla V100-SXM2-32GB, 470.57.02, 23 %, 13 %, 32510 MiB, 29888 MiB, 2622 MiB
2022/04/28 10:54:50.981, Tesla V100-SXM2-32GB, 470.57.02, 34 %, 9 %, 32510 MiB, 29968 MiB, 2542 MiB
2022/04/28 10:54:50.981, Tesla V100-SXM2-32GB, 470.57.02, 61 %, 25 %, 32510 MiB, 29900 MiB, 2610 MiB
2022/04/28 10:54:50.983, Tesla V100-SXM2-32GB, 470.57.02, 53 %, 26 %, 32510 MiB, 29960 MiB, 2550 MiB
2022/04/28 10:54:50.987, Tesla V100-SXM2-32GB, 470.57.02, 48 %, 17 %, 32510 MiB, 29908 MiB, 2602 MiB
2022/04/28 10:54:50.987, Tesla V100-SXM2-32GB, 470.57.02, 61 %, 25 %, 32510 MiB, 29900 MiB, 2610 MiB
2022/04/28 10:54:50.987, Tesla V100-SXM2-32GB, 470.57.02, 44 %, 14 %, 32510 MiB, 29844 MiB, 2666 MiB
2022/04/28 10:54:50.989, Tesla V100-SXM2-32GB, 470.57.02, 34 %, 9 %, 32510 MiB, 29968 MiB, 2542 MiB
2022/04/28 10:54:50.994, Tesla V100-SXM2-32GB, 470.57.02, 64 %, 27 %, 32510 MiB, 29984 MiB, 2526 MiB
2022/04/28 10:54:50.995, Tesla V100-SXM2-32GB, 470.57.02, 44 %, 14 %, 32510 MiB, 29844 MiB, 2666 MiB
2022/04/28 10:54:50.995, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 15 %, 32510 MiB, 29924 MiB, 2586 MiB
2022/04/28 10:54:50.997, Tesla V100-SXM2-32GB, 470.57.02, 61 %, 25 %, 32510 MiB, 29900 MiB, 2610 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/28 10:54:51.002, Tesla V100-SXM2-32GB, 470.57.02, 53 %, 26 %, 32510 MiB, 29960 MiB, 2550 MiB
2022/04/28 10:54:51.003, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 15 %, 32510 MiB, 29924 MiB, 2586 MiB
2022/04/28 10:54:51.005, Tesla V100-SXM2-32GB, 470.57.02, 44 %, 14 %, 32510 MiB, 29844 MiB, 2666 MiB
2022/04/28 10:54:51.008, Tesla V100-SXM2-32GB, 470.57.02, 23 %, 13 %, 32510 MiB, 29888 MiB, 2622 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/28 10:54:51.010, Tesla V100-SXM2-32GB, 470.57.02, 34 %, 9 %, 32510 MiB, 29968 MiB, 2542 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/28 10:54:51.011, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 15 %, 32510 MiB, 29924 MiB, 2586 MiB
2022/04/28 10:54:51.014, Tesla V100-SXM2-32GB, 470.57.02, 48 %, 17 %, 32510 MiB, 29908 MiB, 2602 MiB
2022/04/28 10:54:51.014, Tesla V100-SXM2-32GB, 470.57.02, 23 %, 13 %, 32510 MiB, 29888 MiB, 2622 MiB
2022/04/28 10:54:51.015, Tesla V100-SXM2-32GB, 470.57.02, 61 %, 25 %, 32510 MiB, 29900 MiB, 2610 MiB
2022/04/28 10:54:51.016, Tesla V100-SXM2-32GB, 470.57.02, 23 %, 13 %, 32510 MiB, 29888 MiB, 2622 MiB
2022/04/28 10:54:51.017, Tesla V100-SXM2-32GB, 470.57.02, 23 %, 13 %, 32510 MiB, 29888 MiB, 2622 MiB
2022/04/28 10:54:51.021, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29984 MiB, 2526 MiB
2022/04/28 10:54:51.022, Tesla V100-SXM2-32GB, 470.57.02, 48 %, 17 %, 32510 MiB, 29908 MiB, 2602 MiB
2022/04/28 10:54:51.023, Tesla V100-SXM2-32GB, 470.57.02, 44 %, 14 %, 32510 MiB, 29844 MiB, 2666 MiB
2022/04/28 10:54:51.024, Tesla V100-SXM2-32GB, 470.57.02, 48 %, 17 %, 32510 MiB, 29908 MiB, 2602 MiB
2022/04/28 10:54:51.025, Tesla V100-SXM2-32GB, 470.57.02, 48 %, 17 %, 32510 MiB, 29908 MiB, 2602 MiB
2022/04/28 10:54:51.028, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29960 MiB, 2550 MiB
2022/04/28 10:54:51.033, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29984 MiB, 2526 MiB
2022/04/28 10:54:51.034, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 15 %, 32510 MiB, 29924 MiB, 2586 MiB
2022/04/28 10:54:51.035, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29984 MiB, 2526 MiB
2022/04/28 10:54:51.036, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29984 MiB, 2526 MiB
2022/04/28 10:54:51.039, Tesla V100-SXM2-32GB, 470.57.02, 34 %, 9 %, 32510 MiB, 29968 MiB, 2542 MiB
2022/04/28 10:54:51.043, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29960 MiB, 2550 MiB
2022/04/28 10:54:51.045, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29960 MiB, 2550 MiB
2022/04/28 10:54:51.046, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29960 MiB, 2550 MiB
2022/04/28 10:54:51.048, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29900 MiB, 2610 MiB
2022/04/28 10:54:51.049, Tesla V100-SXM2-32GB, 470.57.02, 34 %, 9 %, 32510 MiB, 29968 MiB, 2542 MiB
2022/04/28 10:54:51.054, Tesla V100-SXM2-32GB, 470.57.02, 34 %, 9 %, 32510 MiB, 29968 MiB, 2542 MiB
2022/04/28 10:54:51.054, Tesla V100-SXM2-32GB, 470.57.02, 34 %, 9 %, 32510 MiB, 29968 MiB, 2542 MiB
2022/04/28 10:54:51.057, Tesla V100-SXM2-32GB, 470.57.02, 44 %, 14 %, 32510 MiB, 29844 MiB, 2666 MiB
2022/04/28 10:54:51.057, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29900 MiB, 2610 MiB
2022/04/28 10:54:51.058, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29900 MiB, 2610 MiB
2022/04/28 10:54:51.059, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29900 MiB, 2610 MiB
2022/04/28 10:54:51.062, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 15 %, 32510 MiB, 29924 MiB, 2586 MiB
2022/04/28 10:54:51.062, Tesla V100-SXM2-32GB, 470.57.02, 44 %, 14 %, 32510 MiB, 29844 MiB, 2666 MiB
2022/04/28 10:54:51.063, Tesla V100-SXM2-32GB, 470.57.02, 44 %, 14 %, 32510 MiB, 29844 MiB, 2666 MiB
2022/04/28 10:54:51.064, Tesla V100-SXM2-32GB, 470.57.02, 44 %, 14 %, 32510 MiB, 29844 MiB, 2666 MiB
2022/04/28 10:54:51.068, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 15 %, 32510 MiB, 29924 MiB, 2586 MiB
2022/04/28 10:54:51.069, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 15 %, 32510 MiB, 29924 MiB, 2586 MiB
2022/04/28 10:54:51.069, Tesla V100-SXM2-32GB, 470.57.02, 42 %, 15 %, 32510 MiB, 29924 MiB, 2586 MiB
[rank:0] iter: 200/1100, loss: 0.4695327281951904, latency(ms): 42.7517463639378548 | 2022-04-28 10:54:55.121
[rank:0] iter: 300/1100, loss: 0.4607780873775482, latency(ms): 41.2956205755472183 | 2022-04-28 10:54:59.251
[rank:0] iter: 400/1100, loss: 0.4519609808921814, latency(ms): 40.1864702627062798 | 2022-04-28 10:55:03.269
[rank:0] iter: 500/1100, loss: 0.4502073526382446, latency(ms): 41.2315840646624565 | 2022-04-28 10:55:07.392
[rank:0] iter: 600/1100, loss: 0.4464746415615082, latency(ms): 41.2886826321482658 | 2022-04-28 10:55:11.521
[rank:0] iter: 700/1100, loss: 0.4402383565902710, latency(ms): 41.4442216977477074 | 2022-04-28 10:55:15.666
[rank:0] iter: 800/1100, loss: 0.4422902762889862, latency(ms): 42.4232701584696770 | 2022-04-28 10:55:19.908
[rank:0] iter: 900/1100, loss: 0.4328004121780396, latency(ms): 40.2843216061592102 | 2022-04-28 10:55:23.936
[rank:0] iter: 1000/1100, loss: 0.4313513934612274, latency(ms): 41.4135541394352913 | 2022-04-28 10:55:28.078
[rank:0] iter: 1100/1100, loss: 0.4328745603561401, latency(ms): 42.0449165254831314 | 2022-04-28 10:55:32.282