loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
loaded library: loaded library: loaded library: loaded library: loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1/usr/lib/x86_64-linux-gnu/libibverbs.so.1

/usr/lib/x86_64-linux-gnu/libibverbs.so.1/usr/lib/x86_64-linux-gnu/libibverbs.so.1
/usr/lib/x86_64-linux-gnu/libibverbs.so.1

loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
W20220515 02:00:19.783335  6435 rpc_client.cpp:190] LoadServer 10.7.54.16 Failed at 0 times error_code 14 error_message failed to connect to all addresses
W20220515 02:00:19.783334  6436 rpc_client.cpp:190] LoadServer 10.7.54.16 Failed at 0 times error_code 14 error_message failed to connect to all addresses
W20220515 02:00:19.784127  6438 rpc_client.cpp:190] LoadServer 10.7.54.16 Failed at 0 times error_code 14 error_message failed to connect to all addresses
------------------------ arguments ------------------------
  batch_size ...................................... 65536
  batch_size_per_proc ............................. 8192
  data_dir ........................................ /dataset/f9f659c5/wdl_ofrecord
  data_part_name_suffix_length .................... 5
  data_part_num ................................... 256
  dataset_format .................................. ofrecord
  ddp ............................................. True
  deep_dropout_rate ............................... 0.5
  deep_embedding_vec_size ......................... 16
  deep_vocab_size ................................. 2322444
  eval_after_training ............................. False
  eval_batchs ..................................... 20
  eval_interval ................................... 0
  execution_mode .................................. eager
  hidden_size ..................................... 1024
  hidden_units_num ................................ 2
  learning_rate ................................... 0.001
  loss_print_every_n_iter ......................... 100
  max_iter ........................................ 1100
  model_load_dir .................................. 
  model_save_dir .................................. ./checkpoint
  num_deep_sparse_fields .......................... 26
  num_dense_fields ................................ 13
  num_wide_sparse_fields .......................... 2
  save_initial_model .............................. False
  save_model_after_each_eval ...................... False
  test_name ....................................... noname_test
  wide_vocab_size ................................. 2322444
-------------------- end of arguments ---------------------
[rank:0] iter: 100/1100, loss: 0.5047759413719177, latency(ms): 75.8916061744093895 | 2022-05-15 02:00:38.942
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/05/15 02:00:39.157, Tesla V100-SXM2-32GB, 470.57.02, 50 %, 19 %, 32510 MiB, 30118 MiB, 2392 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/05/15 02:00:39.159, Tesla V100-SXM2-32GB, 470.57.02, 50 %, 19 %, 32510 MiB, 30118 MiB, 2392 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/05/15 02:00:39.165, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30138 MiB, 2372 MiB
2022/05/15 02:00:39.167, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30138 MiB, 2372 MiB
2022/05/15 02:00:39.167, Tesla V100-SXM2-32GB, 470.57.02, 50 %, 19 %, 32510 MiB, 30118 MiB, 2392 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/05/15 02:00:39.170, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30118 MiB, 2392 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/05/15 02:00:39.172, Tesla V100-SXM2-32GB, 470.57.02, 46 %, 17 %, 32510 MiB, 30214 MiB, 2296 MiB
2022/05/15 02:00:39.173, Tesla V100-SXM2-32GB, 470.57.02, 46 %, 17 %, 32510 MiB, 30214 MiB, 2296 MiB
2022/05/15 02:00:39.174, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30138 MiB, 2372 MiB
2022/05/15 02:00:39.174, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30118 MiB, 2392 MiB
2022/05/15 02:00:39.175, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30118 MiB, 2392 MiB
2022/05/15 02:00:39.177, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30118 MiB, 2392 MiB
2022/05/15 02:00:39.178, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30138 MiB, 2372 MiB
2022/05/15 02:00:39.178, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30118 MiB, 2392 MiB
2022/05/15 02:00:39.179, Tesla V100-SXM2-32GB, 470.57.02, 18 %, 8 %, 32510 MiB, 30190 MiB, 2320 MiB
2022/05/15 02:00:39.182, Tesla V100-SXM2-32GB, 470.57.02, 18 %, 8 %, 32510 MiB, 30190 MiB, 2320 MiB
2022/05/15 02:00:39.182, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30214 MiB, 2296 MiB
2022/05/15 02:00:39.184, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30138 MiB, 2372 MiB
2022/05/15 02:00:39.185, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30138 MiB, 2372 MiB
2022/05/15 02:00:39.187, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30138 MiB, 2372 MiB
2022/05/15 02:00:39.187, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30214 MiB, 2296 MiB
2022/05/15 02:00:39.189, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30138 MiB, 2372 MiB
2022/05/15 02:00:39.189, Tesla V100-SXM2-32GB, 470.57.02, 24 %, 11 %, 32510 MiB, 30198 MiB, 2312 MiB
2022/05/15 02:00:39.191, Tesla V100-SXM2-32GB, 470.57.02, 24 %, 11 %, 32510 MiB, 30198 MiB, 2312 MiB
2022/05/15 02:00:39.192, Tesla V100-SXM2-32GB, 470.57.02, 18 %, 8 %, 32510 MiB, 30190 MiB, 2320 MiB
2022/05/15 02:00:39.194, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30214 MiB, 2296 MiB
2022/05/15 02:00:39.195, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30214 MiB, 2296 MiB
2022/05/15 02:00:39.197, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30214 MiB, 2296 MiB
2022/05/15 02:00:39.197, Tesla V100-SXM2-32GB, 470.57.02, 18 %, 8 %, 32510 MiB, 30190 MiB, 2320 MiB
2022/05/15 02:00:39.198, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30214 MiB, 2296 MiB
2022/05/15 02:00:39.199, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30130 MiB, 2380 MiB
2022/05/15 02:00:39.201, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30130 MiB, 2380 MiB
2022/05/15 02:00:39.202, Tesla V100-SXM2-32GB, 470.57.02, 24 %, 11 %, 32510 MiB, 30198 MiB, 2312 MiB
2022/05/15 02:00:39.204, Tesla V100-SXM2-32GB, 470.57.02, 18 %, 8 %, 32510 MiB, 30190 MiB, 2320 MiB
2022/05/15 02:00:39.205, Tesla V100-SXM2-32GB, 470.57.02, 18 %, 8 %, 32510 MiB, 30190 MiB, 2320 MiB
2022/05/15 02:00:39.207, Tesla V100-SXM2-32GB, 470.57.02, 18 %, 8 %, 32510 MiB, 30190 MiB, 2320 MiB
2022/05/15 02:00:39.207, Tesla V100-SXM2-32GB, 470.57.02, 24 %, 11 %, 32510 MiB, 30198 MiB, 2312 MiB
2022/05/15 02:00:39.209, Tesla V100-SXM2-32GB, 470.57.02, 18 %, 8 %, 32510 MiB, 30190 MiB, 2320 MiB
2022/05/15 02:00:39.209, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30074 MiB, 2436 MiB
2022/05/15 02:00:39.211, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30074 MiB, 2436 MiB
2022/05/15 02:00:39.212, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30130 MiB, 2380 MiB
2022/05/15 02:00:39.214, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30198 MiB, 2312 MiB
2022/05/15 02:00:39.215, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30198 MiB, 2312 MiB
2022/05/15 02:00:39.217, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30198 MiB, 2312 MiB
2022/05/15 02:00:39.217, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30130 MiB, 2380 MiB
2022/05/15 02:00:39.219, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30198 MiB, 2312 MiB
2022/05/15 02:00:39.219, Tesla V100-SXM2-32GB, 470.57.02, 5 %, 3 %, 32510 MiB, 30154 MiB, 2356 MiB
2022/05/15 02:00:39.222, Tesla V100-SXM2-32GB, 470.57.02, 5 %, 3 %, 32510 MiB, 30154 MiB, 2356 MiB
2022/05/15 02:00:39.222, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30074 MiB, 2436 MiB
2022/05/15 02:00:39.224, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30130 MiB, 2380 MiB
2022/05/15 02:00:39.225, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30130 MiB, 2380 MiB
2022/05/15 02:00:39.227, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30130 MiB, 2380 MiB
2022/05/15 02:00:39.228, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30074 MiB, 2436 MiB
2022/05/15 02:00:39.229, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30130 MiB, 2380 MiB
2022/05/15 02:00:39.232, Tesla V100-SXM2-32GB, 470.57.02, 5 %, 3 %, 32510 MiB, 30154 MiB, 2356 MiB
2022/05/15 02:00:39.234, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30074 MiB, 2436 MiB
2022/05/15 02:00:39.235, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30074 MiB, 2436 MiB
2022/05/15 02:00:39.237, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30074 MiB, 2436 MiB
2022/05/15 02:00:39.237, Tesla V100-SXM2-32GB, 470.57.02, 5 %, 3 %, 32510 MiB, 30154 MiB, 2356 MiB
2022/05/15 02:00:39.238, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30074 MiB, 2436 MiB
2022/05/15 02:00:39.243, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30154 MiB, 2356 MiB
2022/05/15 02:00:39.245, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30154 MiB, 2356 MiB
2022/05/15 02:00:39.246, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30154 MiB, 2356 MiB
2022/05/15 02:00:39.248, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30154 MiB, 2356 MiB
[rank:0] iter: 200/1100, loss: 0.4712736904621124, latency(ms): 24.6418562158942223 | 2022-05-15 02:00:41.406
[rank:0] iter: 300/1100, loss: 0.4640163481235504, latency(ms): 20.9479621797800064 | 2022-05-15 02:00:43.501
[rank:0] iter: 400/1100, loss: 0.4603889584541321, latency(ms): 21.2997622787952423 | 2022-05-15 02:00:45.631
[rank:0] iter: 500/1100, loss: 0.4576707780361176, latency(ms): 21.5230183303356171 | 2022-05-15 02:00:47.783
[rank:0] iter: 600/1100, loss: 0.4531716108322144, latency(ms): 21.3238821923732758 | 2022-05-15 02:00:49.915
[rank:0] iter: 700/1100, loss: 0.4460863769054413, latency(ms): 20.5383849516510963 | 2022-05-15 02:00:51.969
[rank:0] iter: 800/1100, loss: 0.4461094737052917, latency(ms): 20.6757565215229988 | 2022-05-15 02:00:54.037
[rank:0] iter: 900/1100, loss: 0.4464496970176697, latency(ms): 20.9715750440955162 | 2022-05-15 02:00:56.134
[rank:0] iter: 1000/1100, loss: 0.4439864754676819, latency(ms): 20.3810410946607590 | 2022-05-15 02:00:58.172
[rank:0] iter: 1100/1100, loss: 0.4471030831336975, latency(ms): 20.0993383675813675 | 2022-05-15 02:01:00.182