loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
loaded library: loaded library: loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1/usr/lib/x86_64-linux-gnu/libibverbs.so.1

/usr/lib/x86_64-linux-gnu/libibverbs.so.1
loaded library: loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1/usr/lib/x86_64-linux-gnu/libibverbs.so.1

loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
W20220428 02:19:52.061692  5237 rpc_client.cpp:190] LoadServer 10.7.174.10 Failed at 0 times error_code 14 error_message failed to connect to all addresses
W20220428 02:19:52.060853  5236 rpc_client.cpp:190] LoadServer 10.7.174.10 Failed at 0 times error_code 14 error_message failed to connect to all addresses
------------------------ arguments ------------------------
  batch_size ...................................... 512
  batch_size_per_proc ............................. 64
  data_dir ........................................ /dataset/f9f659c5/wdl_ofrecord
  data_part_name_suffix_length .................... 5
  data_part_num ................................... 256
  dataset_format .................................. ofrecord
  ddp ............................................. True
  deep_dropout_rate ............................... 0.5
  deep_embedding_vec_size ......................... 16
  deep_vocab_size ................................. 2322444
  eval_after_training ............................. False
  eval_batchs ..................................... 20
  eval_interval ................................... 0
  execution_mode .................................. eager
  hidden_size ..................................... 1024
  hidden_units_num ................................ 2
  learning_rate ................................... 0.001
  loss_print_every_n_iter ......................... 100
  max_iter ........................................ 1100
  model_load_dir .................................. 
  model_save_dir .................................. ./checkpoint
  num_deep_sparse_fields .......................... 26
  num_dense_fields ................................ 13
  num_wide_sparse_fields .......................... 2
  save_initial_model .............................. False
  save_model_after_each_eval ...................... False
  test_name ....................................... noname_test
  wide_vocab_size ................................. 2322444
-------------------- end of arguments ---------------------
[rank:0] iter: 100/1100, loss: 0.5393068790435791, latency(ms): 72.1941700577735901 | 2022-04-28 02:20:10.844
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/28 02:20:10.995, Tesla V100-SXM2-32GB, 470.57.02, 52 %, 45 %, 32510 MiB, 30338 MiB, 2172 MiB
2022/04/28 02:20:11.003, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 33 %, 32510 MiB, 30358 MiB, 2152 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/28 02:20:11.010, Tesla V100-SXM2-32GB, 470.57.02, 79 %, 69 %, 32510 MiB, 30434 MiB, 2076 MiB
2022/04/28 02:20:11.014, Tesla V100-SXM2-32GB, 470.57.02, 52 %, 45 %, 32510 MiB, 30338 MiB, 2172 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/28 02:20:11.017, Tesla V100-SXM2-32GB, 470.57.02, 52 %, 45 %, 32510 MiB, 30338 MiB, 2172 MiB
2022/04/28 02:20:11.018, Tesla V100-SXM2-32GB, 470.57.02, 21 %, 18 %, 32510 MiB, 30410 MiB, 2100 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/28 02:20:11.023, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 33 %, 32510 MiB, 30358 MiB, 2152 MiB
2022/04/28 02:20:11.023, Tesla V100-SXM2-32GB, 470.57.02, 52 %, 45 %, 32510 MiB, 30338 MiB, 2172 MiB
2022/04/28 02:20:11.024, Tesla V100-SXM2-32GB, 470.57.02, 52 %, 45 %, 32510 MiB, 30338 MiB, 2172 MiB
2022/04/28 02:20:11.025, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 33 %, 32510 MiB, 30358 MiB, 2152 MiB
2022/04/28 02:20:11.025, Tesla V100-SXM2-32GB, 470.57.02, 36 %, 31 %, 32510 MiB, 30418 MiB, 2092 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/28 02:20:11.030, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30434 MiB, 2076 MiB
2022/04/28 02:20:11.030, Tesla V100-SXM2-32GB, 470.57.02, 52 %, 45 %, 32510 MiB, 30338 MiB, 2172 MiB
2022/04/28 02:20:11.032, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 33 %, 32510 MiB, 30358 MiB, 2152 MiB
2022/04/28 02:20:11.033, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 33 %, 32510 MiB, 30358 MiB, 2152 MiB
2022/04/28 02:20:11.033, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30434 MiB, 2076 MiB
2022/04/28 02:20:11.033, Tesla V100-SXM2-32GB, 470.57.02, 23 %, 20 %, 32510 MiB, 30350 MiB, 2160 MiB
2022/04/28 02:20:11.033, Tesla V100-SXM2-32GB, 470.57.02, 52 %, 45 %, 32510 MiB, 30338 MiB, 2172 MiB
2022/04/28 02:20:11.036, Tesla V100-SXM2-32GB, 470.57.02, 52 %, 45 %, 32510 MiB, 30338 MiB, 2172 MiB
2022/04/28 02:20:11.038, Tesla V100-SXM2-32GB, 470.57.02, 21 %, 18 %, 32510 MiB, 30410 MiB, 2100 MiB
2022/04/28 02:20:11.039, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 33 %, 32510 MiB, 30358 MiB, 2152 MiB
2022/04/28 02:20:11.041, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30434 MiB, 2076 MiB
2022/04/28 02:20:11.042, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30434 MiB, 2076 MiB
2022/04/28 02:20:11.042, Tesla V100-SXM2-32GB, 470.57.02, 21 %, 18 %, 32510 MiB, 30410 MiB, 2100 MiB
2022/04/28 02:20:11.042, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30294 MiB, 2216 MiB
2022/04/28 02:20:11.044, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 33 %, 32510 MiB, 30358 MiB, 2152 MiB
2022/04/28 02:20:11.046, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 33 %, 32510 MiB, 30358 MiB, 2152 MiB
2022/04/28 02:20:11.048, Tesla V100-SXM2-32GB, 470.57.02, 36 %, 31 %, 32510 MiB, 30418 MiB, 2092 MiB
2022/04/28 02:20:11.049, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30434 MiB, 2076 MiB
2022/04/28 02:20:11.050, Tesla V100-SXM2-32GB, 470.57.02, 21 %, 18 %, 32510 MiB, 30410 MiB, 2100 MiB
2022/04/28 02:20:11.051, Tesla V100-SXM2-32GB, 470.57.02, 21 %, 18 %, 32510 MiB, 30410 MiB, 2100 MiB
2022/04/28 02:20:11.052, Tesla V100-SXM2-32GB, 470.57.02, 36 %, 31 %, 32510 MiB, 30418 MiB, 2092 MiB
2022/04/28 02:20:11.052, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30374 MiB, 2136 MiB
2022/04/28 02:20:11.053, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30434 MiB, 2076 MiB
2022/04/28 02:20:11.056, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30434 MiB, 2076 MiB
2022/04/28 02:20:11.058, Tesla V100-SXM2-32GB, 470.57.02, 23 %, 20 %, 32510 MiB, 30350 MiB, 2160 MiB
2022/04/28 02:20:11.059, Tesla V100-SXM2-32GB, 470.57.02, 21 %, 18 %, 32510 MiB, 30410 MiB, 2100 MiB
2022/04/28 02:20:11.060, Tesla V100-SXM2-32GB, 470.57.02, 36 %, 31 %, 32510 MiB, 30418 MiB, 2092 MiB
2022/04/28 02:20:11.061, Tesla V100-SXM2-32GB, 470.57.02, 36 %, 31 %, 32510 MiB, 30418 MiB, 2092 MiB
2022/04/28 02:20:11.061, Tesla V100-SXM2-32GB, 470.57.02, 23 %, 20 %, 32510 MiB, 30350 MiB, 2160 MiB
2022/04/28 02:20:11.063, Tesla V100-SXM2-32GB, 470.57.02, 21 %, 18 %, 32510 MiB, 30410 MiB, 2100 MiB
2022/04/28 02:20:11.065, Tesla V100-SXM2-32GB, 470.57.02, 21 %, 18 %, 32510 MiB, 30410 MiB, 2100 MiB
2022/04/28 02:20:11.067, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30294 MiB, 2216 MiB
2022/04/28 02:20:11.068, Tesla V100-SXM2-32GB, 470.57.02, 36 %, 31 %, 32510 MiB, 30418 MiB, 2092 MiB
2022/04/28 02:20:11.069, Tesla V100-SXM2-32GB, 470.57.02, 23 %, 20 %, 32510 MiB, 30350 MiB, 2160 MiB
2022/04/28 02:20:11.070, Tesla V100-SXM2-32GB, 470.57.02, 23 %, 20 %, 32510 MiB, 30350 MiB, 2160 MiB
2022/04/28 02:20:11.070, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30294 MiB, 2216 MiB
2022/04/28 02:20:11.072, Tesla V100-SXM2-32GB, 470.57.02, 36 %, 31 %, 32510 MiB, 30418 MiB, 2092 MiB
2022/04/28 02:20:11.074, Tesla V100-SXM2-32GB, 470.57.02, 36 %, 31 %, 32510 MiB, 30418 MiB, 2092 MiB
2022/04/28 02:20:11.076, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30374 MiB, 2136 MiB
2022/04/28 02:20:11.077, Tesla V100-SXM2-32GB, 470.57.02, 23 %, 20 %, 32510 MiB, 30350 MiB, 2160 MiB
2022/04/28 02:20:11.078, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30294 MiB, 2216 MiB
2022/04/28 02:20:11.079, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30294 MiB, 2216 MiB
2022/04/28 02:20:11.079, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30374 MiB, 2136 MiB
2022/04/28 02:20:11.081, Tesla V100-SXM2-32GB, 470.57.02, 23 %, 20 %, 32510 MiB, 30350 MiB, 2160 MiB
2022/04/28 02:20:11.084, Tesla V100-SXM2-32GB, 470.57.02, 23 %, 20 %, 32510 MiB, 30350 MiB, 2160 MiB
2022/04/28 02:20:11.086, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30294 MiB, 2216 MiB
2022/04/28 02:20:11.088, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30374 MiB, 2136 MiB
2022/04/28 02:20:11.088, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30374 MiB, 2136 MiB
2022/04/28 02:20:11.090, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30294 MiB, 2216 MiB
2022/04/28 02:20:11.092, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30294 MiB, 2216 MiB
2022/04/28 02:20:11.095, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30374 MiB, 2136 MiB
2022/04/28 02:20:11.101, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30374 MiB, 2136 MiB
2022/04/28 02:20:11.103, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30374 MiB, 2136 MiB
[rank:0] iter: 200/1100, loss: 0.4911498725414276, latency(ms): 8.8967929780483246 | 2022-04-28 02:20:11.734
[rank:0] iter: 300/1100, loss: 0.4947015643119812, latency(ms): 6.3131017610430717 | 2022-04-28 02:20:12.365
[rank:0] iter: 400/1100, loss: 0.4925323128700256, latency(ms): 6.4820468425750732 | 2022-04-28 02:20:13.013
[rank:0] iter: 500/1100, loss: 0.4969005882740021, latency(ms): 6.5426242724061012 | 2022-04-28 02:20:13.668
[rank:0] iter: 600/1100, loss: 0.4804946780204773, latency(ms): 6.7625175788998604 | 2022-04-28 02:20:14.344
[rank:0] iter: 700/1100, loss: 0.4729176461696625, latency(ms): 6.5202464908361435 | 2022-04-28 02:20:14.996
[rank:0] iter: 800/1100, loss: 0.4827856719493866, latency(ms): 6.1302545666694641 | 2022-04-28 02:20:15.609
[rank:0] iter: 900/1100, loss: 0.4839979410171509, latency(ms): 6.2840386480093002 | 2022-04-28 02:20:16.237
[rank:0] iter: 1000/1100, loss: 0.4808856248855591, latency(ms): 6.4108557626605034 | 2022-04-28 02:20:16.878
[rank:0] iter: 1100/1100, loss: 0.4756367206573486, latency(ms): 6.2945856526494026 | 2022-04-28 02:20:17.508