loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
loaded library: loaded library: loaded library: loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1/usr/lib/x86_64-linux-gnu/libibverbs.so.1
/usr/lib/x86_64-linux-gnu/libibverbs.so.1/usr/lib/x86_64-linux-gnu/libibverbs.so.1


loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1
W20220422 10:31:11.335108  2004 rpc_client.cpp:190] LoadServer 10.7.0.203 Failed at 0 times error_code 14 error_message failed to connect to all addresses
W20220422 10:31:11.335856  2003 rpc_client.cpp:190] LoadServer 10.7.0.203 Failed at 0 times error_code 14 error_message failed to connect to all addresses
W20220422 10:31:11.335846  2002 rpc_client.cpp:190] LoadServer 10.7.0.203 Failed at 0 times error_code 14 error_message failed to connect to all addresses
------------------------ arguments ------------------------
  batch_size ...................................... 65536
  batch_size_per_proc ............................. 8192
  data_dir ........................................ /dataset/f9f659c5/wdl_ofrecord
  data_part_name_suffix_length .................... 5
  data_part_num ................................... 256
  dataset_format .................................. ofrecord
  ddp ............................................. False
  deep_dropout_rate ............................... 0.5
  deep_embedding_vec_size ......................... 16
  deep_vocab_size ................................. 2322444
  eval_after_training ............................. False
  eval_batchs ..................................... 20
  eval_interval ................................... 0
  execution_mode .................................. graph
  hidden_size ..................................... 1024
  hidden_units_num ................................ 2
  learning_rate ................................... 0.001
  loss_print_every_n_iter ......................... 100
  max_iter ........................................ 1100
  model_load_dir .................................. 
  model_save_dir .................................. ./checkpoint
  num_deep_sparse_fields .......................... 26
  num_dense_fields ................................ 13
  num_wide_sparse_fields .......................... 2
  save_initial_model .............................. False
  save_model_after_each_eval ...................... False
  test_name ....................................... noname_test
  wide_vocab_size ................................. 2322444
-------------------- end of arguments ---------------------
[rank:0] iter: 100/1100, loss: 0.0620782151818275, latency(ms): 188.0600667744874954 | 2022-04-22 10:31:41.909
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/22 10:31:42.066, Tesla V100-SXM2-32GB, 470.57.02, 8 %, 3 %, 32510 MiB, 30039 MiB, 2471 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/22 10:31:42.073, Tesla V100-SXM2-32GB, 470.57.02, 32 %, 11 %, 32510 MiB, 30091 MiB, 2419 MiB
2022/04/22 10:31:42.076, Tesla V100-SXM2-32GB, 470.57.02, 8 %, 3 %, 32510 MiB, 30039 MiB, 2471 MiB
2022/04/22 10:31:42.077, Tesla V100-SXM2-32GB, 470.57.02, 8 %, 3 %, 32510 MiB, 30039 MiB, 2471 MiB
2022/04/22 10:31:42.078, Tesla V100-SXM2-32GB, 470.57.02, 8 %, 3 %, 32510 MiB, 30039 MiB, 2471 MiB
2022/04/22 10:31:42.080, Tesla V100-SXM2-32GB, 470.57.02, 84 %, 30 %, 32510 MiB, 30211 MiB, 2299 MiB
2022/04/22 10:31:42.085, Tesla V100-SXM2-32GB, 470.57.02, 32 %, 11 %, 32510 MiB, 30091 MiB, 2419 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/22 10:31:42.086, Tesla V100-SXM2-32GB, 470.57.02, 32 %, 11 %, 32510 MiB, 30091 MiB, 2419 MiB
2022/04/22 10:31:42.088, Tesla V100-SXM2-32GB, 470.57.02, 32 %, 11 %, 32510 MiB, 30091 MiB, 2419 MiB
2022/04/22 10:31:42.090, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 13 %, 32510 MiB, 30195 MiB, 2315 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2022/04/22 10:31:42.093, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30211 MiB, 2299 MiB
2022/04/22 10:31:42.095, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30211 MiB, 2299 MiB
2022/04/22 10:31:42.094, Tesla V100-SXM2-32GB, 470.57.02, 8 %, 3 %, 32510 MiB, 30039 MiB, 2471 MiB
2022/04/22 10:31:42.096, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30211 MiB, 2299 MiB
2022/04/22 10:31:42.097, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30179 MiB, 2331 MiB
2022/04/22 10:31:42.097, Tesla V100-SXM2-32GB, 470.57.02, 8 %, 3 %, 32510 MiB, 30039 MiB, 2471 MiB
2022/04/22 10:31:42.099, Tesla V100-SXM2-32GB, 470.57.02, 8 %, 3 %, 32510 MiB, 30039 MiB, 2471 MiB
2022/04/22 10:31:42.101, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 13 %, 32510 MiB, 30195 MiB, 2315 MiB
2022/04/22 10:31:42.101, Tesla V100-SXM2-32GB, 470.57.02, 8 %, 3 %, 32510 MiB, 30039 MiB, 2471 MiB
2022/04/22 10:31:42.103, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 13 %, 32510 MiB, 30195 MiB, 2315 MiB
2022/04/22 10:31:42.103, Tesla V100-SXM2-32GB, 470.57.02, 32 %, 11 %, 32510 MiB, 30091 MiB, 2419 MiB
2022/04/22 10:31:42.104, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 13 %, 32510 MiB, 30195 MiB, 2315 MiB
2022/04/22 10:31:42.106, Tesla V100-SXM2-32GB, 470.57.02, 66 %, 24 %, 32510 MiB, 30043 MiB, 2467 MiB
2022/04/22 10:31:42.107, Tesla V100-SXM2-32GB, 470.57.02, 32 %, 11 %, 32510 MiB, 30091 MiB, 2419 MiB
2022/04/22 10:31:42.110, Tesla V100-SXM2-32GB, 470.57.02, 32 %, 11 %, 32510 MiB, 30091 MiB, 2419 MiB
2022/04/22 10:31:42.111, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30179 MiB, 2331 MiB
2022/04/22 10:31:42.112, Tesla V100-SXM2-32GB, 470.57.02, 32 %, 11 %, 32510 MiB, 30091 MiB, 2419 MiB
2022/04/22 10:31:42.113, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30179 MiB, 2331 MiB
2022/04/22 10:31:42.113, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30211 MiB, 2299 MiB
2022/04/22 10:31:42.114, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30179 MiB, 2331 MiB
2022/04/22 10:31:42.116, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29931 MiB, 2579 MiB
2022/04/22 10:31:42.117, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30211 MiB, 2299 MiB
2022/04/22 10:31:42.120, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30211 MiB, 2299 MiB
2022/04/22 10:31:42.121, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30043 MiB, 2467 MiB
2022/04/22 10:31:42.122, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30211 MiB, 2299 MiB
2022/04/22 10:31:42.123, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30043 MiB, 2467 MiB
2022/04/22 10:31:42.123, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 13 %, 32510 MiB, 30195 MiB, 2315 MiB
2022/04/22 10:31:42.124, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30043 MiB, 2467 MiB
2022/04/22 10:31:42.126, Tesla V100-SXM2-32GB, 470.57.02, 1 %, 0 %, 32510 MiB, 30123 MiB, 2387 MiB
2022/04/22 10:31:42.127, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 13 %, 32510 MiB, 30195 MiB, 2315 MiB
2022/04/22 10:31:42.130, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 13 %, 32510 MiB, 30195 MiB, 2315 MiB
2022/04/22 10:31:42.131, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29931 MiB, 2579 MiB
2022/04/22 10:31:42.132, Tesla V100-SXM2-32GB, 470.57.02, 38 %, 13 %, 32510 MiB, 30195 MiB, 2315 MiB
2022/04/22 10:31:42.133, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29931 MiB, 2579 MiB
2022/04/22 10:31:42.133, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30179 MiB, 2331 MiB
2022/04/22 10:31:42.134, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29931 MiB, 2579 MiB
2022/04/22 10:31:42.137, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30179 MiB, 2331 MiB
2022/04/22 10:31:42.140, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30179 MiB, 2331 MiB
2022/04/22 10:31:42.141, Tesla V100-SXM2-32GB, 470.57.02, 1 %, 0 %, 32510 MiB, 30123 MiB, 2387 MiB
2022/04/22 10:31:42.142, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30179 MiB, 2331 MiB
2022/04/22 10:31:42.143, Tesla V100-SXM2-32GB, 470.57.02, 1 %, 0 %, 32510 MiB, 30123 MiB, 2387 MiB
2022/04/22 10:31:42.143, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30043 MiB, 2467 MiB
2022/04/22 10:31:42.144, Tesla V100-SXM2-32GB, 470.57.02, 1 %, 0 %, 32510 MiB, 30123 MiB, 2387 MiB
2022/04/22 10:31:42.147, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30043 MiB, 2467 MiB
2022/04/22 10:31:42.150, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30043 MiB, 2467 MiB
2022/04/22 10:31:42.152, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 30043 MiB, 2467 MiB
2022/04/22 10:31:42.153, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29931 MiB, 2579 MiB
2022/04/22 10:31:42.157, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29931 MiB, 2579 MiB
2022/04/22 10:31:42.159, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29931 MiB, 2579 MiB
2022/04/22 10:31:42.160, Tesla V100-SXM2-32GB, 470.57.02, 0 %, 0 %, 32510 MiB, 29931 MiB, 2579 MiB
2022/04/22 10:31:42.162, Tesla V100-SXM2-32GB, 470.57.02, 1 %, 0 %, 32510 MiB, 30123 MiB, 2387 MiB
2022/04/22 10:31:42.165, Tesla V100-SXM2-32GB, 470.57.02, 1 %, 0 %, 32510 MiB, 30123 MiB, 2387 MiB
2022/04/22 10:31:42.167, Tesla V100-SXM2-32GB, 470.57.02, 1 %, 0 %, 32510 MiB, 30123 MiB, 2387 MiB
2022/04/22 10:31:42.168, Tesla V100-SXM2-32GB, 470.57.02, 1 %, 0 %, 32510 MiB, 30123 MiB, 2387 MiB
[rank:0] iter: 200/1100, loss: 0.0584514476358891, latency(ms): 15.0397500395774841 | 2022-04-22 10:31:43.413
[rank:0] iter: 300/1100, loss: 0.0577457770705223, latency(ms): 12.2339709475636482 | 2022-04-22 10:31:44.636
[rank:0] iter: 400/1100, loss: 0.0573659427464008, latency(ms): 12.2615105286240578 | 2022-04-22 10:31:45.862
[rank:0] iter: 500/1100, loss: 0.0570722371339798, latency(ms): 12.3557006195187569 | 2022-04-22 10:31:47.098
[rank:0] iter: 600/1100, loss: 0.0565957389771938, latency(ms): 12.5106833502650261 | 2022-04-22 10:31:48.349
[rank:0] iter: 700/1100, loss: 0.0558313131332397, latency(ms): 11.8272380903363228 | 2022-04-22 10:31:49.531
[rank:0] iter: 800/1100, loss: 0.0558002889156342, latency(ms): 11.9490972533822060 | 2022-04-22 10:31:50.726
[rank:0] iter: 900/1100, loss: 0.0557189732789993, latency(ms): 11.7521426081657410 | 2022-04-22 10:31:51.902
[rank:0] iter: 1000/1100, loss: 0.0555099882185459, latency(ms): 12.1871373429894447 | 2022-04-22 10:31:53.120
[rank:0] iter: 1100/1100, loss: 0.0556718111038208, latency(ms): 12.2139457613229752 | 2022-04-22 10:31:54.342