loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1 loaded library: /usr/lib/x86_64-linux-gnu/libibverbs.so.1 ------------------------ arguments ------------------------ batch_size ...................................... 512 batch_size_per_proc ............................. 512 data_dir ........................................ /dataset/f9f659c5/wdl_ofrecord data_part_name_suffix_length .................... 5 data_part_num ................................... 256 dataset_format .................................. ofrecord ddp ............................................. False deep_dropout_rate ............................... 0.5 deep_embedding_vec_size ......................... 16 deep_vocab_size ................................. 2322444 eval_after_training ............................. False eval_batchs ..................................... 20 eval_interval ................................... 0 execution_mode .................................. graph hidden_size ..................................... 1024 hidden_units_num ................................ 2 learning_rate ................................... 0.001 loss_print_every_n_iter ......................... 100 max_iter ........................................ 1100 model_load_dir .................................. model_save_dir .................................. ./checkpoint num_deep_sparse_fields .......................... 26 num_dense_fields ................................ 13 num_wide_sparse_fields .......................... 2 save_initial_model .............................. False save_model_after_each_eval ...................... False test_name ....................................... noname_test wide_vocab_size ................................. 2322444 -------------------- end of arguments --------------------- [rank:0] iter: 100/1100, loss: 0.5248926281929016, latency(ms): 38.0227279290556908 | 2022-04-30 02:08:54.088 timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB] 2022/04/30 02:08:54.113, Tesla V100-SXM2-32GB, 470.57.02, 20 %, 4 %, 32510 MiB, 31032 MiB, 1478 MiB [rank:0] iter: 200/1100, loss: 0.4996577203273773, latency(ms): 1.7566780373454094 | 2022-04-30 02:08:54.264 [rank:0] iter: 300/1100, loss: 0.4975259900093079, latency(ms): 1.6819770261645317 | 2022-04-30 02:08:54.432 [rank:0] iter: 400/1100, loss: 0.4916962683200836, latency(ms): 1.3963253796100616 | 2022-04-30 02:08:54.572 [rank:0] iter: 500/1100, loss: 0.4951523244380951, latency(ms): 1.6379926726222038 | 2022-04-30 02:08:54.736 [rank:0] iter: 600/1100, loss: 0.4827251136302948, latency(ms): 1.4443489909172058 | 2022-04-30 02:08:54.880 [rank:0] iter: 700/1100, loss: 0.4873725771903992, latency(ms): 1.6563988849520683 | 2022-04-30 02:08:55.046 [rank:0] iter: 800/1100, loss: 0.4875635802745819, latency(ms): 1.3590997830033302 | 2022-04-30 02:08:55.182 [rank:0] iter: 900/1100, loss: 0.4842551052570343, latency(ms): 1.7024048790335655 | 2022-04-30 02:08:55.352 [rank:0] iter: 1000/1100, loss: 0.4832169711589813, latency(ms): 1.6978674381971359 | 2022-04-30 02:08:55.522 [rank:0] iter: 1100/1100, loss: 0.4839311540126801, latency(ms): 1.4826621115207672 | 2022-04-30 02:08:55.670