[03/11 01:12:16 libai]: Rank of current process: 0. World size: 8
[03/11 01:12:16 libai]: Command line arguments: Namespace(config_file='configs/swin_imagenet.py', eval_only=False, fast_dev_run=False, opts=['model.cfg.hidden_dropout_prob=0.1', 'model.cfg.attention_probs_dropout_prob=0.1', 'model.cfg.bias_dropout_fusion=true', 'model.cfg.hidden_layers=12', 'model.cfg.hidden_size=768', 'model.cfg.num_attention_heads=12', 'model.cfg.intermediate_size=3072', 'model.cfg.ffn_hidden_size=3072', 'model.cfg.head_size=64', 'graph.enabled=true', 'train.dist.pipeline_num_layers=12', 'train.train_micro_batch_size=64', 'train.global_batch_size=2048', 'train.dist.tensor_parallel_size=2', 'train.dist.pipeline_parallel_size=1', 'train.amp.enabled=true', 'train.activation_checkpoint.enabled=true', 'train.num_accumulation_steps=8', 'train.evaluation.enabled=false', 'train.train_iter=220', 'train.train_epoch=0', 'train.log_period=100', 'train.zero_optimization.enabled=true', 'train.zero_optimization.stage=2', 'train.load_weight=', 'train.output_dir=test_logs/oneflow-28/NVIDIA_GeForce_RTX_3080_Ti/1ea2bb7/LibAI_swin_imagenet_graph_nl12_nah12_hs768_FP16_actrue_DP4_MP2_PP1_zerotrue_stage2_mbs64_gbs2048_acc8_1n8g'], resume=False)
[03/11 01:12:16 libai]: Contents of args.config_file=configs/swin_imagenet.py:
from libai.config import LazyCall
from .common.models.swin.swin_tiny_patch4_window7_224 import model
from .common.models.graph import graph
from .common.train import train
from .common.optim import optim
from .common.data.imagenet import dataloader

from flowvision.data import Mixup
from flowvision.loss.cross_entropy import SoftTargetCrossEntropy

# Refine data path to imagenet
dataloader.train.dataset[0].root = "/ssd/dataset/ImageNet/extract"
dataloader.test[0].dataset.root = "/ssd/dataset/ImageNet/extract"

# Add Mixup Func
dataloader.train.mixup_func = LazyCall(Mixup)(
    mixup_alpha=0.8,
    cutmix_alpha=1.0,
    prob=1.0,
    switch_prob=0.5,
    mode="batch",
    num_classes=1000,
)

# Refine model cfg for vit training on imagenet
model.cfg.num_classes = 1000
model.cfg.loss_func = SoftTargetCrossEntropy()
# Refine optimizer cfg for vit model
optim.lr = 1e-3
optim.eps = 1e-8
optim.weight_decay = 0.05
optim.params.clip_grad_max_norm = None
optim.params.clip_grad_norm_type = None

# Refine train cfg for vit model
train.train_micro_batch_size = 128
train.test_micro_batch_size = 128
train.train_epoch = 300
train.warmup_ratio = 20 / 300
train.eval_period = 1562
train.log_period = 100

# Scheduler
train.scheduler.warmup_factor = 0.001
train.scheduler.alpha = 0.01
train.scheduler.warmup_method = "linear"

# Set fp16 ON
train.amp.enabled = True

[03/11 01:12:16 libai]: Full config saved to test_logs/oneflow-28/NVIDIA_GeForce_RTX_3080_Ti/1ea2bb7/LibAI_swin_imagenet_graph_nl12_nah12_hs768_FP16_actrue_DP4_MP2_PP1_zerotrue_stage2_mbs64_gbs2048_acc8_1n8g/config.yaml
[03/11 01:12:16 lb.engine.default]: > compiling dataset index builder ...
make: Entering directory '/ssd/home/ouyangyu/libai_week_test/libai/libai/data/data_utils'
make: Nothing to be done for 'default'.
make: Leaving directory '/ssd/home/ouyangyu/libai_week_test/libai/libai/data/data_utils'
[03/11 01:12:16 lb.engine.default]: >>> done with dataset index builder. Compilation time: 0.054 seconds
[03/11 01:12:16 lb.engine.default]: >>> done with compiling. Compilation time: 0.056 seconds
[03/11 01:12:16 lb.engine.default]: Prepare training, validating, testing set
[03/11 01:12:20 lb.engine.default]: Prepare testing set
[03/11 01:12:30 lb.engine.default]: Auto-scaling the config to train.train_iter=220, train.warmup_iter=15
[03/11 01:12:30 libai]: > Start building model...
W20230311 01:12:33.588057 3210148 eager_local_op_interpreter.cpp:256] Casting a local tensor to a global tensor with Broadcast sbp will modify the data of input! If you want to keep the input local tensor unchanged, please set the arg copy to True.
[03/11 01:12:35 lb.engine.default]: Model:
SwinTransformer(
  (patch_embed): PatchEmbed(
    (proj): Conv2d(3, 96, kernel_size=(4, 4), stride=(4, 4))
    (norm): LayerNorm((96,), eps=1e-05, elementwise_affine=True)
  )
  (pos_drop): Dropout(p=0.0, inplace=False)
  (layers): ModuleList(
    (0): BasicLayer(
      (blocks): ModuleList(
        (0): SwinTransformerBlock(
          (norm1): LayerNorm((96,), eps=1e-05, elementwise_affine=True)
          (attn): WindowAttention(
            (qkv): Linear1D(in_features=96, out_features=288, bias=True, parallel=data)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear1D(in_features=96, out_features=96, bias=True, parallel=data)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (softmax): Softmax(dim=-1)
          )
          (drop_path): Identity()
          (norm2): LayerNorm((96,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0
            (dense_h_to_4h): Linear1D(in_features=96, out_features=384, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=384, out_features=96, bias=True, parallel=row)
          )
        )
        (1): SwinTransformerBlock(
          (norm1): LayerNorm((96,), eps=1e-05, elementwise_affine=True)
          (attn): WindowAttention(
            (qkv): Linear1D(in_features=96, out_features=288, bias=True, parallel=data)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear1D(in_features=96, out_features=96, bias=True, parallel=data)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (softmax): Softmax(dim=-1)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((96,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0
            (dense_h_to_4h): Linear1D(in_features=96, out_features=384, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=384, out_features=96, bias=True, parallel=row)
          )
        )
      )
      (downsample): PatchMerging(
        (reduction): Linear1D(in_features=384, out_features=192, bias=False, parallel=data)
        (norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
      )
    )
    (1): BasicLayer(
      (blocks): ModuleList(
        (0): SwinTransformerBlock(
          (norm1): LayerNorm((192,), eps=1e-05, elementwise_affine=True)
          (attn): WindowAttention(
            (qkv): Linear1D(in_features=192, out_features=576, bias=True, parallel=data)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear1D(in_features=192, out_features=192, bias=True, parallel=data)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (softmax): Softmax(dim=-1)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((192,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0
            (dense_h_to_4h): Linear1D(in_features=192, out_features=768, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=768, out_features=192, bias=True, parallel=row)
          )
        )
        (1): SwinTransformerBlock(
          (norm1): LayerNorm((192,), eps=1e-05, elementwise_affine=True)
          (attn): WindowAttention(
            (qkv): Linear1D(in_features=192, out_features=576, bias=True, parallel=data)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear1D(in_features=192, out_features=192, bias=True, parallel=data)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (softmax): Softmax(dim=-1)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((192,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0
            (dense_h_to_4h): Linear1D(in_features=192, out_features=768, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=768, out_features=192, bias=True, parallel=row)
          )
        )
      )
      (downsample): PatchMerging(
        (reduction): Linear1D(in_features=768, out_features=384, bias=False, parallel=data)
        (norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
      )
    )
    (2): BasicLayer(
      (blocks): ModuleList(
        (0): SwinTransformerBlock(
          (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
          (attn): WindowAttention(
            (qkv): Linear1D(in_features=384, out_features=1152, bias=True, parallel=data)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (softmax): Softmax(dim=-1)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0
            (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row)
          )
        )
        (1): SwinTransformerBlock(
          (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
          (attn): WindowAttention(
            (qkv): Linear1D(in_features=384, out_features=1152, bias=True, parallel=data)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (softmax): Softmax(dim=-1)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0
            (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row)
          )
        )
        (2): SwinTransformerBlock(
          (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
          (attn): WindowAttention(
            (qkv): Linear1D(in_features=384, out_features=1152, bias=True, parallel=data)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (softmax): Softmax(dim=-1)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0
            (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row)
          )
        )
        (3): SwinTransformerBlock(
          (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
          (attn): WindowAttention(
            (qkv): Linear1D(in_features=384, out_features=1152, bias=True, parallel=data)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (softmax): Softmax(dim=-1)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0
            (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row)
          )
        )
        (4): SwinTransformerBlock(
          (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
          (attn): WindowAttention(
            (qkv): Linear1D(in_features=384, out_features=1152, bias=True, parallel=data)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (softmax): Softmax(dim=-1)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0
            (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row)
          )
        )
        (5): SwinTransformerBlock(
          (norm1): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
          (attn): WindowAttention(
            (qkv): Linear1D(in_features=384, out_features=1152, bias=True, parallel=data)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear1D(in_features=384, out_features=384, bias=True, parallel=data)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (softmax): Softmax(dim=-1)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((384,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0
            (dense_h_to_4h): Linear1D(in_features=384, out_features=1536, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=1536, out_features=384, bias=True, parallel=row)
          )
        )
      )
      (downsample): PatchMerging(
        (reduction): Linear1D(in_features=1536, out_features=768, bias=False, parallel=data)
        (norm): LayerNorm((1536,), eps=1e-05, elementwise_affine=True)
      )
    )
    (3): BasicLayer(
      (blocks): ModuleList(
        (0): SwinTransformerBlock(
          (norm1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
          (attn): WindowAttention(
            (qkv): Linear1D(in_features=768, out_features=2304, bias=True, parallel=data)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear1D(in_features=768, out_features=768, bias=True, parallel=data)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (softmax): Softmax(dim=-1)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0
            (dense_h_to_4h): Linear1D(in_features=768, out_features=3072, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=3072, out_features=768, bias=True, parallel=row)
          )
        )
        (1): SwinTransformerBlock(
          (norm1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
          (attn): WindowAttention(
            (qkv): Linear1D(in_features=768, out_features=2304, bias=True, parallel=data)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear1D(in_features=768, out_features=768, bias=True, parallel=data)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (softmax): Softmax(dim=-1)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
          (mlp): MLP(
            bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.0
            (dense_h_to_4h): Linear1D(in_features=768, out_features=3072, bias=True, parallel=col)
            (dense_4h_to_h): Linear1D(in_features=3072, out_features=768, bias=True, parallel=row)
          )
        )
      )
    )
  )
  (norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
  (avgpool): AdaptiveAvgPool1d()
  (head): Linear1D(in_features=768, out_features=1000, bias=True, parallel=data)
  (loss_func): SoftTargetCrossEntropy()
)
[03/11 01:12:35 libai]: >>> done with building model. Building time: 4.554 seconds
[03/11 01:12:35 lb.engine.trainer]: Starting training from iteration 0
[03/11 01:12:37 lb.models.utils.graph_base]: Start compiling the train graph which may take some time. Please wait for a moment ...
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2023/03/11 01:19:21.646, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 8609 MiB, 3444 MiB
2023/03/11 01:19:21.653, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 8 %, 2 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.662, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 18 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2023/03/11 01:19:21.668, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 70 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.670, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 8609 MiB, 3444 MiB
2023/03/11 01:19:21.668, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 8609 MiB, 3444 MiB
2023/03/11 01:19:21.669, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 8609 MiB, 3444 MiB
2023/03/11 01:19:21.672, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 8609 MiB, 3444 MiB
2023/03/11 01:19:21.673, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 74 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.678, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 8 %, 2 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.679, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 8 %, 2 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.680, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 8 %, 2 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.681, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 8 %, 2 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.683, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 21 %, 0 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.685, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 18 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.686, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 18 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.690, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 18 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.690, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 18 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.693, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.696, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 70 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.699, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 70 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.701, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 70 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.701, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 70 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.703, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 55 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.706, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 74 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.711, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 74 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.711, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 74 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.712, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 74 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.717, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 21 %, 0 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.720, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 21 %, 0 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.722, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 21 %, 0 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.723, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 21 %, 0 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.726, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 96 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.730, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 96 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.731, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 96 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.733, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 96 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.737, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 55 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2023/03/11 01:19:21.740, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 55 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.740, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 55 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.742, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 55 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.743, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 8609 MiB, 3444 MiB
2023/03/11 01:19:21.753, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 96 %, 0 %, 12288 MiB, 8619 MiB, 3434 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2023/03/11 01:19:21.757, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 100 %, 0 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.766, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 8609 MiB, 3444 MiB
2023/03/11 01:19:21.769, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 70 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.7722023/03/11 01:19:21.772, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 74 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 96 %, 0 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.774, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 21 %, 0 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.775, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 100 %, 0 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.775, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 96 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.776, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 55 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.776, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 70 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.778, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 74 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.781, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 100 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.792, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 96 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:21.793, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 55 %, 1 %, 12288 MiB, 8619 MiB, 3434 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2023/03/11 01:19:24.524, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 99 %, 33 %, 12288 MiB, 8609 MiB, 3444 MiB
2023/03/11 01:19:24.525, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 51 %, 15 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:24.527, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 98 %, 33 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:24.527, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 76 %, 18 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:24.529, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 73 %, 18 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:24.530, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 99 %, 33 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:24.530, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 55 %, 13 %, 12288 MiB, 8619 MiB, 3434 MiB
2023/03/11 01:19:24.531, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 78 %, 27 %, 12288 MiB, 8619 MiB, 3434 MiB
[03/11 01:19:28 lb.utils.events]:  eta: 0:07:33  iteration: 99/220  consumed_samples: 204800  total_loss: 6.886  time: 3.7570 s/iter  data_time: 1.0963 s/iter total_throughput: 545.11 samples/s lr: 5.82e-04  
[03/11 01:25:47 lb.utils.events]:  eta: 0:01:15  iteration: 199/220  consumed_samples: 409600  total_loss: 6.857  time: 3.7735 s/iter  data_time: 0.9067 s/iter total_throughput: 542.74 samples/s lr: 3.21e-05  
[03/11 01:27:01 lb.utils.events]:  eta: 0:00:00  iteration: 219/220  consumed_samples: 450560  total_loss: 6.848  time: 3.7694 s/iter  data_time: 1.0623 s/iter total_throughput: 543.32 samples/s lr: 1.01e-05  
[03/11 01:27:02 lb.engine.hooks]: Overall training speed: 218 iterations in 0:13:41 (3.7695 s / it)
[03/11 01:27:02 lb.engine.hooks]: Total training time: 0:13:41 (0:00:00 on hooks)
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
oneflow-version(git_commit)=0.9.1.dev20230309+cu117
oneflow-commit(git_commit)=1ea2bb7
oneflow-libai(git_commit)=50a973dc5de635b8613ad7666c073c763e238850