[03/10 19:04:06 libai]: Rank of current process: 0. World size: 4
[03/10 19:04:06 libai]: Command line arguments: Namespace(config_file='configs/bert_large_pretrain.py', eval_only=False, fast_dev_run=False, opts=['model.cfg.hidden_dropout_prob=0.1', 'model.cfg.attention_probs_dropout_prob=0.1', 'model.cfg.bias_dropout_fusion=true', 'model.cfg.hidden_layers=48', 'model.cfg.hidden_size=1024', 'model.cfg.num_attention_heads=16', 'model.cfg.intermediate_size=4096', 'model.cfg.ffn_hidden_size=4096', 'model.cfg.head_size=64', 'graph.enabled=true', 'train.dist.pipeline_num_layers=48', 'train.train_micro_batch_size=16', 'train.global_batch_size=128', 'train.dist.tensor_parallel_size=1', 'train.dist.pipeline_parallel_size=4', 'train.amp.enabled=true', 'train.activation_checkpoint.enabled=true', 'train.num_accumulation_steps=8', 'train.evaluation.enabled=false', 'train.train_iter=220', 'train.train_epoch=0', 'train.log_period=100', 'train.zero_optimization.enabled=false', 'train.zero_optimization.stage=0', 'train.load_weight=', 'train.output_dir=test_logs/oneflow-28/NVIDIA_GeForce_RTX_3080_Ti/1ea2bb7/LibAI_bert_large_pretrain_graph_nl48_nah16_hs1024_FP16_actrue_DP1_MP1_PP4_zerofalse_stage0_mbs16_gbs128_acc8_1n4g'], resume=False)
[03/10 19:04:06 libai]: Contents of args.config_file=configs/bert_large_pretrain.py:
from libai.config import LazyCall
from libai.evaluation import PPLEvaluator
from .common.models.bert import pretrain_model as model
from .common.models.graph import graph
from .common.train import train
from .common.optim import optim
from .common.data.bert_dataset import dataloader, tokenization

vocab_file = "./data_test/bert_data/bert-base-chinese-vocab.txt"
data_prefix = "./data_test/bert_data/loss_compara_content_sentence"

tokenization.tokenizer.vocab_file = vocab_file
dataloader.train.dataset[0].data_prefix = data_prefix
dataloader.train.dataset[0].indexed_dataset.data_prefix = data_prefix

# Bert-large model config
model.cfg.num_attention_heads = 16
model.cfg.hidden_size = 768
model.cfg.hidden_layers = 8

train.input_placement_device = "cpu"

train.dist.pipeline_num_layers = model.cfg.hidden_layers

train.train_micro_batch_size = 16

train.amp.enabled = True

for ds in dataloader.train.dataset:
    ds.max_seq_length = model.cfg.max_position_embeddings

train.evaluation.evaluator = LazyCall(PPLEvaluator)()

train.output_dir = "output/bert_output"

[03/10 19:04:06 libai]: Full config saved to test_logs/oneflow-28/NVIDIA_GeForce_RTX_3080_Ti/1ea2bb7/LibAI_bert_large_pretrain_graph_nl48_nah16_hs1024_FP16_actrue_DP1_MP1_PP4_zerofalse_stage0_mbs16_gbs128_acc8_1n4g/config.yaml
[03/10 19:04:06 lb.engine.default]: > compiling dataset index builder ...
make: Entering directory '/ssd/home/ouyangyu/libai_week_test/libai/libai/data/data_utils'
make: Nothing to be done for 'default'.
make: Leaving directory '/ssd/home/ouyangyu/libai_week_test/libai/libai/data/data_utils'
[03/10 19:04:06 lb.engine.default]: >>> done with dataset index builder. Compilation time: 0.057 seconds
[03/10 19:04:06 lb.engine.default]: >>> done with compiling. Compilation time: 0.058 seconds
[03/10 19:04:06 lb.engine.default]: Prepare training, validating, testing set
[03/10 19:04:06 lb.data.data_utils.indexed_dataset]: building dataset index ...
[03/10 19:04:06 lb.data.data_utils.indexed_dataset]: warming up index mmap file...
[03/10 19:04:06 lb.data.data_utils.indexed_dataset]: reading sizes...
[03/10 19:04:06 lb.data.data_utils.indexed_dataset]: reading pointers...
[03/10 19:04:06 lb.data.data_utils.indexed_dataset]: reading document index...
[03/10 19:04:06 lb.data.data_utils.indexed_dataset]: warming up data mmap file...
[03/10 19:04:06 lb.data.data_utils.indexed_dataset]: creating numpy buffer of mmap...
[03/10 19:04:06 lb.data.data_utils.indexed_dataset]: creating memory view of numpy buffer...
[03/10 19:04:06 lb.data.data_utils.indexed_dataset]: Finished creating indexed dataset in 0.073104 seconds
[03/10 19:04:06 lb.data.data_utils.indexed_dataset]: indexed dataset stats:
[03/10 19:04:06 lb.data.data_utils.indexed_dataset]: number of documents: 50000
[03/10 19:04:06 lb.data.data_utils.indexed_dataset]: number of sentences: 1249934
[03/10 19:04:06 lb.data.data_utils.dataset_utils]:  > loading indexed mapping from ./data_test/bert_data/loss_compara_content_sentence_bert_indexmap_28160mns_509msl_0.10ssp_1234s.npy
[03/10 19:04:06 lb.data.data_utils.dataset_utils]:     loaded indexed file in 0.003 seconds
[03/10 19:04:06 lb.data.data_utils.dataset_utils]:     total number of samples: 113036
[03/10 19:04:06 lb.data.data_utils.dataset_utils]:  > loading indexed mapping from ./data_test/bert_data/loss_compara_content_sentence_bert_indexmap_32mns_509msl_0.10ssp_1234s.npy
[03/10 19:04:06 lb.data.data_utils.dataset_utils]:     loaded indexed file in 0.000 seconds
[03/10 19:04:06 lb.data.data_utils.dataset_utils]:     total number of samples: 5884
[03/10 19:04:06 lb.data.data_utils.dataset_utils]:  > loading indexed mapping from ./data_test/bert_data/loss_compara_content_sentence_bert_indexmap_32mns_509msl_0.10ssp_1234s.npy
[03/10 19:04:06 lb.data.data_utils.dataset_utils]:     loaded indexed file in 0.000 seconds
[03/10 19:04:06 lb.data.data_utils.dataset_utils]:     total number of samples: 5884
[03/10 19:04:07 lb.engine.default]: Auto-scaling the config to train.train_iter=220, train.warmup_iter=0
[03/10 19:04:07 libai]: > Start building model...
[03/10 19:04:09 lb.engine.default]: Model:
BertForPreTraining(
  (bert): BertModel(
    (embeddings): BertEmbeddings(
      (vocab_embeddings): VocabEmbedding(num_embeddings=21248, embedding_dim=1024)
      (position_embeddings): Embedding(num_embeddings=512, embedding_dim=1024)
      (tokentype_embeddings): Embedding(num_embeddings=2, embedding_dim=1024)
      (embedding_dropout): Dropout(p=0.1, inplace=False)
    )
    (extended_attn_mask): BertExtendedAttnMask()
    (encoders): ModuleList(
      (0): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (1): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (2): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (3): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (4): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (5): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (6): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (7): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (8): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (9): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (10): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (11): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (12): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (13): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (14): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (15): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (16): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (17): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (18): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (19): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (20): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (21): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (22): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (23): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (24): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (25): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (26): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (27): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (28): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (29): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (30): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (31): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (32): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (33): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (34): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (35): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (36): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (37): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (38): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (39): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (40): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (41): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (42): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (43): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (44): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (45): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (46): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
      (47): TransformerLayer(
        (drop_path): Identity()
        (input_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (self_attention): MultiheadAttention(
          hidden_size=1024, num_heads=16, is_cross_attention=False
          (dropout): Dropout(p=0.1, inplace=False)
          (query_key_value): Linear1D(in_features=1024, out_features=3072, bias=True, parallel=col)
          (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=row)
        )
        (post_attention_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
        (mlp): MLP(
          bias_gelu_fusion=True, bias_dropout_fusion=True, dropout=0.1
          (dense_h_to_4h): Linear1D(in_features=1024, out_features=4096, bias=True, parallel=col)
          (dense_4h_to_h): Linear1D(in_features=4096, out_features=1024, bias=True, parallel=row)
        )
      )
    )
    (final_layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
    (pooler): BertPooler(
      (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=col)
      (activation_func): Tanh()
    )
  )
  (cls_head): BertPreTrainingHeads(
    (predictions): BertLMPredictionHead(
      (dense): Linear1D(in_features=1024, out_features=1024, bias=True, parallel=data)
      (activation_func): GELU()
      (layernorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
    )
    (seq_relationship): Linear1D(in_features=1024, out_features=2, bias=True, parallel=data)
    (lm_logits): LMLogits()
    (loss_func): BertLoss(
      (lm_loss): ParallelCrossEntropyLoss()
    )
  )
)
[03/10 19:04:09 libai]: >>> done with building model. Building time: 1.677 seconds
WARNING [03/10 19:04:09 lb.scheduler.lr_scheduler]: warmup iters equals to zero, return CosineLR
[03/10 19:04:09 lb.engine.trainer]: Starting training from iteration 0
[03/10 19:04:09 lb.models.utils.graph_base]: Start compiling the train graph which may take some time. Please wait for a moment ...
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2023/03/10 19:09:11.990, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 88 %, 50 %, 12288 MiB, 5459 MiB, 6594 MiB
2023/03/10 19:09:11.991, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 4 %, 4 %, 12288 MiB, 5820 MiB, 6233 MiB
2023/03/10 19:09:11.992, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 3 %, 3 %, 12288 MiB, 5824 MiB, 6229 MiB
2023/03/10 19:09:11.993, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 8125 MiB, 3928 MiB
2023/03/10 19:09:11.994, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 12051 MiB, 2 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2023/03/10 19:09:11.995, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 12051 MiB, 2 MiB
2023/03/10 19:09:11.995, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 88 %, 50 %, 12288 MiB, 5459 MiB, 6594 MiB
2023/03/10 19:09:11.995, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 88 %, 50 %, 12288 MiB, 5459 MiB, 6594 MiB
2023/03/10 19:09:11.997, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 12051 MiB, 2 MiB
2023/03/10 19:09:11.997, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 4 %, 4 %, 12288 MiB, 5820 MiB, 6233 MiB
2023/03/10 19:09:11.998, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 4 %, 4 %, 12288 MiB, 5820 MiB, 6233 MiB
2023/03/10 19:09:11.999, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 12051 MiB, 2 MiB
2023/03/10 19:09:12.000, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 3 %, 3 %, 12288 MiB, 5824 MiB, 6229 MiB
2023/03/10 19:09:12.001, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 3 %, 3 %, 12288 MiB, 5824 MiB, 6229 MiB
2023/03/10 19:09:12.004, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 8125 MiB, 3928 MiB
2023/03/10 19:09:12.004, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 8125 MiB, 3928 MiB
2023/03/10 19:09:12.007, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 12051 MiB, 2 MiB
2023/03/10 19:09:12.008, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 12051 MiB, 2 MiB
2023/03/10 19:09:12.009, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 12051 MiB, 2 MiB
2023/03/10 19:09:12.009, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 12051 MiB, 2 MiB
2023/03/10 19:09:12.011, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 12051 MiB, 2 MiB
2023/03/10 19:09:12.011, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 12051 MiB, 2 MiB
2023/03/10 19:09:12.012, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 12051 MiB, 2 MiB
2023/03/10 19:09:12.013, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 12051 MiB, 2 MiB
timestamp, name, driver_version, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]
2023/03/10 19:09:14.668, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 88 %, 50 %, 12288 MiB, 5459 MiB, 6594 MiB
2023/03/10 19:09:14.669, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 20 %, 12 %, 12288 MiB, 5820 MiB, 6233 MiB
2023/03/10 19:09:14.670, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 5824 MiB, 6229 MiB
2023/03/10 19:09:14.671, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 2 %, 3 %, 12288 MiB, 8125 MiB, 3928 MiB
2023/03/10 19:09:14.672, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 12051 MiB, 2 MiB
2023/03/10 19:09:14.673, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 12051 MiB, 2 MiB
2023/03/10 19:09:14.674, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 12051 MiB, 2 MiB
2023/03/10 19:09:14.675, NVIDIA GeForce RTX 3080 Ti, 515.65.01, 0 %, 0 %, 12288 MiB, 12051 MiB, 2 MiB
[03/10 19:09:17 lb.utils.events]:  eta: 0:05:22  iteration: 99/220  consumed_samples: 12800  total_loss: 7.987  lm_loss: 7.277  sop_loss: 0.7019  time: 2.6758 s/iter  data_time: 0.0105 s/iter total_throughput: 47.84 samples/s lr: 5.82e-05  
[03/10 19:13:46 lb.utils.events]:  eta: 0:00:53  iteration: 199/220  consumed_samples: 25600  total_loss: 7.929  lm_loss: 7.226  sop_loss: 0.6974  time: 2.6815 s/iter  data_time: 0.0096 s/iter total_throughput: 47.73 samples/s lr: 3.21e-06  
[03/10 19:14:39 lb.utils.events]:  eta: 0:00:00  iteration: 219/220  consumed_samples: 28160  total_loss: 7.916  lm_loss: 7.213  sop_loss: 0.6967  time: 2.6820 s/iter  data_time: 0.0099 s/iter total_throughput: 47.73 samples/s lr: 1.01e-06  
[03/10 19:14:39 lb.engine.hooks]: Overall training speed: 218 iterations in 0:09:44 (2.6820 s / it)
[03/10 19:14:39 lb.engine.hooks]: Total training time: 0:09:44 (0:00:00 on hooks)
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
oneflow-version(git_commit)=0.9.1.dev20230309+cu117
oneflow-commit(git_commit)=1ea2bb7
oneflow-libai(git_commit)=50a973dc5de635b8613ad7666c073c763e238850