[2024-06-19 00:13:36,884] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-06-19 00:13:46,857] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect) please install petrel_client Replace train sampler!! petrel_client is not installed. Using PIL to load images. [2024-06-19 00:13:50,410] [INFO] [comm.py:637:init_distributed] cdb=None [2024-06-19 00:13:50,410] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl 06/19/2024 00:13:50 - WARNING - __main__ - Process rank: 0, device: cuda:0, n_gpu: 1distributed training: True, 16-bits training: False 06/19/2024 00:13:50 - INFO - __main__ - Training/evaluation parameters TrainingArguments( _n_gpu=1, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, bf16=True, bf16_full_eval=False, data_seed=None, dataloader_drop_last=False, dataloader_num_workers=4, dataloader_persistent_workers=False, dataloader_pin_memory=True, ddp_backend=None, ddp_broadcast_buffers=None, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=1800, debug=[], deepspeed=/dss/dssmcmlfs01/pn34sa/pn34sa-dss-0000/haowei/InternVL/internvl_chat/zero_stage3_config.json, disable_tqdm=False, dispatch_batches=None, do_eval=False, do_predict=False, do_train=True, eval_accumulation_steps=None, eval_delay=0, eval_steps=None, evaluation_strategy=no, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=None, full_determinism=False, gradient_accumulation_steps=2, gradient_checkpointing=False, gradient_checkpointing_kwargs=None, greater_is_better=None, group_by_length=True, half_precision_backend=auto, hub_always_push=False, hub_model_id=None, hub_private_repo=False, hub_strategy=every_save, hub_token=, ignore_data_skip=False, include_inputs_for_metrics=False, include_num_input_tokens_seen=False, include_tokens_per_second=False, jit_mode_eval=False, label_names=None, label_smoothing_factor=0.0, learning_rate=0.0001, length_column_name=length, load_best_model_at_end=False, local_rank=0, log_level=passive, log_level_replica=warning, log_on_each_node=True, logging_dir=ckpts/baseline2_2_epochs/runs/Jun19_00-13-50_lrz-hgx-a100-003, logging_first_step=False, logging_nan_inf_filter=True, logging_steps=1.0, logging_strategy=steps, lr_scheduler_kwargs={}, lr_scheduler_type=cosine, max_grad_norm=1.0, max_steps=-1, metric_for_best_model=None, mp_parameters=, neftune_noise_alpha=None, no_cuda=False, num_train_epochs=2.0, optim=adamw_torch, optim_args=None, output_dir=ckpts/baseline2_2_epochs/, overwrite_output_dir=True, past_index=-1, per_device_eval_batch_size=8, per_device_train_batch_size=2, prediction_loss_only=False, push_to_hub=False, push_to_hub_model_id=None, push_to_hub_organization=None, push_to_hub_token=, ray_scope=last, remove_unused_columns=False, report_to=[], resume_from_checkpoint=None, run_name=ckpts/baseline2_2_epochs/, save_on_each_node=False, save_only_model=False, save_safetensors=True, save_steps=10000, save_strategy=steps, save_total_limit=3, seed=42, skip_memory_metrics=True, split_batches=False, tf32=None, torch_compile=False, torch_compile_backend=None, torch_compile_mode=None, torchdynamo=None, tpu_metrics_debug=False, tpu_num_cores=None, use_cpu=False, use_ipex=False, use_legacy_prediction_loop=False, use_mps_device=False, warmup_ratio=0.03, warmup_steps=0, weight_decay=0.05, ) 06/19/2024 00:13:50 - INFO - __main__ - Loading Tokenizer: /dss/dssmcmlfs01/pn34sa/pn34sa-dss-0000/haowei/HF_models/InternVL-Chat-V1-5 [INFO|tokenization_utils_base.py:2025] 2024-06-19 00:13:50,503 >> loading file ./tokenizer.model [INFO|tokenization_utils_base.py:2025] 2024-06-19 00:13:50,503 >> loading file added_tokens.json [INFO|tokenization_utils_base.py:2025] 2024-06-19 00:13:50,503 >> loading file special_tokens_map.json [INFO|tokenization_utils_base.py:2025] 2024-06-19 00:13:50,503 >> loading file tokenizer_config.json [INFO|tokenization_utils_base.py:2025] 2024-06-19 00:13:50,503 >> loading file tokenizer.json [WARNING|logging.py:314] 2024-06-19 00:13:50,714 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. 06/19/2024 00:13:50 - INFO - __main__ - Loading InternVLChatModel... [INFO|configuration_utils.py:727] 2024-06-19 00:13:50,861 >> loading configuration file /dss/dssmcmlfs01/pn34sa/pn34sa-dss-0000/haowei/HF_models/InternVL-Chat-V1-5/config.json [INFO|configuration_utils.py:792] 2024-06-19 00:13:50,862 >> Model config InternVLChatConfig { "_commit_hash": null, "_name_or_path": "OpenGVLab/InternVL-Chat-V1-5", "architectures": [ "InternVLChatModel" ], "auto_map": { "AutoConfig": "configuration_internvl_chat.InternVLChatConfig", "AutoModel": "modeling_internvl_chat.InternVLChatModel", "AutoModelForCausalLM": "modeling_internvl_chat.InternVLChatModel" }, "downsample_ratio": 0.5, "dynamic_image_size": true, "force_image_size": 448, "llm_config": { "_name_or_path": "pretrained/internlm2-chat-20b/", "add_cross_attention": false, "architectures": [ "InternLM2ForCausalLM" ], "attn_implementation": "flash_attention_2", "auto_map": { "AutoConfig": "configuration_internlm2.InternLM2Config", "AutoModel": "modeling_internlm2.InternLM2ForCausalLM", "AutoModelForCausalLM": "modeling_internlm2.InternLM2ForCausalLM" }, "bad_words_ids": null, "begin_suppress_tokens": null, "bias": false, "bos_token_id": 1, "chunk_size_feed_forward": 0, "cross_attention_hidden_size": null, "decoder_start_token_id": null, "diversity_penalty": 0.0, "do_sample": false, "early_stopping": false, "encoder_no_repeat_ngram_size": 0, "eos_token_id": 2, "exponential_decay_length_penalty": null, "finetuning_task": null, "forced_bos_token_id": null, "forced_eos_token_id": null, "hidden_act": "silu", "hidden_size": 6144, "id2label": { "0": "LABEL_0", "1": "LABEL_1" }, "initializer_range": 0.02, "intermediate_size": 16384, "is_decoder": false, "is_encoder_decoder": false, "label2id": { "LABEL_0": 0, "LABEL_1": 1 }, "length_penalty": 1.0, "max_length": 20, "max_position_embeddings": 32768, "min_length": 0, "model_type": "internlm2", "no_repeat_ngram_size": 0, "num_attention_heads": 48, "num_beam_groups": 1, "num_beams": 1, "num_hidden_layers": 48, "num_key_value_heads": 8, "num_return_sequences": 1, "output_attentions": false, "output_hidden_states": false, "output_scores": false, "pad_token_id": 2, "prefix": null, "problem_type": null, "pruned_heads": {}, "remove_invalid_values": false, "repetition_penalty": 1.0, "return_dict": true, "return_dict_in_generate": false, "rms_norm_eps": 1e-05, "rope_scaling": { "factor": 3.0, "type": "dynamic" }, "rope_theta": 1000000, "sep_token_id": null, "suppress_tokens": null, "task_specific_params": null, "temperature": 1.0, "tf_legacy_loss": false, "tie_encoder_decoder": false, "tie_word_embeddings": false, "tokenizer_class": null, "top_k": 50, "top_p": 1.0, "torch_dtype": "bfloat16", "torchscript": false, "transformers_version": "4.37.2", "typical_p": 1.0, "use_bfloat16": false, "use_cache": true, "vocab_size": 92553 }, "max_dynamic_patch": 12, "min_dynamic_patch": 1, "model_type": "internvl_chat", "pad2square": false, "ps_version": "v2", "select_layer": -1, "template": "internlm2-chat", "torch_dtype": "bfloat16", "transformers_version": null, "use_backbone_lora": 0, "use_llm_lora": 0, "use_thumbnail": true, "vision_config": { "_name_or_path": "OpenGVLab/InternViT-6B-448px-V1-5", "add_cross_attention": false, "architectures": [ "InternVisionModel" ], "attention_dropout": 0.0, "auto_map": { "AutoConfig": "configuration_intern_vit.InternVisionConfig", "AutoModel": "modeling_intern_vit.InternVisionModel" }, "bad_words_ids": null, "begin_suppress_tokens": null, "bos_token_id": null, "chunk_size_feed_forward": 0, "cross_attention_hidden_size": null, "decoder_start_token_id": null, "diversity_penalty": 0.0, "do_sample": false, "drop_path_rate": 0.4, "dropout": 0.0, "early_stopping": false, "encoder_no_repeat_ngram_size": 0, "eos_token_id": null, "exponential_decay_length_penalty": null, "finetuning_task": null, "forced_bos_token_id": null, "forced_eos_token_id": null, "hidden_act": "gelu", "hidden_size": 3200, "id2label": { "0": "LABEL_0", "1": "LABEL_1" }, "image_size": 448, "initializer_factor": 0.1, "initializer_range": 1e-10, "intermediate_size": 12800, "is_decoder": false, "is_encoder_decoder": false, "label2id": { "LABEL_0": 0, "LABEL_1": 1 }, "layer_norm_eps": 1e-06, "length_penalty": 1.0, "max_length": 20, "min_length": 0, "model_type": "intern_vit_6b", "no_repeat_ngram_size": 0, "norm_type": "rms_norm", "num_attention_heads": 25, "num_beam_groups": 1, "num_beams": 1, "num_channels": 3, "num_hidden_layers": 45, "num_return_sequences": 1, "output_attentions": false, "output_hidden_states": false, "output_scores": false, "pad_token_id": null, "patch_size": 14, "prefix": null, "problem_type": null, "pruned_heads": {}, "qk_normalization": true, "qkv_bias": false, "remove_invalid_values": false, "repetition_penalty": 1.0, "return_dict": true, "return_dict_in_generate": false, "sep_token_id": null, "suppress_tokens": null, "task_specific_params": null, "temperature": 1.0, "tf_legacy_loss": false, "tie_encoder_decoder": false, "tie_word_embeddings": true, "tokenizer_class": null, "top_k": 50, "top_p": 1.0, "torch_dtype": "bfloat16", "torchscript": false, "transformers_version": "4.37.2", "typical_p": 1.0, "use_bfloat16": true, "use_flash_attn": true } } 06/19/2024 00:13:50 - INFO - __main__ - Using flash_attention_2 for InternLM [INFO|modeling_utils.py:3473] 2024-06-19 00:13:50,863 >> loading weights file /dss/dssmcmlfs01/pn34sa/pn34sa-dss-0000/haowei/HF_models/InternVL-Chat-V1-5/model.safetensors.index.json [INFO|modeling_utils.py:1426] 2024-06-19 00:13:50,865 >> Instantiating InternVLChatModel model under default dtype torch.bfloat16. [INFO|modeling_utils.py:3582] 2024-06-19 00:13:50,865 >> Detected DeepSpeed ZeRO-3: activating zero.init() for this model [INFO|configuration_utils.py:826] 2024-06-19 00:13:50,874 >> Generate config GenerationConfig {} [INFO|configuration_utils.py:826] 2024-06-19 00:13:51,745 >> Generate config GenerationConfig { "bos_token_id": 1, "eos_token_id": 2, "pad_token_id": 2 } [2024-06-19 00:13:51,973] [INFO] [partition_parameters.py:343:__exit__] finished initializing model - num_params = 934, num_elems = 25.51B Loading checkpoint shards: 0%| | 0/11 [00:00> All model checkpoint weights were used when initializing InternVLChatModel. [INFO|modeling_utils.py:4358] 2024-06-19 00:14:45,001 >> All the weights of InternVLChatModel were initialized from the model checkpoint at /dss/dssmcmlfs01/pn34sa/pn34sa-dss-0000/haowei/HF_models/InternVL-Chat-V1-5. If your task is similar to the task the model of the checkpoint was trained on, you can already use InternVLChatModel for predictions without further training. [INFO|configuration_utils.py:779] 2024-06-19 00:14:45,010 >> loading configuration file /dss/dssmcmlfs01/pn34sa/pn34sa-dss-0000/haowei/HF_models/InternVL-Chat-V1-5/generation_config.json [INFO|configuration_utils.py:826] 2024-06-19 00:14:45,010 >> Generate config GenerationConfig {} 06/19/2024 00:14:45 - INFO - __main__ - Finished 06/19/2024 00:14:45 - INFO - __main__ - model.config.force_image_size: 448 06/19/2024 00:14:45 - INFO - __main__ - data_args.force_image_size: 448 06/19/2024 00:14:45 - INFO - __main__ - model.config.vision_config.image_size: 448 06/19/2024 00:14:45 - INFO - __main__ - [Dataset] num_image_token: 256 06/19/2024 00:14:45 - INFO - __main__ - [Dataset] dynamic_image_size: True 06/19/2024 00:14:45 - INFO - __main__ - [Dataset] use_thumbnail: True 06/19/2024 00:14:45 - INFO - __main__ - [Dataset] min_dynamic_patch: 1, max_dynamic_patch: 6 06/19/2024 00:14:45 - INFO - __main__ - Formatting inputs...Skip in lazy mode 06/19/2024 00:14:45 - INFO - __main__ - Add dataset:decovqa_0 with length: 400 trainable params: 11,010,048 || all params: 19,872,270,336 || trainable%: 0.0554 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.0.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.1.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.2.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.3.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.4.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.5.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.6.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.7.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.8.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.9.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.10.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.11.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.12.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.13.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.14.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.15.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.16.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.17.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.18.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.19.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.20.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.21.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.22.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.23.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.24.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.25.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.26.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.27.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.28.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.29.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.30.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.31.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.32.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.33.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.34.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.35.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.36.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.37.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.38.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.39.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.40.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.41.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.42.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.43.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.44.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.45.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.46.attention.wqkv.lora_B.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wqkv.lora_A.default.weight 06/19/2024 00:14:45 - INFO - __main__ - language_model.base_model.model.model.layers.47.attention.wqkv.lora_B.default.weight [INFO|trainer.py:571] 2024-06-19 00:14:45,605 >> Using auto half precision backend [2024-06-19 00:14:45,719] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.13.5, git-hash=unknown, git-branch=unknown [2024-06-19 00:14:45,751] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False Using /dss/dsshome1/00/di93zun/.cache/torch_extensions/py310_cu118 as PyTorch extensions root... Detected CUDA files, patching ldflags Emitting ninja build file /dss/dsshome1/00/di93zun/.cache/torch_extensions/py310_cu118/fused_adam/build.ninja... Building extension module fused_adam... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module fused_adam... Time to load fused_adam op: 0.2858915328979492 seconds [2024-06-19 00:14:46,042] [INFO] [logging.py:96:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer [2024-06-19 00:14:46,042] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer [2024-06-19 00:14:46,083] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = FusedAdam [2024-06-19 00:14:46,083] [INFO] [utils.py:56:is_zero_supported_optimizer] Checking ZeRO support for optimizer=FusedAdam type= [2024-06-19 00:14:46,083] [INFO] [logging.py:96:log_dist] [Rank 0] Creating fp16 ZeRO stage 3 optimizer, MiCS is enabled False, Hierarchical params gather False [2024-06-19 00:14:46,083] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 3 optimizer [2024-06-19 00:14:46,193] [INFO] [utils.py:800:see_memory_usage] Stage 3 initialize beginning [2024-06-19 00:14:46,194] [INFO] [utils.py:801:see_memory_usage] MA 48.29 GB Max_MA 50.39 GB CA 49.98 GB Max_CA 52 GB [2024-06-19 00:14:46,194] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 36.03 GB, percent = 3.6% [2024-06-19 00:14:46,201] [INFO] [stage3.py:130:__init__] Reduce bucket size 1000000000 [2024-06-19 00:14:46,201] [INFO] [stage3.py:131:__init__] Prefetch bucket size 1000000000 [2024-06-19 00:14:46,294] [INFO] [utils.py:800:see_memory_usage] DeepSpeedZeRoOffload initialize [begin] [2024-06-19 00:14:46,295] [INFO] [utils.py:801:see_memory_usage] MA 48.29 GB Max_MA 48.29 GB CA 49.98 GB Max_CA 50 GB [2024-06-19 00:14:46,295] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 36.03 GB, percent = 3.6% Parameter Offload: Total persistent parameters: 18539904 in 606 params [2024-06-19 00:14:46,495] [INFO] [utils.py:800:see_memory_usage] DeepSpeedZeRoOffload initialize [end] [2024-06-19 00:14:46,496] [INFO] [utils.py:801:see_memory_usage] MA 48.29 GB Max_MA 48.3 GB CA 49.98 GB Max_CA 50 GB [2024-06-19 00:14:46,496] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 36.03 GB, percent = 3.6% [2024-06-19 00:14:46,598] [INFO] [utils.py:800:see_memory_usage] Before creating fp16 partitions [2024-06-19 00:14:46,599] [INFO] [utils.py:801:see_memory_usage] MA 48.29 GB Max_MA 48.29 GB CA 49.98 GB Max_CA 50 GB [2024-06-19 00:14:46,599] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 36.03 GB, percent = 3.6% [2024-06-19 00:14:46,824] [INFO] [utils.py:800:see_memory_usage] After creating fp16 partitions: 1 [2024-06-19 00:14:46,825] [INFO] [utils.py:801:see_memory_usage] MA 48.29 GB Max_MA 48.29 GB CA 49.96 GB Max_CA 50 GB [2024-06-19 00:14:46,825] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 36.04 GB, percent = 3.6% [2024-06-19 00:14:46,927] [INFO] [utils.py:800:see_memory_usage] Before creating fp32 partitions [2024-06-19 00:14:46,927] [INFO] [utils.py:801:see_memory_usage] MA 48.29 GB Max_MA 48.29 GB CA 49.96 GB Max_CA 50 GB [2024-06-19 00:14:46,928] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 36.04 GB, percent = 3.6% [2024-06-19 00:14:47,030] [INFO] [utils.py:800:see_memory_usage] After creating fp32 partitions [2024-06-19 00:14:47,030] [INFO] [utils.py:801:see_memory_usage] MA 48.34 GB Max_MA 48.36 GB CA 49.96 GB Max_CA 50 GB [2024-06-19 00:14:47,030] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 36.04 GB, percent = 3.6% [2024-06-19 00:14:47,132] [INFO] [utils.py:800:see_memory_usage] Before initializing optimizer states [2024-06-19 00:14:47,133] [INFO] [utils.py:801:see_memory_usage] MA 48.34 GB Max_MA 48.34 GB CA 49.96 GB Max_CA 50 GB [2024-06-19 00:14:47,133] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 36.04 GB, percent = 3.6% [2024-06-19 00:14:47,134] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | init_optimizer_state: 0.07 [2024-06-19 00:14:47,235] [INFO] [utils.py:800:see_memory_usage] After initializing optimizer states [2024-06-19 00:14:47,236] [INFO] [utils.py:801:see_memory_usage] MA 48.34 GB Max_MA 48.38 GB CA 49.96 GB Max_CA 50 GB [2024-06-19 00:14:47,236] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 36.04 GB, percent = 3.6% [2024-06-19 00:14:47,236] [INFO] [stage3.py:486:_setup_for_real_optimizer] optimizer state initialized [2024-06-19 00:14:47,366] [INFO] [utils.py:800:see_memory_usage] After initializing ZeRO optimizer [2024-06-19 00:14:47,367] [INFO] [utils.py:801:see_memory_usage] MA 50.22 GB Max_MA 50.22 GB CA 51.82 GB Max_CA 52 GB [2024-06-19 00:14:47,367] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory: used = 36.04 GB, percent = 3.6% [2024-06-19 00:14:47,367] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw [2024-06-19 00:14:47,367] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed using client callable to create LR scheduler [2024-06-19 00:14:47,367] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed LR Scheduler = [2024-06-19 00:14:47,367] [INFO] [logging.py:96:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0], mom=[[0.9, 0.999]] [2024-06-19 00:14:47,370] [INFO] [config.py:996:print] DeepSpeedEngine configuration: [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] activation_checkpointing_config { "partition_activations": false, "contiguous_memory_optimization": false, "cpu_checkpointing": false, "number_checkpoints": null, "synchronize_checkpoint_boundary": false, "profile": false } [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True} [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] amp_enabled .................. False [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] amp_params ................... False [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] autotuning_config ............ { "enabled": false, "start_step": null, "end_step": null, "metric_path": null, "arg_mappings": null, "metric": "throughput", "model_info": null, "results_dir": "autotuning_results", "exps_dir": "autotuning_exps", "overwrite": true, "fast": true, "start_profile_step": 3, "end_profile_step": 5, "tuner_type": "gridsearch", "tuner_early_stopping": 5, "tuner_num_trials": 50, "model_info_path": null, "mp_size": 1, "max_train_batch_size": null, "min_train_batch_size": 1, "max_train_micro_batch_size_per_gpu": 1.024000e+03, "min_train_micro_batch_size_per_gpu": 1, "num_tuning_micro_batch_sizes": 3 } [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] bfloat16_enabled ............. True [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] bfloat16_immediate_grad_update False [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] checkpoint_parallel_write_pipeline False [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] checkpoint_tag_validation_enabled True [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] checkpoint_tag_validation_fail False [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] comms_config ................. [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] communication_data_type ...... None [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] compile_config ............... enabled=False backend='inductor' kwargs={} [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}} [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] curriculum_enabled_legacy .... False [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] curriculum_params_legacy ..... False [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}} [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] data_efficiency_enabled ...... False [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] dataloader_drop_last ......... False [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] disable_allgather ............ False [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] dump_state ................... False [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] dynamic_loss_scale_args ...... None [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] eigenvalue_enabled ........... False [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] eigenvalue_gas_boundary_resolution 1 [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] eigenvalue_layer_name ........ bert.encoder.layer [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] eigenvalue_layer_num ......... 0 [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] eigenvalue_max_iter .......... 100 [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] eigenvalue_stability ......... 1e-06 [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] eigenvalue_tol ............... 0.01 [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] eigenvalue_verbose ........... False [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] elasticity_enabled ........... False [2024-06-19 00:14:47,371] [INFO] [config.py:1000:print] flops_profiler_config ........ { "enabled": false, "recompute_fwd_factor": 0.0, "profile_step": 1, "module_depth": -1, "top_modules": 1, "detailed": true, "output_file": null } [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] fp16_auto_cast ............... None [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] fp16_enabled ................. False [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] fp16_master_weights_and_gradients False [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] global_rank .................. 0 [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] grad_accum_dtype ............. None [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] gradient_accumulation_steps .. 2 [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] gradient_clipping ............ 1.0 [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] gradient_predivide_factor .... 1.0 [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] graph_harvesting ............. False [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8 [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] initial_dynamic_scale ........ 1 [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] load_universal_checkpoint .... False [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] loss_scale ................... 1.0 [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] memory_breakdown ............. False [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] mics_hierarchial_params_gather False [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] mics_shard_size .............. -1 [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] nebula_config ................ { "enabled": false, "persistent_storage_path": null, "persistent_time_interval": 100, "num_of_version_in_retention": 2, "enable_nebula_load": true, "load_path": null } [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] optimizer_legacy_fusion ...... False [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] optimizer_name ............... adamw [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] optimizer_params ............. {'lr': 0.0001, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.05} [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True} [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] pld_enabled .................. False [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] pld_params ................... False [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] prescale_gradients ........... False [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] scheduler_name ............... None [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] scheduler_params ............. None [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] seq_parallel_communication_data_type torch.float32 [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] sparse_attention ............. None [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] sparse_gradients_enabled ..... False [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] steps_per_print .............. inf [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] train_batch_size ............. 4 [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] train_micro_batch_size_per_gpu 2 [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] use_data_before_expert_parallel_ False [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] use_node_local_storage ....... False [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] wall_clock_breakdown ......... True [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] weight_quantization_config ... None [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] world_size ................... 1 [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] zero_allow_untested_optimizer False [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] zero_config .................. stage=3 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=1000000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=500,000,000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=1000000000 param_persistence_threshold=10000000 model_persistence_threshold=sys.maxsize max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=True stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] zero_enabled ................. True [2024-06-19 00:14:47,372] [INFO] [config.py:1000:print] zero_force_ds_cpu_optimizer .. True [2024-06-19 00:14:47,373] [INFO] [config.py:1000:print] zero_optimization_stage ...... 3 [2024-06-19 00:14:47,373] [INFO] [config.py:986:print_user_config] json = { "zero_optimization": { "stage": 3, "overlap_comm": true, "contiguous_gradients": true, "sub_group_size": 1.000000e+09, "reduce_bucket_size": 1.000000e+09, "stage3_prefetch_bucket_size": 1.000000e+09, "stage3_param_persistence_threshold": 1.000000e+07, "stage3_max_live_parameters": 1.000000e+09, "stage3_max_reuse_distance": 1.000000e+09, "stage3_gather_16bit_weights_on_model_save": true }, "fp16": { "enabled": false, "auto_cast": true, "loss_scale": 0, "initial_scale_power": 32, "loss_scale_window": 1000, "hysteresis": 2, "min_loss_scale": 1 }, "bf16": { "enabled": true }, "optimizer": { "type": "AdamW", "params": { "lr": 0.0001, "betas": [0.9, 0.999], "eps": 1e-08, "weight_decay": 0.05 } }, "gradient_accumulation_steps": 2, "gradient_clipping": 1.0, "steps_per_print": inf, "train_batch_size": 4, "train_micro_batch_size_per_gpu": 2, "wall_clock_breakdown": true } [INFO|trainer.py:1721] 2024-06-19 00:14:47,373 >> ***** Running training ***** [INFO|trainer.py:1722] 2024-06-19 00:14:47,373 >> Num examples = 400 [INFO|trainer.py:1723] 2024-06-19 00:14:47,373 >> Num Epochs = 2 [INFO|trainer.py:1724] 2024-06-19 00:14:47,373 >> Instantaneous batch size per device = 2 [INFO|trainer.py:1727] 2024-06-19 00:14:47,373 >> Total train batch size (w. parallel, distributed & accumulation) = 4 [INFO|trainer.py:1728] 2024-06-19 00:14:47,373 >> Gradient Accumulation steps = 2 [INFO|trainer.py:1729] 2024-06-19 00:14:47,373 >> Total optimization steps = 200 [INFO|trainer.py:1730] 2024-06-19 00:14:47,377 >> Number of trainable parameters = 11,010,048 0%| | 0/200 [00:00) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.6380, device='cuda:0', grad_fn=) [2024-06-19 00:15:17,487] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 4282.10 | bwd_microstep: 2018.50 | bwd_inner_microstep: 1955.77 | bwd_allreduce_microstep: 62.54 | step_microstep: 0.05 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1625, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1041, device='cuda:0', grad_fn=) [2024-06-19 00:15:24,047] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 2.28 [2024-06-19 00:15:24,047] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 4582.24 | bwd_microstep: 1803.76 | bwd_inner_microstep: 1798.22 | bwd_allreduce_microstep: 5.41 | step_microstep: 85.30 [2024-06-19 00:15:24,048] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 8864.35 | bwd: 3822.24 | bwd_inner: 3754.08 | bwd_allreduce: 67.96 | step: 85.36 0%| | 1/200 [00:36<2:01:38, 36.68s/it] {'loss': 1.3711, 'learning_rate': 1.6666666666666667e-05, 'epoch': 0.01} 0%| | 1/200 [00:36<2:01:38, 36.68s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.5165, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.4227, device='cuda:0', grad_fn=) [2024-06-19 00:15:29,437] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3489.92 | bwd_microstep: 1806.40 | bwd_inner_microstep: 1801.48 | bwd_allreduce_microstep: 4.79 | step_microstep: 0.05 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.3058, device='cuda:0', grad_fn=) tensor(0.7006, device='cuda:0', grad_fn=) tensor(1.2453, device='cuda:0', grad_fn=) [2024-06-19 00:15:34,876] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.89 [2024-06-19 00:15:34,877] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3486.47 | bwd_microstep: 1804.84 | bwd_inner_microstep: 1799.21 | bwd_allreduce_microstep: 5.48 | step_microstep: 60.66 [2024-06-19 00:15:34,877] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6976.38 | bwd: 3611.23 | bwd_inner: 3600.76 | bwd_allreduce: 10.28 | step: 60.71 1%| | 2/200 [00:47<1:10:51, 21.47s/it] {'loss': 1.334, 'learning_rate': 3.3333333333333335e-05, 'epoch': 0.02} 1%| | 2/200 [00:47<1:10:51, 21.47s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.4739, device='cuda:0', grad_fn=) tensor(0.7006, device='cuda:0', grad_fn=) tensor(1.3966, device='cuda:0', grad_fn=) [2024-06-19 00:15:40,252] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3484.26 | bwd_microstep: 1799.71 | bwd_inner_microstep: 1794.83 | bwd_allreduce_microstep: 4.77 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.4650, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.3763, device='cuda:0', grad_fn=) [2024-06-19 00:15:45,959] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.86 [2024-06-19 00:15:45,960] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3574.07 | bwd_microstep: 1970.20 | bwd_inner_microstep: 1964.64 | bwd_allreduce_microstep: 5.36 | step_microstep: 60.58 [2024-06-19 00:15:45,960] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7058.32 | bwd: 3769.90 | bwd_inner: 3759.54 | bwd_allreduce: 10.10 | step: 60.66 2%|▏ | 3/200 [00:58<54:55, 16.73s/it] {'loss': 1.3864, 'learning_rate': 5e-05, 'epoch': 0.03} 2%|▏ | 3/200 [00:58<54:55, 16.73s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.8646, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.7363, device='cuda:0', grad_fn=) [2024-06-19 00:15:51,160] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3440.25 | bwd_microstep: 1675.78 | bwd_inner_microstep: 1670.87 | bwd_allreduce_microstep: 4.81 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(2.9886, device='cuda:0', grad_fn=) tensor(0.7006, device='cuda:0', grad_fn=) tensor(2.7598, device='cuda:0', grad_fn=) [2024-06-19 00:15:56,371] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:15:56,371] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3423.53 | bwd_microstep: 1650.38 | bwd_inner_microstep: 1644.69 | bwd_allreduce_microstep: 5.57 | step_microstep: 60.87 [2024-06-19 00:15:56,372] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6863.77 | bwd: 3326.15 | bwd_inner: 3315.57 | bwd_allreduce: 10.38 | step: 60.96 2%|▏ | 4/200 [01:09<46:29, 14.23s/it] {'loss': 2.248, 'learning_rate': 6.666666666666667e-05, 'epoch': 0.04} 2%|▏ | 4/200 [01:09<46:29, 14.23s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(2.9915, device='cuda:0', grad_fn=) tensor(0.8190, device='cuda:0', grad_fn=) tensor(2.7743, device='cuda:0', grad_fn=) [2024-06-19 00:16:01,523] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3420.83 | bwd_microstep: 1649.60 | bwd_inner_microstep: 1644.47 | bwd_allreduce_microstep: 5.01 | step_microstep: 0.09 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1706, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1114, device='cuda:0', grad_fn=) [2024-06-19 00:16:07,201] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.90 [2024-06-19 00:16:07,201] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3568.78 | bwd_microstep: 1950.87 | bwd_inner_microstep: 1945.29 | bwd_allreduce_microstep: 5.40 | step_microstep: 60.92 [2024-06-19 00:16:07,202] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6989.59 | bwd: 3600.46 | bwd_inner: 3589.82 | bwd_allreduce: 10.39 | step: 61.01 2%|▎ | 5/200 [01:19<42:16, 13.01s/it] {'loss': 1.9428, 'learning_rate': 8.333333333333334e-05, 'epoch': 0.05} 2%|▎ | 5/200 [01:19<42:16, 13.01s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.6156, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.5118, device='cuda:0', grad_fn=) [2024-06-19 00:16:12,595] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3492.45 | bwd_microstep: 1808.18 | bwd_inner_microstep: 1803.29 | bwd_allreduce_microstep: 4.78 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.7034, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.5908, device='cuda:0', grad_fn=) [2024-06-19 00:16:17,315] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:16:17,315] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2745.77 | bwd_microstep: 1821.11 | bwd_inner_microstep: 1815.48 | bwd_allreduce_microstep: 5.52 | step_microstep: 60.56 [2024-06-19 00:16:17,316] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6238.20 | bwd: 3629.28 | bwd_inner: 3618.76 | bwd_allreduce: 10.30 | step: 60.64 3%|▎ | 6/200 [01:29<38:52, 12.02s/it] {'loss': 1.5513, 'learning_rate': 0.0001, 'epoch': 0.06} 3%|▎ | 6/200 [01:29<38:52, 12.02s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.4080, device='cuda:0', grad_fn=) tensor(0.7006, device='cuda:0', grad_fn=) tensor(1.3373, device='cuda:0', grad_fn=) [2024-06-19 00:16:22,706] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3491.13 | bwd_microstep: 1808.59 | bwd_inner_microstep: 1803.61 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1280) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1280, 6144]) tensor(1.5046, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.4119, device='cuda:0', grad_fn=) [2024-06-19 00:16:28,064] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.87 [2024-06-19 00:16:28,064] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3267.16 | bwd_microstep: 1925.66 | bwd_inner_microstep: 1919.96 | bwd_allreduce_microstep: 5.57 | step_microstep: 63.14 [2024-06-19 00:16:28,065] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6758.27 | bwd: 3734.25 | bwd_inner: 3723.63 | bwd_allreduce: 10.34 | step: 63.22 4%|▎ | 7/200 [01:40<37:20, 11.61s/it] {'loss': 1.3746, 'learning_rate': 9.999344418328162e-05, 'epoch': 0.07} 4%|▎ | 7/200 [01:40<37:20, 11.61s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(1.2387, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1727, device='cuda:0', grad_fn=) [2024-06-19 00:16:32,709] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2741.56 | bwd_microstep: 1803.91 | bwd_inner_microstep: 1798.93 | bwd_allreduce_microstep: 4.87 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.5202, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.4260, device='cuda:0', grad_fn=) [2024-06-19 00:16:38,300] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.89 [2024-06-19 00:16:38,301] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3539.00 | bwd_microstep: 1894.47 | bwd_inner_microstep: 1888.77 | bwd_allreduce_microstep: 5.53 | step_microstep: 61.08 [2024-06-19 00:16:38,301] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6280.54 | bwd: 3698.37 | bwd_inner: 3687.76 | bwd_allreduce: 10.41 | step: 61.16 4%|▍ | 8/200 [01:50<35:44, 11.17s/it] {'loss': 1.2993, 'learning_rate': 9.997377845227576e-05, 'epoch': 0.08} 4%|▍ | 8/200 [01:50<35:44, 11.17s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.7908, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.6695, device='cuda:0', grad_fn=) [2024-06-19 00:16:43,698] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3490.86 | bwd_microstep: 1810.87 | bwd_inner_microstep: 1805.92 | bwd_allreduce_microstep: 4.77 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(3.2031, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(2.9409, device='cuda:0', grad_fn=) [2024-06-19 00:16:48,911] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:16:48,912] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3424.79 | bwd_microstep: 1651.09 | bwd_inner_microstep: 1645.51 | bwd_allreduce_microstep: 5.47 | step_microstep: 60.69 [2024-06-19 00:16:48,912] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6915.64 | bwd: 3461.96 | bwd_inner: 3451.48 | bwd_allreduce: 10.22 | step: 60.77 4%|▍ | 9/200 [02:01<35:00, 11.00s/it] {'loss': 2.3052, 'learning_rate': 9.994100796397954e-05, 'epoch': 0.09} 4%|▍ | 9/200 [02:01<35:00, 11.00s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9785, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.9385, device='cuda:0', grad_fn=) [2024-06-19 00:16:54,422] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3536.06 | bwd_microstep: 1879.50 | bwd_inner_microstep: 1874.59 | bwd_allreduce_microstep: 4.81 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.6156, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.5118, device='cuda:0', grad_fn=) [2024-06-19 00:17:00,039] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.85 [2024-06-19 00:17:00,040] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3654.97 | bwd_microstep: 1810.44 | bwd_inner_microstep: 1804.74 | bwd_allreduce_microstep: 5.58 | step_microstep: 63.04 [2024-06-19 00:17:00,040] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7191.01 | bwd: 3689.94 | bwd_inner: 3679.34 | bwd_allreduce: 10.39 | step: 63.12 5%|▌ | 10/200 [02:12<34:56, 11.04s/it] {'loss': 1.2252, 'learning_rate': 9.989514131188559e-05, 'epoch': 0.1} 5%|▌ | 10/200 [02:12<34:56, 11.04s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.7703, device='cuda:0', grad_fn=) tensor(0.7006, device='cuda:0', grad_fn=) tensor(0.7634, device='cuda:0', grad_fn=) [2024-06-19 00:17:05,235] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3435.24 | bwd_microstep: 1676.21 | bwd_inner_microstep: 1671.29 | bwd_allreduce_microstep: 4.81 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.6371, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.5312, device='cuda:0', grad_fn=) [2024-06-19 00:17:10,829] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:17:10,829] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3543.84 | bwd_microstep: 1894.85 | bwd_inner_microstep: 1889.47 | bwd_allreduce_microstep: 5.27 | step_microstep: 60.36 [2024-06-19 00:17:10,830] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6979.06 | bwd: 3571.06 | bwd_inner: 3560.78 | bwd_allreduce: 10.09 | step: 60.44 6%|▌ | 11/200 [02:23<34:31, 10.96s/it] {'loss': 1.1473, 'learning_rate': 9.983619052372848e-05, 'epoch': 0.11} 6%|▌ | 11/200 [02:23<34:31, 10.96s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(1.6941, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.5828, device='cuda:0', grad_fn=) [2024-06-19 00:17:15,671] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2929.62 | bwd_microstep: 1816.99 | bwd_inner_microstep: 1812.05 | bwd_allreduce_microstep: 4.77 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.4526, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.3652, device='cuda:0', grad_fn=) [2024-06-19 00:17:21,130] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.83 [2024-06-19 00:17:21,130] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3491.21 | bwd_microstep: 1817.32 | bwd_inner_microstep: 1811.88 | bwd_allreduce_microstep: 5.32 | step_microstep: 60.16 [2024-06-19 00:17:21,131] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6420.82 | bwd: 3634.30 | bwd_inner: 3623.99 | bwd_allreduce: 10.10 | step: 60.24 6%|▌ | 12/200 [02:33<33:42, 10.76s/it] {'loss': 1.474, 'learning_rate': 9.97641710583307e-05, 'epoch': 0.12} 6%|▌ | 12/200 [02:33<33:42, 10.76s/it]warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(1.6724, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.5630, device='cuda:0', grad_fn=) [2024-06-19 00:17:23,902] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 1756.33 | bwd_microstep: 946.63 | bwd_inner_microstep: 941.73 | bwd_allreduce_microstep: 4.79 | step_microstep: 0.09 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1280) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1280, 6144]) tensor(1.6193, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.5152, device='cuda:0', grad_fn=) [2024-06-19 00:17:29,160] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.78 [2024-06-19 00:17:29,160] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3235.61 | bwd_microstep: 1865.80 | bwd_inner_microstep: 1860.22 | bwd_allreduce_microstep: 5.40 | step_microstep: 60.27 [2024-06-19 00:17:29,161] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 4991.93 | bwd: 2812.42 | bwd_inner: 2802.01 | bwd_allreduce: 10.17 | step: 60.36 6%|▋ | 13/200 [02:41<30:57, 9.93s/it] {'loss': 1.5391, 'learning_rate': 9.967910180154889e-05, 'epoch': 0.13} 6%|▋ | 13/200 [02:41<30:57, 9.93s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1064, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0535, device='cuda:0', grad_fn=) [2024-06-19 00:17:34,604] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3507.62 | bwd_microstep: 1838.09 | bwd_inner_microstep: 1833.00 | bwd_allreduce_microstep: 4.95 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0271, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(0.9940, device='cuda:0', grad_fn=) [2024-06-19 00:17:39,166] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.85 [2024-06-19 00:17:39,167] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.14 | bwd_microstep: 1596.75 | bwd_inner_microstep: 1591.01 | bwd_allreduce_microstep: 5.62 | step_microstep: 62.69 [2024-06-19 00:17:39,167] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6328.74 | bwd: 3434.84 | bwd_inner: 3424.06 | bwd_allreduce: 10.58 | step: 62.76 7%|▋ | 14/200 [02:51<30:51, 9.96s/it] {'loss': 1.0238, 'learning_rate': 9.958100506132127e-05, 'epoch': 0.14} 7%|▋ | 14/200 [02:51<30:51, 9.96s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(0.1317, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.1767, device='cuda:0', grad_fn=) [2024-06-19 00:17:43,488] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2640.83 | bwd_microstep: 1596.01 | bwd_inner_microstep: 1591.03 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1024) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1024, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1490, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0919, device='cuda:0', grad_fn=) [2024-06-19 00:17:48,448] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.86 [2024-06-19 00:17:48,449] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3035.25 | bwd_microstep: 1772.40 | bwd_inner_microstep: 1766.84 | bwd_allreduce_microstep: 5.45 | step_microstep: 60.27 [2024-06-19 00:17:48,449] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 5676.06 | bwd: 3368.40 | bwd_inner: 3357.92 | bwd_allreduce: 10.24 | step: 60.35 8%|▊ | 15/200 [03:01<30:04, 9.75s/it] {'loss': 0.6343, 'learning_rate': 9.946990656181781e-05, 'epoch': 0.15} 8%|▊ | 15/200 [03:01<30:04, 9.75s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.6347, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.5291, device='cuda:0', grad_fn=) [2024-06-19 00:17:53,989] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3543.59 | bwd_microstep: 1897.04 | bwd_inner_microstep: 1892.08 | bwd_allreduce_microstep: 4.79 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2925, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.2210, device='cuda:0', grad_fn=) [2024-06-19 00:17:59,581] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.85 [2024-06-19 00:17:59,582] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3542.80 | bwd_microstep: 1893.12 | bwd_inner_microstep: 1887.61 | bwd_allreduce_microstep: 5.40 | step_microstep: 60.33 [2024-06-19 00:17:59,582] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7086.38 | bwd: 3790.16 | bwd_inner: 3779.74 | bwd_allreduce: 10.17 | step: 60.41 8%|▊ | 16/200 [03:12<31:10, 10.17s/it] {'loss': 1.375, 'learning_rate': 9.934583543669453e-05, 'epoch': 0.16} 8%|▊ | 16/200 [03:12<31:10, 10.17s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.7402, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.7247, device='cuda:0', grad_fn=) [2024-06-19 00:18:04,777] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3433.02 | bwd_microstep: 1677.56 | bwd_inner_microstep: 1672.67 | bwd_allreduce_microstep: 4.78 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2800, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.2102, device='cuda:0', grad_fn=) [2024-06-19 00:18:09,462] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:18:09,462] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2743.65 | bwd_microstep: 1792.72 | bwd_inner_microstep: 1787.21 | bwd_allreduce_microstep: 5.34 | step_microstep: 60.46 [2024-06-19 00:18:09,463] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6176.65 | bwd: 3470.28 | bwd_inner: 3459.93 | bwd_allreduce: 10.13 | step: 60.54 8%|▊ | 17/200 [03:22<30:44, 10.08s/it] {'loss': 0.9674, 'learning_rate': 9.920882422145372e-05, 'epoch': 0.17} 8%|▊ | 17/200 [03:22<30:44, 10.08s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0226, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.9788, device='cuda:0', grad_fn=) [2024-06-19 00:18:14,622] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3426.80 | bwd_microstep: 1651.03 | bwd_inner_microstep: 1645.98 | bwd_allreduce_microstep: 4.92 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.2688, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(0.3116, device='cuda:0', grad_fn=) [2024-06-19 00:18:19,877] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.79 [2024-06-19 00:18:19,878] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3437.73 | bwd_microstep: 1678.17 | bwd_inner_microstep: 1672.77 | bwd_allreduce_microstep: 5.29 | step_microstep: 60.32 [2024-06-19 00:18:19,878] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6864.51 | bwd: 3329.20 | bwd_inner: 3318.79 | bwd_allreduce: 10.22 | step: 60.40 9%|▉ | 18/200 [03:32<30:53, 10.18s/it] {'loss': 0.6452, 'learning_rate': 9.905890884491195e-05, 'epoch': 0.18} 9%|▉ | 18/200 [03:32<30:53, 10.18s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.8062, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.7837, device='cuda:0', grad_fn=) [2024-06-19 00:18:25,272] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3495.11 | bwd_microstep: 1807.09 | bwd_inner_microstep: 1801.96 | bwd_allreduce_microstep: 5.01 | step_microstep: 0.09 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.2288, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.2638, device='cuda:0', grad_fn=) [2024-06-19 00:18:30,535] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:18:30,536] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3439.03 | bwd_microstep: 1682.11 | bwd_inner_microstep: 1676.67 | bwd_allreduce_microstep: 5.27 | step_microstep: 60.89 [2024-06-19 00:18:30,536] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6934.12 | bwd: 3489.19 | bwd_inner: 3478.69 | bwd_allreduce: 10.29 | step: 60.98 10%|▉ | 19/200 [03:43<31:08, 10.32s/it] {'loss': 0.5237, 'learning_rate': 9.889612861977853e-05, 'epoch': 0.19} 10%|▉ | 19/200 [03:43<31:08, 10.32s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.5880, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(1.4989, device='cuda:0', grad_fn=) [2024-06-19 00:18:36,045] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3536.63 | bwd_microstep: 1878.52 | bwd_inner_microstep: 1873.51 | bwd_allreduce_microstep: 4.84 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1554, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0977, device='cuda:0', grad_fn=) [2024-06-19 00:18:41,509] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:18:41,510] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3496.30 | bwd_microstep: 1817.76 | bwd_inner_microstep: 1812.31 | bwd_allreduce_microstep: 5.34 | step_microstep: 60.41 [2024-06-19 00:18:41,510] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7032.92 | bwd: 3696.27 | bwd_inner: 3685.87 | bwd_allreduce: 10.20 | step: 60.49 10%|█ | 20/200 [03:54<31:33, 10.52s/it] {'loss': 1.2983, 'learning_rate': 9.872052623234632e-05, 'epoch': 0.2} 10%|█ | 20/200 [03:54<31:33, 10.52s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.3011, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.2292, device='cuda:0', grad_fn=) [2024-06-19 00:18:46,897] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3491.66 | bwd_microstep: 1804.82 | bwd_inner_microstep: 1799.95 | bwd_allreduce_microstep: 4.77 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(0.5413, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(0.5571, device='cuda:0', grad_fn=) [2024-06-19 00:18:51,232] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:18:51,233] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2627.94 | bwd_microstep: 1569.83 | bwd_inner_microstep: 1564.29 | bwd_allreduce_microstep: 5.43 | step_microstep: 60.74 [2024-06-19 00:18:51,233] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6119.58 | bwd: 3374.65 | bwd_inner: 3364.25 | bwd_allreduce: 10.21 | step: 60.82 10%|█ | 21/200 [04:03<30:40, 10.28s/it] {'loss': 0.8931, 'learning_rate': 9.853214773129796e-05, 'epoch': 0.21} 10%|█ | 21/200 [04:03<30:40, 10.28s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1694, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1103, device='cuda:0', grad_fn=) [2024-06-19 00:18:56,770] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3541.56 | bwd_microstep: 1895.30 | bwd_inner_microstep: 1890.43 | bwd_allreduce_microstep: 4.76 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1709, device='cuda:0', grad_fn=) tensor(0.7006, device='cuda:0', grad_fn=) tensor(1.1239, device='cuda:0', grad_fn=) [2024-06-19 00:19:02,216] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:19:02,216] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3493.62 | bwd_microstep: 1805.90 | bwd_inner_microstep: 1800.45 | bwd_allreduce_microstep: 5.34 | step_microstep: 60.64 [2024-06-19 00:19:02,217] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7035.16 | bwd: 3701.19 | bwd_inner: 3690.89 | bwd_allreduce: 10.12 | step: 60.72 11%|█ | 22/200 [04:14<31:07, 10.49s/it] {'loss': 1.1171, 'learning_rate': 9.833104251563056e-05, 'epoch': 0.22} 11%|█ | 22/200 [04:14<31:07, 10.49s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2980, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.2260, device='cuda:0', grad_fn=) [2024-06-19 00:19:07,721] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3537.15 | bwd_microstep: 1874.02 | bwd_inner_microstep: 1869.07 | bwd_allreduce_microstep: 4.78 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.5416, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.4453, device='cuda:0', grad_fn=) [2024-06-19 00:19:13,302] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:19:13,302] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3540.94 | bwd_microstep: 1886.75 | bwd_inner_microstep: 1881.40 | bwd_allreduce_microstep: 5.23 | step_microstep: 60.29 [2024-06-19 00:19:13,303] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7078.08 | bwd: 3760.77 | bwd_inner: 3750.52 | bwd_allreduce: 10.02 | step: 60.37 12%|█▏ | 23/200 [04:25<31:28, 10.67s/it] {'loss': 1.3356, 'learning_rate': 9.811726332170153e-05, 'epoch': 0.23} 12%|█▏ | 23/200 [04:25<31:28, 10.67s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.2847, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.3147, device='cuda:0', grad_fn=) [2024-06-19 00:19:18,503] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3440.74 | bwd_microstep: 1676.71 | bwd_inner_microstep: 1671.76 | bwd_allreduce_microstep: 4.83 | step_microstep: 0.09 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.6995, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.5874, device='cuda:0', grad_fn=) [2024-06-19 00:19:22,888] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.85 [2024-06-19 00:19:22,889] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2640.42 | bwd_microstep: 1599.24 | bwd_inner_microstep: 1593.93 | bwd_allreduce_microstep: 5.19 | step_microstep: 60.72 [2024-06-19 00:19:22,889] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6081.17 | bwd: 3275.94 | bwd_inner: 3265.72 | bwd_allreduce: 10.02 | step: 60.81 12%|█▏ | 24/200 [04:35<30:20, 10.34s/it] {'loss': 0.9511, 'learning_rate': 9.789086620939936e-05, 'epoch': 0.24} 12%|█▏ | 24/200 [04:35<30:20, 10.34s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9554, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.9180, device='cuda:0', grad_fn=) [2024-06-19 00:19:28,283] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3493.97 | bwd_microstep: 1806.82 | bwd_inner_microstep: 1801.87 | bwd_allreduce_microstep: 4.78 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2596, device='cuda:0', grad_fn=) tensor(0.7006, device='cuda:0', grad_fn=) tensor(1.2037, device='cuda:0', grad_fn=) [2024-06-19 00:19:33,733] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:19:33,733] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3492.83 | bwd_microstep: 1808.88 | bwd_inner_microstep: 1803.54 | bwd_allreduce_microstep: 5.22 | step_microstep: 60.46 [2024-06-19 00:19:33,734] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6986.78 | bwd: 3615.69 | bwd_inner: 3605.46 | bwd_allreduce: 10.01 | step: 60.54 12%|█▎ | 25/200 [04:46<30:36, 10.49s/it] {'loss': 1.0608, 'learning_rate': 9.765191054744305e-05, 'epoch': 0.25} 12%|█▎ | 25/200 [04:46<30:36, 10.49s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.8159, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(0.8043, device='cuda:0', grad_fn=) [2024-06-19 00:19:39,124] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3492.66 | bwd_microstep: 1806.44 | bwd_inner_microstep: 1801.57 | bwd_allreduce_microstep: 4.75 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2947, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.2230, device='cuda:0', grad_fn=) [2024-06-19 00:19:44,737] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:19:44,737] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3549.58 | bwd_microstep: 1905.05 | bwd_inner_microstep: 1899.67 | bwd_allreduce_microstep: 5.28 | step_microstep: 60.28 [2024-06-19 00:19:44,738] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7042.22 | bwd: 3711.48 | bwd_inner: 3701.26 | bwd_allreduce: 10.03 | step: 60.35 13%|█▎ | 26/200 [04:57<30:52, 10.65s/it] {'loss': 1.0137, 'learning_rate': 9.740045899781352e-05, 'epoch': 0.26} 13%|█▎ | 26/200 [04:57<30:52, 10.65s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1450, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.0886, device='cuda:0', grad_fn=) [2024-06-19 00:19:50,295] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3549.77 | bwd_microstep: 1908.72 | bwd_inner_microstep: 1903.92 | bwd_allreduce_microstep: 4.68 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0369, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.9917, device='cuda:0', grad_fn=) [2024-06-19 00:19:55,652] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:19:55,653] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3473.27 | bwd_microstep: 1737.71 | bwd_inner_microstep: 1732.32 | bwd_allreduce_microstep: 5.28 | step_microstep: 60.53 [2024-06-19 00:19:55,653] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7023.02 | bwd: 3646.42 | bwd_inner: 3636.25 | bwd_allreduce: 9.97 | step: 60.61 14%|█▎ | 27/200 [05:08<30:55, 10.73s/it] {'loss': 1.0402, 'learning_rate': 9.713657749932172e-05, 'epoch': 0.27} 14%|█▎ | 27/200 [05:08<30:55, 10.73s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1075, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0546, device='cuda:0', grad_fn=) [2024-06-19 00:20:01,057] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3500.66 | bwd_microstep: 1810.44 | bwd_inner_microstep: 1805.64 | bwd_allreduce_microstep: 4.70 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0720, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.1229, device='cuda:0', grad_fn=) [2024-06-19 00:20:06,315] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:20:06,316] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3436.39 | bwd_microstep: 1681.18 | bwd_inner_microstep: 1675.81 | bwd_allreduce_microstep: 5.26 | step_microstep: 60.54 [2024-06-19 00:20:06,316] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6937.03 | bwd: 3491.62 | bwd_inner: 3481.46 | bwd_allreduce: 9.97 | step: 60.62 14%|█▍ | 28/200 [05:18<30:41, 10.71s/it] {'loss': 0.5888, 'learning_rate': 9.686033525031719e-05, 'epoch': 0.28} 14%|█▍ | 28/200 [05:18<30:41, 10.71s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9799, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(0.9516, device='cuda:0', grad_fn=) [2024-06-19 00:20:11,819] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3538.99 | bwd_microstep: 1871.56 | bwd_inner_microstep: 1866.56 | bwd_allreduce_microstep: 4.89 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2693, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.2002, device='cuda:0', grad_fn=) [2024-06-19 00:20:17,417] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.90 [2024-06-19 00:20:17,417] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3546.19 | bwd_microstep: 1894.64 | bwd_inner_microstep: 1889.18 | bwd_allreduce_microstep: 5.34 | step_microstep: 61.12 [2024-06-19 00:20:17,418] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7085.16 | bwd: 3766.19 | bwd_inner: 3755.76 | bwd_allreduce: 10.23 | step: 61.20 14%|█▍ | 29/200 [05:30<30:51, 10.83s/it] {'loss': 1.0759, 'learning_rate': 9.657180469054213e-05, 'epoch': 0.29} 14%|█▍ | 29/200 [05:30<30:51, 10.83s/it]warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1725, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1131, device='cuda:0', grad_fn=) [2024-06-19 00:20:21,747] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2642.49 | bwd_microstep: 1597.79 | bwd_inner_microstep: 1592.98 | bwd_allreduce_microstep: 4.70 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1024) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1024, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2065, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1437, device='cuda:0', grad_fn=) [2024-06-19 00:20:26,830] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:20:26,831] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3083.72 | bwd_microstep: 1842.88 | bwd_inner_microstep: 1837.54 | bwd_allreduce_microstep: 5.23 | step_microstep: 60.15 [2024-06-19 00:20:26,831] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 5726.19 | bwd: 3440.66 | bwd_inner: 3430.53 | bwd_allreduce: 9.93 | step: 60.23 15%|█▌ | 30/200 [05:39<29:28, 10.40s/it] {'loss': 1.1284, 'learning_rate': 9.627106148213522e-05, 'epoch': 0.3} 15%|█▌ | 30/200 [05:39<29:28, 10.40s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0805, device='cuda:0', grad_fn=) tensor(0.8231, device='cuda:0', grad_fn=) tensor(0.1548, device='cuda:0', grad_fn=) [2024-06-19 00:20:31,992] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3427.24 | bwd_microstep: 1652.36 | bwd_inner_microstep: 1647.47 | bwd_allreduce_microstep: 4.77 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(0.9904, device='cuda:0', grad_fn=) tensor(0.7006, device='cuda:0', grad_fn=) tensor(0.9614, device='cuda:0', grad_fn=) [2024-06-19 00:20:36,548] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:20:36,548] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2689.09 | bwd_microstep: 1721.01 | bwd_inner_microstep: 1715.65 | bwd_allreduce_microstep: 5.24 | step_microstep: 60.53 [2024-06-19 00:20:36,549] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6116.32 | bwd: 3373.36 | bwd_inner: 3363.15 | bwd_allreduce: 10.02 | step: 60.61 16%|█▌ | 31/200 [05:49<28:43, 10.20s/it] {'loss': 0.5581, 'learning_rate': 9.595818448979061e-05, 'epoch': 0.31} 16%|█▌ | 31/200 [05:49<28:43, 10.20s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0002, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0583, device='cuda:0', grad_fn=) [2024-06-19 00:20:41,754] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3438.05 | bwd_microstep: 1682.21 | bwd_inner_microstep: 1677.19 | bwd_allreduce_microstep: 4.84 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.1439, device='cuda:0', grad_fn=) tensor(0.7039, device='cuda:0', grad_fn=) tensor(0.1999, device='cuda:0', grad_fn=) [2024-06-19 00:20:47,005] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.79 [2024-06-19 00:20:47,006] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3434.73 | bwd_microstep: 1676.97 | bwd_inner_microstep: 1671.47 | bwd_allreduce_microstep: 5.33 | step_microstep: 60.01 [2024-06-19 00:20:47,006] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6872.76 | bwd: 3359.18 | bwd_inner: 3348.75 | bwd_allreduce: 10.16 | step: 60.09 16%|█▌ | 32/200 [05:59<28:46, 10.28s/it] {'loss': 0.1291, 'learning_rate': 9.563325576007701e-05, 'epoch': 0.32} 16%|█▌ | 32/200 [05:59<28:46, 10.28s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(0.1120, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.1587, device='cuda:0', grad_fn=) [2024-06-19 00:20:51,329] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2641.72 | bwd_microstep: 1596.72 | bwd_inner_microstep: 1591.73 | bwd_allreduce_microstep: 4.77 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0857, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0350, device='cuda:0', grad_fn=) [2024-06-19 00:20:56,805] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:20:56,806] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3498.64 | bwd_microstep: 1824.76 | bwd_inner_microstep: 1819.31 | bwd_allreduce_microstep: 5.34 | step_microstep: 59.77 [2024-06-19 00:20:56,806] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6140.34 | bwd: 3421.47 | bwd_inner: 3411.14 | bwd_allreduce: 10.12 | step: 59.85 16%|█▋ | 33/200 [06:09<28:12, 10.13s/it] {'loss': 0.5968, 'learning_rate': 9.529636049992234e-05, 'epoch': 0.33} 16%|█▋ | 33/200 [06:09<28:12, 10.13s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0258, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.0811, device='cuda:0', grad_fn=) [2024-06-19 00:21:02,007] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3438.80 | bwd_microstep: 1678.28 | bwd_inner_microstep: 1673.15 | bwd_allreduce_microstep: 4.95 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0017, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0596, device='cuda:0', grad_fn=) [2024-06-19 00:21:07,273] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:21:07,274] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3441.72 | bwd_microstep: 1683.07 | bwd_inner_microstep: 1677.53 | bwd_allreduce_microstep: 5.36 | step_microstep: 60.76 [2024-06-19 00:21:07,274] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6880.50 | bwd: 3361.34 | bwd_inner: 3350.77 | bwd_allreduce: 10.32 | step: 60.84 17%|█▋ | 34/200 [06:19<28:18, 10.23s/it] {'loss': 0.0704, 'learning_rate': 9.494758705426978e-05, 'epoch': 0.34} 17%|█▋ | 34/200 [06:19<28:18, 10.23s/it]warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0756, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(0.1380, device='cuda:0', grad_fn=) [2024-06-19 00:21:11,762] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2821.02 | bwd_microstep: 1586.36 | bwd_inner_microstep: 1581.32 | bwd_allreduce_microstep: 4.86 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0423, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.9959, device='cuda:0', grad_fn=) [2024-06-19 00:21:17,319] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:21:17,320] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3533.29 | bwd_microstep: 1873.80 | bwd_inner_microstep: 1868.39 | bwd_allreduce_microstep: 5.31 | step_microstep: 60.12 [2024-06-19 00:21:17,320] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6354.29 | bwd: 3460.16 | bwd_inner: 3449.76 | bwd_allreduce: 10.16 | step: 60.20 18%|█▊ | 35/200 [06:29<27:59, 10.18s/it] {'loss': 0.567, 'learning_rate': 9.458702688291073e-05, 'epoch': 0.35} 18%|█▊ | 35/200 [06:29<27:59, 10.18s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(0.7911, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.7701, device='cuda:0', grad_fn=) [2024-06-19 00:21:21,840] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.34 | bwd_microstep: 1603.01 | bwd_inner_microstep: 1598.09 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0741, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.1249, device='cuda:0', grad_fn=) [2024-06-19 00:21:26,210] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:21:26,211] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2636.25 | bwd_microstep: 1594.13 | bwd_inner_microstep: 1588.62 | bwd_allreduce_microstep: 5.33 | step_microstep: 60.25 [2024-06-19 00:21:26,211] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 5464.57 | bwd: 3197.13 | bwd_inner: 3186.76 | bwd_allreduce: 10.12 | step: 60.33 18%|█▊ | 36/200 [06:38<26:45, 9.79s/it] {'loss': 0.4475, 'learning_rate': 9.421477453650118e-05, 'epoch': 0.36} 18%|█▊ | 36/200 [06:38<26:45, 9.79s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.8091, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(1.6979, device='cuda:0', grad_fn=) [2024-06-19 00:21:31,593] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3490.59 | bwd_microstep: 1800.94 | bwd_inner_microstep: 1796.00 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2525, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1851, device='cuda:0', grad_fn=) [2024-06-19 00:21:37,199] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:21:37,199] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3547.53 | bwd_microstep: 1900.86 | bwd_inner_microstep: 1895.47 | bwd_allreduce_microstep: 5.29 | step_microstep: 60.16 [2024-06-19 00:21:37,200] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7038.10 | bwd: 3701.80 | bwd_inner: 3691.52 | bwd_allreduce: 10.10 | step: 60.24 18%|█▊ | 37/200 [06:49<27:34, 10.15s/it] {'loss': 1.4415, 'learning_rate': 9.38309276317674e-05, 'epoch': 0.37} 18%|█▊ | 37/200 [06:49<27:34, 10.15s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0126, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(0.0813, device='cuda:0', grad_fn=) [2024-06-19 00:21:42,396] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3435.73 | bwd_microstep: 1677.41 | bwd_inner_microstep: 1672.46 | bwd_allreduce_microstep: 4.78 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1280) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1280, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.5660, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.4672, device='cuda:0', grad_fn=) [2024-06-19 00:21:47,653] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:21:47,653] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3238.46 | bwd_microstep: 1861.19 | bwd_inner_microstep: 1855.81 | bwd_allreduce_microstep: 5.27 | step_microstep: 60.19 [2024-06-19 00:21:47,653] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6674.18 | bwd: 3538.59 | bwd_inner: 3528.32 | bwd_allreduce: 10.04 | step: 60.28 19%|█▉ | 38/200 [07:00<27:39, 10.24s/it] {'loss': 0.7743, 'learning_rate': 9.343558682590756e-05, 'epoch': 0.38} 19%|█▉ | 38/200 [07:00<27:39, 10.24s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2156, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1519, device='cuda:0', grad_fn=) [2024-06-19 00:21:53,184] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3542.84 | bwd_microstep: 1889.60 | bwd_inner_microstep: 1884.55 | bwd_allreduce_microstep: 4.94 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(0.0486, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(0.1137, device='cuda:0', grad_fn=) [2024-06-19 00:21:57,558] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.86 [2024-06-19 00:21:57,559] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2639.43 | bwd_microstep: 1595.56 | bwd_inner_microstep: 1590.02 | bwd_allreduce_microstep: 5.36 | step_microstep: 60.68 [2024-06-19 00:21:57,559] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6182.25 | bwd: 3485.15 | bwd_inner: 3474.63 | bwd_allreduce: 10.28 | step: 60.77 20%|█▉ | 39/200 [07:10<27:12, 10.14s/it] {'loss': 0.6328, 'learning_rate': 9.302885579019627e-05, 'epoch': 0.39} 20%|█▉ | 39/200 [07:10<27:12, 10.14s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.3874, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(0.4183, device='cuda:0', grad_fn=) [2024-06-19 00:22:02,755] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3435.35 | bwd_microstep: 1676.70 | bwd_inner_microstep: 1671.82 | bwd_allreduce_microstep: 4.81 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0206, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0767, device='cuda:0', grad_fn=) [2024-06-19 00:22:08,011] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.95 [2024-06-19 00:22:08,012] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3438.65 | bwd_microstep: 1678.20 | bwd_inner_microstep: 1672.65 | bwd_allreduce_microstep: 5.37 | step_microstep: 60.39 [2024-06-19 00:22:08,012] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6873.98 | bwd: 3354.90 | bwd_inner: 3344.50 | bwd_allreduce: 10.17 | step: 60.47 20%|██ | 40/200 [07:20<27:17, 10.23s/it] {'loss': 0.2475, 'learning_rate': 9.261084118279847e-05, 'epoch': 0.4} 20%|██ | 40/200 [07:20<27:17, 10.23s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1579, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0999, device='cuda:0', grad_fn=) [2024-06-19 00:22:13,522] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3537.04 | bwd_microstep: 1877.06 | bwd_inner_microstep: 1872.12 | bwd_allreduce_microstep: 4.84 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(1.4788, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.3888, device='cuda:0', grad_fn=) [2024-06-19 00:22:18,113] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:22:18,114] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2702.33 | bwd_microstep: 1735.82 | bwd_inner_microstep: 1730.29 | bwd_allreduce_microstep: 5.36 | step_microstep: 60.43 [2024-06-19 00:22:18,114] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6239.36 | bwd: 3612.87 | bwd_inner: 3602.46 | bwd_allreduce: 10.18 | step: 60.50 20%|██ | 41/200 [07:30<27:00, 10.19s/it] {'loss': 1.2443, 'learning_rate': 9.218165262080023e-05, 'epoch': 0.41} 20%|██ | 41/200 [07:30<27:00, 10.19s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9579, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.9203, device='cuda:0', grad_fn=) [2024-06-19 00:22:23,636] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3544.39 | bwd_microstep: 1883.01 | bwd_inner_microstep: 1878.21 | bwd_allreduce_microstep: 4.69 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.7098, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.6966, device='cuda:0', grad_fn=) [2024-06-19 00:22:29,216] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.78 [2024-06-19 00:22:29,217] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3545.69 | bwd_microstep: 1883.72 | bwd_inner_microstep: 1878.27 | bwd_allreduce_microstep: 5.33 | step_microstep: 60.13 [2024-06-19 00:22:29,217] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7090.06 | bwd: 3766.72 | bwd_inner: 3756.51 | bwd_allreduce: 10.00 | step: 60.21 21%|██ | 42/200 [07:41<27:33, 10.47s/it] {'loss': 0.8085, 'learning_rate': 9.174140265146356e-05, 'epoch': 0.42} 21%|██ | 42/200 [07:41<27:33, 10.47s/it]warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.5234, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.5289, device='cuda:0', grad_fn=) [2024-06-19 00:22:33,716] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2692.09 | bwd_microstep: 1715.87 | bwd_inner_microstep: 1711.08 | bwd_allreduce_microstep: 4.68 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0728, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.0240, device='cuda:0', grad_fn=) [2024-06-19 00:22:39,276] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.79 [2024-06-19 00:22:39,276] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3537.14 | bwd_microstep: 1874.76 | bwd_inner_microstep: 1869.42 | bwd_allreduce_microstep: 5.23 | step_microstep: 60.30 [2024-06-19 00:22:39,277] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6229.22 | bwd: 3590.63 | bwd_inner: 3580.51 | bwd_allreduce: 9.91 | step: 60.38 22%|██▏ | 43/200 [07:51<27:04, 10.34s/it] {'loss': 0.7764, 'learning_rate': 9.129020672271283e-05, 'epoch': 0.43} 22%|██▏ | 43/200 [07:51<27:04, 10.34s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0201, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.0759, device='cuda:0', grad_fn=) [2024-06-19 00:22:44,482] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3437.42 | bwd_microstep: 1682.93 | bwd_inner_microstep: 1677.94 | bwd_allreduce_microstep: 4.87 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9644, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.9258, device='cuda:0', grad_fn=) [2024-06-19 00:22:49,938] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.92 [2024-06-19 00:22:49,939] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3498.76 | bwd_microstep: 1808.45 | bwd_inner_microstep: 1803.00 | bwd_allreduce_microstep: 5.34 | step_microstep: 61.05 [2024-06-19 00:22:49,939] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6936.16 | bwd: 3491.38 | bwd_inner: 3480.95 | bwd_allreduce: 10.22 | step: 61.13 22%|██▏ | 44/200 [08:02<27:08, 10.44s/it] {'loss': 0.5008, 'learning_rate': 9.082818315286055e-05, 'epoch': 0.44} 22%|██▏ | 44/200 [08:02<27:08, 10.44s/it]warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0211, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0771, device='cuda:0', grad_fn=) [2024-06-19 00:22:54,467] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2829.28 | bwd_microstep: 1615.51 | bwd_inner_microstep: 1610.75 | bwd_allreduce_microstep: 4.65 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0904, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0392, device='cuda:0', grad_fn=) [2024-06-19 00:23:00,077] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.83 [2024-06-19 00:23:00,077] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3552.36 | bwd_microstep: 1901.18 | bwd_inner_microstep: 1895.74 | bwd_allreduce_microstep: 5.32 | step_microstep: 60.58 [2024-06-19 00:23:00,078] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6381.63 | bwd: 3516.68 | bwd_inner: 3506.50 | bwd_allreduce: 9.97 | step: 60.66 22%|██▎ | 45/200 [08:12<26:44, 10.35s/it] {'loss': 0.5582, 'learning_rate': 9.035545309958046e-05, 'epoch': 0.45} 22%|██▎ | 45/200 [08:12<26:44, 10.35s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.4544, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.3671, device='cuda:0', grad_fn=) [2024-06-19 00:23:05,701] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3574.37 | bwd_microstep: 1948.00 | bwd_inner_microstep: 1943.19 | bwd_allreduce_microstep: 4.69 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1886, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(1.1394, device='cuda:0', grad_fn=) [2024-06-19 00:23:11,152] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:23:11,153] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3495.38 | bwd_microstep: 1807.94 | bwd_inner_microstep: 1802.55 | bwd_allreduce_microstep: 5.27 | step_microstep: 60.38 [2024-06-19 00:23:11,153] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7069.73 | bwd: 3755.93 | bwd_inner: 3745.77 | bwd_allreduce: 9.96 | step: 60.46 23%|██▎ | 46/200 [08:23<27:07, 10.57s/it] {'loss': 1.2532, 'learning_rate': 8.987214052813604e-05, 'epoch': 0.46} 23%|██▎ | 46/200 [08:23<27:07, 10.57s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0014, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0594, device='cuda:0', grad_fn=) [2024-06-19 00:23:16,361] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3439.08 | bwd_microstep: 1683.83 | bwd_inner_microstep: 1679.05 | bwd_allreduce_microstep: 4.66 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0881, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0371, device='cuda:0', grad_fn=) [2024-06-19 00:23:21,836] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:23:21,837] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3501.50 | bwd_microstep: 1821.31 | bwd_inner_microstep: 1815.98 | bwd_allreduce_microstep: 5.21 | step_microstep: 59.92 [2024-06-19 00:23:21,837] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6940.56 | bwd: 3505.14 | bwd_inner: 3495.07 | bwd_allreduce: 9.88 | step: 60.00 24%|██▎ | 47/200 [08:34<27:02, 10.60s/it] {'loss': 0.5482, 'learning_rate': 8.937837217887273e-05, 'epoch': 0.47} 24%|██▎ | 47/200 [08:34<27:02, 10.60s/it]warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(1.3461, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.2693, device='cuda:0', grad_fn=) [2024-06-19 00:23:24,820] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 1948.75 | bwd_microstep: 965.59 | bwd_inner_microstep: 960.79 | bwd_allreduce_microstep: 4.70 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1464, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0896, device='cuda:0', grad_fn=) [2024-06-19 00:23:30,297] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.85 [2024-06-19 00:23:30,298] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3506.70 | bwd_microstep: 1818.85 | bwd_inner_microstep: 1813.28 | bwd_allreduce_microstep: 5.45 | step_microstep: 61.31 [2024-06-19 00:23:30,299] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 5455.45 | bwd: 2784.44 | bwd_inner: 2774.09 | bwd_allreduce: 10.15 | step: 61.39 24%|██▍ | 48/200 [08:42<25:13, 9.96s/it] {'loss': 1.1794, 'learning_rate': 8.887427753398248e-05, 'epoch': 0.48} 24%|██▍ | 48/200 [08:42<25:13, 9.96s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1985, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1364, device='cuda:0', grad_fn=) [2024-06-19 00:23:35,685] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3493.14 | bwd_microstep: 1802.33 | bwd_inner_microstep: 1797.55 | bwd_allreduce_microstep: 4.66 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(1.1050, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0524, device='cuda:0', grad_fn=) [2024-06-19 00:23:37,788] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:23:37,789] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 1444.23 | bwd_microstep: 543.37 | bwd_inner_microstep: 538.03 | bwd_allreduce_microstep: 5.23 | step_microstep: 59.97 [2024-06-19 00:23:37,789] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 4937.35 | bwd: 2345.69 | bwd_inner: 2335.61 | bwd_allreduce: 9.89 | step: 60.05 24%|██▍ | 49/200 [08:50<23:12, 9.22s/it] {'loss': 1.0944, 'learning_rate': 8.835998878354931e-05, 'epoch': 0.49} 24%|██▍ | 49/200 [08:50<23:12, 9.22s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1280) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1280, 6144]) tensor(0.9162, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.8827, device='cuda:0', grad_fn=) [2024-06-19 00:23:42,961] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3231.98 | bwd_microstep: 1843.84 | bwd_inner_microstep: 1838.98 | bwd_allreduce_microstep: 4.75 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0970, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0451, device='cuda:0', grad_fn=) [2024-06-19 00:23:48,429] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.83 [2024-06-19 00:23:48,429] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3498.42 | bwd_microstep: 1817.77 | bwd_inner_microstep: 1812.37 | bwd_allreduce_microstep: 5.28 | step_microstep: 60.24 [2024-06-19 00:23:48,430] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6730.39 | bwd: 3661.60 | bwd_inner: 3651.38 | bwd_allreduce: 10.03 | step: 60.32 25%|██▌ | 50/200 [09:01<24:06, 9.65s/it] {'loss': 0.9639, 'learning_rate': 8.783564079088477e-05, 'epoch': 0.5} 25%|██▌ | 50/200 [09:01<24:06, 9.65s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(0.0004, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.0582, device='cuda:0', grad_fn=) [2024-06-19 00:23:52,971] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.99 | bwd_microstep: 1623.02 | bwd_inner_microstep: 1618.20 | bwd_allreduce_microstep: 4.70 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2936, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.2221, device='cuda:0', grad_fn=) [2024-06-19 00:23:58,419] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:23:58,420] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3494.31 | bwd_microstep: 1805.65 | bwd_inner_microstep: 1800.32 | bwd_allreduce_microstep: 5.21 | step_microstep: 60.26 [2024-06-19 00:23:58,420] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6326.27 | bwd: 3428.66 | bwd_inner: 3418.54 | bwd_allreduce: 9.92 | step: 60.34 26%|██▌ | 51/200 [09:11<24:12, 9.75s/it] {'loss': 0.6401, 'learning_rate': 8.73013710571623e-05, 'epoch': 0.51} 26%|██▌ | 51/200 [09:11<24:12, 9.75s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2436, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1770, device='cuda:0', grad_fn=) [2024-06-19 00:24:03,812] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3495.67 | bwd_microstep: 1805.01 | bwd_inner_microstep: 1800.16 | bwd_allreduce_microstep: 4.74 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1280) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1280, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1561, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0983, device='cuda:0', grad_fn=) [2024-06-19 00:24:09,041] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:24:09,042] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3235.36 | bwd_microstep: 1843.52 | bwd_inner_microstep: 1838.09 | bwd_allreduce_microstep: 5.32 | step_microstep: 60.57 [2024-06-19 00:24:09,042] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6731.02 | bwd: 3648.52 | bwd_inner: 3638.27 | bwd_allreduce: 10.06 | step: 60.65 26%|██▌ | 52/200 [09:21<24:41, 10.01s/it] {'loss': 1.1377, 'learning_rate': 8.675731968536002e-05, 'epoch': 0.52} 26%|██▌ | 52/200 [09:21<24:41, 10.01s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0558, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.1083, device='cuda:0', grad_fn=) [2024-06-19 00:24:14,243] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3437.96 | bwd_microstep: 1679.30 | bwd_inner_microstep: 1674.42 | bwd_allreduce_microstep: 4.71 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0521, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(1.0165, device='cuda:0', grad_fn=) [2024-06-19 00:24:19,810] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:24:19,811] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3542.21 | bwd_microstep: 1876.68 | bwd_inner_microstep: 1871.29 | bwd_allreduce_microstep: 5.28 | step_microstep: 60.06 [2024-06-19 00:24:19,811] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6980.16 | bwd: 3555.97 | bwd_inner: 3545.76 | bwd_allreduce: 10.00 | step: 60.13 26%|██▋ | 53/200 [09:32<25:05, 10.24s/it] {'loss': 0.5624, 'learning_rate': 8.620362934352109e-05, 'epoch': 0.53} 26%|██▋ | 53/200 [09:32<25:05, 10.24s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2566, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.1891, device='cuda:0', grad_fn=) [2024-06-19 00:24:25,202] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3494.32 | bwd_microstep: 1803.06 | bwd_inner_microstep: 1797.99 | bwd_allreduce_microstep: 4.96 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1809, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1207, device='cuda:0', grad_fn=) [2024-06-19 00:24:30,686] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.93 [2024-06-19 00:24:30,686] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3502.66 | bwd_microstep: 1827.26 | bwd_inner_microstep: 1821.83 | bwd_allreduce_microstep: 5.32 | step_microstep: 61.19 [2024-06-19 00:24:30,687] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6996.96 | bwd: 3630.31 | bwd_inner: 3619.84 | bwd_allreduce: 10.28 | step: 61.28 27%|██▋ | 54/200 [09:43<25:22, 10.43s/it] {'loss': 1.1549, 'learning_rate': 8.564044522734147e-05, 'epoch': 0.54} 27%|██▋ | 54/200 [09:43<25:22, 10.43s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0075, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(0.0764, device='cuda:0', grad_fn=) [2024-06-19 00:24:35,886] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3437.05 | bwd_microstep: 1679.10 | bwd_inner_microstep: 1674.20 | bwd_allreduce_microstep: 4.79 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.4279, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.3433, device='cuda:0', grad_fn=) [2024-06-19 00:24:41,526] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.85 [2024-06-19 00:24:41,526] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3557.48 | bwd_microstep: 1923.41 | bwd_inner_microstep: 1917.92 | bwd_allreduce_microstep: 5.38 | step_microstep: 60.26 [2024-06-19 00:24:41,527] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6994.52 | bwd: 3602.50 | bwd_inner: 3592.13 | bwd_allreduce: 10.18 | step: 60.34 28%|██▊ | 55/200 [09:54<25:30, 10.55s/it] {'loss': 0.7098, 'learning_rate': 8.506791502209496e-05, 'epoch': 0.55} 28%|██▊ | 55/200 [09:54<25:30, 10.55s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0057, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(0.0748, device='cuda:0', grad_fn=) [2024-06-19 00:24:46,726] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3434.90 | bwd_microstep: 1680.22 | bwd_inner_microstep: 1675.27 | bwd_allreduce_microstep: 4.77 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1992, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1371, device='cuda:0', grad_fn=) [2024-06-19 00:24:52,322] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:24:52,322] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3545.03 | bwd_microstep: 1894.96 | bwd_inner_microstep: 1889.57 | bwd_allreduce_microstep: 5.28 | step_microstep: 60.05 [2024-06-19 00:24:52,323] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6979.91 | bwd: 3575.17 | bwd_inner: 3564.90 | bwd_allreduce: 10.06 | step: 60.12 28%|██▊ | 56/200 [10:04<25:30, 10.63s/it] {'loss': 0.6059, 'learning_rate': 8.448618886390522e-05, 'epoch': 0.56} 28%|██▊ | 56/200 [10:04<25:30, 10.63s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0056, device='cuda:0', grad_fn=) tensor(0.8148, device='cuda:0', grad_fn=) tensor(0.0865, device='cuda:0', grad_fn=) [2024-06-19 00:24:57,487] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3429.88 | bwd_microstep: 1652.62 | bwd_inner_microstep: 1647.59 | bwd_allreduce_microstep: 4.86 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(1.3216, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(1.2591, device='cuda:0', grad_fn=) [2024-06-19 00:25:01,847] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:25:01,848] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2638.57 | bwd_microstep: 1577.76 | bwd_inner_microstep: 1572.29 | bwd_allreduce_microstep: 5.36 | step_microstep: 60.50 [2024-06-19 00:25:01,848] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6068.42 | bwd: 3230.38 | bwd_inner: 3219.93 | bwd_allreduce: 10.21 | step: 60.58 28%|██▊ | 57/200 [10:14<24:32, 10.30s/it] {'loss': 0.6728, 'learning_rate': 8.389541930037516e-05, 'epoch': 0.57} 28%|██▊ | 57/200 [10:14<24:32, 10.30s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2873, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(1.2285, device='cuda:0', grad_fn=) [2024-06-19 00:25:07,409] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3548.35 | bwd_microstep: 1913.53 | bwd_inner_microstep: 1908.64 | bwd_allreduce_microstep: 4.79 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.8487, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.8216, device='cuda:0', grad_fn=) [2024-06-19 00:25:12,597] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.86 [2024-06-19 00:25:12,598] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3415.20 | bwd_microstep: 1629.64 | bwd_inner_microstep: 1624.00 | bwd_allreduce_microstep: 5.53 | step_microstep: 60.55 [2024-06-19 00:25:12,598] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6963.53 | bwd: 3543.16 | bwd_inner: 3532.65 | bwd_allreduce: 10.32 | step: 60.63 29%|██▉ | 58/200 [10:25<24:41, 10.43s/it] {'loss': 1.0251, 'learning_rate': 8.329576125058406e-05, 'epoch': 0.58} 29%|██▉ | 58/200 [10:25<24:41, 10.43s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.8577, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.8304, device='cuda:0', grad_fn=) [2024-06-19 00:25:18,135] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3543.56 | bwd_microstep: 1893.96 | bwd_inner_microstep: 1889.05 | bwd_allreduce_microstep: 4.79 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.7675, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.7493, device='cuda:0', grad_fn=) [2024-06-19 00:25:23,725] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.89 [2024-06-19 00:25:23,726] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3545.46 | bwd_microstep: 1890.67 | bwd_inner_microstep: 1885.07 | bwd_allreduce_microstep: 5.42 | step_microstep: 60.41 [2024-06-19 00:25:23,726] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7089.00 | bwd: 3784.62 | bwd_inner: 3774.19 | bwd_allreduce: 10.19 | step: 60.49 30%|██▉ | 59/200 [10:36<25:00, 10.64s/it] {'loss': 0.7898, 'learning_rate': 8.268737196446264e-05, 'epoch': 0.59} 30%|██▉ | 59/200 [10:36<25:00, 10.64s/it]warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.4628, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.4746, device='cuda:0', grad_fn=) [2024-06-19 00:25:28,048] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2644.12 | bwd_microstep: 1593.59 | bwd_inner_microstep: 1588.46 | bwd_allreduce_microstep: 5.02 | step_microstep: 0.14 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2847, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(1.2262, device='cuda:0', grad_fn=) [2024-06-19 00:25:33,631] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:25:33,631] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3545.09 | bwd_microstep: 1883.91 | bwd_inner_microstep: 1878.49 | bwd_allreduce_microstep: 5.30 | step_microstep: 60.39 [2024-06-19 00:25:33,632] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6189.19 | bwd: 3477.49 | bwd_inner: 3466.97 | bwd_allreduce: 10.34 | step: 60.53 30%|███ | 60/200 [10:46<24:18, 10.42s/it] {'loss': 0.8504, 'learning_rate': 8.2070410981557e-05, 'epoch': 0.6} 30%|███ | 60/200 [10:46<24:18, 10.42s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0011, device='cuda:0', grad_fn=) tensor(0.5880, device='cuda:0', grad_fn=) tensor(0.0598, device='cuda:0', grad_fn=) [2024-06-19 00:25:38,830] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3437.04 | bwd_microstep: 1677.30 | bwd_inner_microstep: 1672.44 | bwd_allreduce_microstep: 4.76 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1700, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(1.1230, device='cuda:0', grad_fn=) [2024-06-19 00:25:44,264] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:25:44,265] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3489.50 | bwd_microstep: 1799.28 | bwd_inner_microstep: 1793.87 | bwd_allreduce_microstep: 5.30 | step_microstep: 60.44 [2024-06-19 00:25:44,265] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6926.53 | bwd: 3476.57 | bwd_inner: 3466.33 | bwd_allreduce: 10.06 | step: 60.52 30%|███ | 61/200 [10:56<24:17, 10.48s/it] {'loss': 0.5914, 'learning_rate': 8.144504008919222e-05, 'epoch': 0.61} 30%|███ | 61/200 [10:56<24:17, 10.48s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1594, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.1019, device='cuda:0', grad_fn=) [2024-06-19 00:25:49,801] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3545.36 | bwd_microstep: 1892.27 | bwd_inner_microstep: 1887.08 | bwd_allreduce_microstep: 5.08 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.3007, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.2288, device='cuda:0', grad_fn=) [2024-06-19 00:25:55,285] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:25:55,286] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3503.04 | bwd_microstep: 1827.48 | bwd_inner_microstep: 1821.92 | bwd_allreduce_microstep: 5.38 | step_microstep: 60.19 [2024-06-19 00:25:55,286] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7048.39 | bwd: 3719.74 | bwd_inner: 3709.05 | bwd_allreduce: 10.44 | step: 60.27 31%|███ | 62/200 [11:07<24:29, 10.65s/it] {'loss': 1.1653, 'learning_rate': 8.081142328004637e-05, 'epoch': 0.62} 31%|███ | 62/200 [11:07<24:29, 10.65s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0917, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.0407, device='cuda:0', grad_fn=) [2024-06-19 00:26:00,814] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3553.55 | bwd_microstep: 1880.15 | bwd_inner_microstep: 1875.07 | bwd_allreduce_microstep: 4.97 | step_microstep: 0.09 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(1.3212, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.2475, device='cuda:0', grad_fn=) [2024-06-19 00:26:02,925] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:26:02,926] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 1446.82 | bwd_microstep: 547.74 | bwd_inner_microstep: 542.23 | bwd_allreduce_microstep: 5.33 | step_microstep: 60.68 [2024-06-19 00:26:02,926] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 5000.35 | bwd: 2427.88 | bwd_inner: 2417.35 | bwd_allreduce: 10.28 | step: 60.77 32%|███▏ | 63/200 [11:15<22:14, 9.74s/it] {'loss': 1.1441, 'learning_rate': 8.016972670914624e-05, 'epoch': 0.63} 32%|███▏ | 63/200 [11:15<22:14, 9.74s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2205, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.1570, device='cuda:0', grad_fn=) [2024-06-19 00:26:08,338] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3494.18 | bwd_microstep: 1820.96 | bwd_inner_microstep: 1816.09 | bwd_allreduce_microstep: 4.75 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1765, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.1173, device='cuda:0', grad_fn=) [2024-06-19 00:26:13,797] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.88 [2024-06-19 00:26:13,798] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3500.39 | bwd_microstep: 1809.78 | bwd_inner_microstep: 1804.19 | bwd_allreduce_microstep: 5.40 | step_microstep: 60.25 [2024-06-19 00:26:13,798] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6994.55 | bwd: 3630.74 | bwd_inner: 3620.34 | bwd_allreduce: 10.14 | step: 60.33 32%|███▏ | 64/200 [11:26<22:51, 10.08s/it] {'loss': 1.1371, 'learning_rate': 7.952011865029614e-05, 'epoch': 0.64} 32%|███▏ | 64/200 [11:26<22:51, 10.08s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0505, device='cuda:0', grad_fn=) tensor(0.7031, device='cuda:0', grad_fn=) tensor(0.1157, device='cuda:0', grad_fn=) [2024-06-19 00:26:18,962] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3429.39 | bwd_microstep: 1653.86 | bwd_inner_microstep: 1648.83 | bwd_allreduce_microstep: 4.85 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.4204, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.3368, device='cuda:0', grad_fn=) [2024-06-19 00:26:24,425] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.85 [2024-06-19 00:26:24,426] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3499.45 | bwd_microstep: 1813.72 | bwd_inner_microstep: 1808.13 | bwd_allreduce_microstep: 5.41 | step_microstep: 60.14 [2024-06-19 00:26:24,426] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6928.83 | bwd: 3467.58 | bwd_inner: 3457.05 | bwd_allreduce: 10.23 | step: 60.22 32%|███▎ | 65/200 [11:37<23:03, 10.25s/it] {'loss': 0.7263, 'learning_rate': 7.886276945195099e-05, 'epoch': 0.65} 32%|███▎ | 65/200 [11:37<23:03, 10.25s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2989, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.2271, device='cuda:0', grad_fn=) [2024-06-19 00:26:29,954] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3542.90 | bwd_microstep: 1887.48 | bwd_inner_microstep: 1882.56 | bwd_allreduce_microstep: 4.82 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0440, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.0981, device='cuda:0', grad_fn=) [2024-06-19 00:26:35,217] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:26:35,218] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3440.03 | bwd_microstep: 1683.53 | bwd_inner_microstep: 1678.09 | bwd_allreduce_microstep: 5.33 | step_microstep: 59.89 [2024-06-19 00:26:35,218] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6982.92 | bwd: 3571.00 | bwd_inner: 3560.66 | bwd_allreduce: 10.16 | step: 59.98 33%|███▎ | 66/200 [11:47<23:14, 10.41s/it] {'loss': 0.6626, 'learning_rate': 7.819785149254532e-05, 'epoch': 0.66} 33%|███▎ | 66/200 [11:47<23:14, 10.41s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1513, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.0946, device='cuda:0', grad_fn=) [2024-06-19 00:26:40,622] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3500.71 | bwd_microstep: 1810.07 | bwd_inner_microstep: 1805.09 | bwd_allreduce_microstep: 4.81 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1233, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0688, device='cuda:0', grad_fn=) [2024-06-19 00:26:46,091] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:26:46,092] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3497.46 | bwd_microstep: 1821.06 | bwd_inner_microstep: 1815.66 | bwd_allreduce_microstep: 5.30 | step_microstep: 60.31 [2024-06-19 00:26:46,092] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6998.16 | bwd: 3631.12 | bwd_inner: 3620.79 | bwd_allreduce: 10.11 | step: 60.39 34%|███▎ | 67/200 [11:58<23:23, 10.55s/it] {'loss': 1.0817, 'learning_rate': 7.752553913529018e-05, 'epoch': 0.67} 34%|███▎ | 67/200 [11:58<23:23, 10.55s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(1.2821, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.2117, device='cuda:0', grad_fn=) [2024-06-19 00:26:50,438] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2655.60 | bwd_microstep: 1601.89 | bwd_inner_microstep: 1596.85 | bwd_allreduce_microstep: 4.92 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0124, device='cuda:0', grad_fn=) tensor(0.6989, device='cuda:0', grad_fn=) tensor(0.0811, device='cuda:0', grad_fn=) [2024-06-19 00:26:55,658] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.83 [2024-06-19 00:26:55,659] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3429.66 | bwd_microstep: 1653.48 | bwd_inner_microstep: 1648.00 | bwd_allreduce_microstep: 5.32 | step_microstep: 60.51 [2024-06-19 00:26:55,659] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6085.25 | bwd: 3255.36 | bwd_inner: 3244.90 | bwd_allreduce: 10.22 | step: 60.60 34%|███▍ | 68/200 [12:08<22:33, 10.25s/it] {'loss': 0.6464, 'learning_rate': 7.68460086824492e-05, 'epoch': 0.68} 34%|███▍ | 68/200 [12:08<22:33, 10.25s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0175, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.9736, device='cuda:0', grad_fn=) [2024-06-19 00:27:01,073] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3499.16 | bwd_microstep: 1819.08 | bwd_inner_microstep: 1814.05 | bwd_allreduce_microstep: 4.85 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.1884, device='cuda:0', grad_fn=) tensor(0.5913, device='cuda:0', grad_fn=) tensor(0.2287, device='cuda:0', grad_fn=) [2024-06-19 00:27:05,408] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.83 [2024-06-19 00:27:05,409] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2631.29 | bwd_microstep: 1567.40 | bwd_inner_microstep: 1562.04 | bwd_allreduce_microstep: 5.26 | step_microstep: 59.91 [2024-06-19 00:27:05,409] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6130.44 | bwd: 3386.48 | bwd_inner: 3376.14 | bwd_allreduce: 10.09 | step: 59.98 34%|███▍ | 69/200 [12:18<22:03, 10.10s/it] {'loss': 0.6011, 'learning_rate': 7.61594383291065e-05, 'epoch': 0.69} 34%|███▍ | 69/200 [12:18<22:03, 10.10s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.3525, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.2758, device='cuda:0', grad_fn=) [2024-06-19 00:27:10,915] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3538.54 | bwd_microstep: 1873.01 | bwd_inner_microstep: 1868.03 | bwd_allreduce_microstep: 4.87 | step_microstep: 0.13 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.8742, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.8449, device='cuda:0', grad_fn=) [2024-06-19 00:27:16,364] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:27:16,365] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3495.86 | bwd_microstep: 1804.78 | bwd_inner_microstep: 1799.32 | bwd_allreduce_microstep: 5.35 | step_microstep: 60.31 [2024-06-19 00:27:16,365] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7034.39 | bwd: 3677.78 | bwd_inner: 3667.36 | bwd_allreduce: 10.23 | step: 60.45 35%|███▌ | 70/200 [12:28<22:26, 10.36s/it] {'loss': 1.0604, 'learning_rate': 7.546600811643816e-05, 'epoch': 0.7} 35%|███▌ | 70/200 [12:28<22:26, 10.36s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0097, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.9669, device='cuda:0', grad_fn=) [2024-06-19 00:27:21,918] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3551.50 | bwd_microstep: 1900.78 | bwd_inner_microstep: 1895.81 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0076, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.0653, device='cuda:0', grad_fn=) [2024-06-19 00:27:27,184] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:27:27,184] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3442.66 | bwd_microstep: 1682.02 | bwd_inner_microstep: 1676.51 | bwd_allreduce_microstep: 5.34 | step_microstep: 60.23 [2024-06-19 00:27:27,185] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6994.14 | bwd: 3582.80 | bwd_inner: 3572.41 | bwd_allreduce: 10.10 | step: 60.31 36%|███▌ | 71/200 [12:39<22:34, 10.50s/it] {'loss': 0.5161, 'learning_rate': 7.476589988449939e-05, 'epoch': 0.71} 36%|███▌ | 71/200 [12:39<22:34, 10.50s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.7688, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.7501, device='cuda:0', grad_fn=) [2024-06-19 00:27:32,584] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3498.17 | bwd_microstep: 1807.84 | bwd_inner_microstep: 1802.91 | bwd_allreduce_microstep: 4.79 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.8986, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.8666, device='cuda:0', grad_fn=) [2024-06-19 00:27:38,062] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:27:38,063] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3501.28 | bwd_microstep: 1825.57 | bwd_inner_microstep: 1820.17 | bwd_allreduce_microstep: 5.29 | step_microstep: 60.26 [2024-06-19 00:27:38,063] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6999.44 | bwd: 3633.40 | bwd_inner: 3623.13 | bwd_allreduce: 10.09 | step: 60.34 36%|███▌ | 72/200 [12:50<22:38, 10.61s/it] {'loss': 0.8083, 'learning_rate': 7.405929722454026e-05, 'epoch': 0.72} 36%|███▌ | 72/200 [12:50<22:38, 10.61s/it]warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.7803, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.7607, device='cuda:0', grad_fn=) [2024-06-19 00:27:42,792] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2893.52 | bwd_microstep: 1744.16 | bwd_inner_microstep: 1739.12 | bwd_allreduce_microstep: 4.92 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.4719, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.3825, device='cuda:0', grad_fn=) [2024-06-19 00:27:47,477] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.90 [2024-06-19 00:27:47,477] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2740.50 | bwd_microstep: 1795.14 | bwd_inner_microstep: 1789.65 | bwd_allreduce_microstep: 5.38 | step_microstep: 61.26 [2024-06-19 00:27:47,478] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 5634.00 | bwd: 3539.29 | bwd_inner: 3528.77 | bwd_allreduce: 10.31 | step: 61.34 36%|███▋ | 73/200 [13:00<21:42, 10.25s/it] {'loss': 1.0716, 'learning_rate': 7.334638543086203e-05, 'epoch': 0.73} 36%|███▋ | 73/200 [13:00<21:42, 10.25s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0172, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.0739, device='cuda:0', grad_fn=) [2024-06-19 00:27:52,673] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3435.65 | bwd_microstep: 1676.21 | bwd_inner_microstep: 1671.37 | bwd_allreduce_microstep: 4.74 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(1.1707, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1115, device='cuda:0', grad_fn=) [2024-06-19 00:27:57,585] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:27:57,586] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2937.78 | bwd_microstep: 1823.14 | bwd_inner_microstep: 1817.65 | bwd_allreduce_microstep: 5.29 | step_microstep: 60.35 [2024-06-19 00:27:57,586] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6373.42 | bwd: 3499.34 | bwd_inner: 3489.09 | bwd_allreduce: 10.01 | step: 60.43 37%|███▋ | 74/200 [13:10<21:26, 10.21s/it] {'loss': 0.5927, 'learning_rate': 7.262735145222696e-05, 'epoch': 0.74} 37%|███▋ | 74/200 [13:10<21:26, 10.21s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.4927, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.4019, device='cuda:0', grad_fn=) [2024-06-19 00:28:03,103] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3543.02 | bwd_microstep: 1879.13 | bwd_inner_microstep: 1874.22 | bwd_allreduce_microstep: 4.79 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0651, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.0168, device='cuda:0', grad_fn=) [2024-06-19 00:28:08,558] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:28:08,559] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3492.96 | bwd_microstep: 1809.51 | bwd_inner_microstep: 1803.93 | bwd_allreduce_microstep: 5.47 | step_microstep: 61.28 [2024-06-19 00:28:08,559] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7035.97 | bwd: 3688.64 | bwd_inner: 3678.17 | bwd_allreduce: 10.27 | step: 61.36 38%|███▊ | 75/200 [13:21<21:44, 10.44s/it] {'loss': 1.2094, 'learning_rate': 7.190238384283412e-05, 'epoch': 0.75} 38%|███▊ | 75/200 [13:21<21:44, 10.44s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0019, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0598, device='cuda:0', grad_fn=) [2024-06-19 00:28:13,756] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3437.26 | bwd_microstep: 1677.28 | bwd_inner_microstep: 1672.37 | bwd_allreduce_microstep: 4.75 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0049, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.9629, device='cuda:0', grad_fn=) [2024-06-19 00:28:19,207] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:28:19,207] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3495.36 | bwd_microstep: 1807.35 | bwd_inner_microstep: 1801.98 | bwd_allreduce_microstep: 5.26 | step_microstep: 60.31 [2024-06-19 00:28:19,208] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6932.60 | bwd: 3484.63 | bwd_inner: 3474.40 | bwd_allreduce: 10.01 | step: 60.39 38%|███▊ | 76/200 [13:31<21:42, 10.50s/it] {'loss': 0.5113, 'learning_rate': 7.117167271287453e-05, 'epoch': 0.76} 38%|███▊ | 76/200 [13:31<21:42, 10.50s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.8560, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(0.8400, device='cuda:0', grad_fn=) [2024-06-19 00:28:24,738] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3547.60 | bwd_microstep: 1886.68 | bwd_inner_microstep: 1881.85 | bwd_allreduce_microstep: 4.72 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2058, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(1.1552, device='cuda:0', grad_fn=) [2024-06-19 00:28:29,092] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:28:29,092] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2634.32 | bwd_microstep: 1575.57 | bwd_inner_microstep: 1569.87 | bwd_allreduce_microstep: 5.58 | step_microstep: 62.36 [2024-06-19 00:28:29,093] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6181.91 | bwd: 3462.24 | bwd_inner: 3451.74 | bwd_allreduce: 10.30 | step: 62.44 38%|███▊ | 77/200 [13:41<21:08, 10.32s/it] {'loss': 0.9976, 'learning_rate': 7.043540967867782e-05, 'epoch': 0.77} 38%|███▊ | 77/200 [13:41<21:08, 10.32s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.5406, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.4450, device='cuda:0', grad_fn=) [2024-06-19 00:28:34,605] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3539.54 | bwd_microstep: 1878.57 | bwd_inner_microstep: 1873.78 | bwd_allreduce_microstep: 4.66 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.7056, device='cuda:0', grad_fn=) tensor(0.5880, device='cuda:0', grad_fn=) tensor(0.6939, device='cuda:0', grad_fn=) [2024-06-19 00:28:39,859] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:28:39,860] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3438.19 | bwd_microstep: 1677.81 | bwd_inner_microstep: 1672.46 | bwd_allreduce_microstep: 5.24 | step_microstep: 60.40 [2024-06-19 00:28:39,860] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6977.72 | bwd: 3556.38 | bwd_inner: 3546.27 | bwd_allreduce: 9.90 | step: 60.48 39%|███▉ | 78/200 [13:52<21:15, 10.45s/it] {'loss': 1.0694, 'learning_rate': 6.969378781246436e-05, 'epoch': 0.78} 39%|███▉ | 78/200 [13:52<21:15, 10.45s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0290, device='cuda:0', grad_fn=) tensor(0.6956, device='cuda:0', grad_fn=) tensor(0.0956, device='cuda:0', grad_fn=) [2024-06-19 00:28:45,059] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3437.76 | bwd_microstep: 1678.32 | bwd_inner_microstep: 1673.45 | bwd_allreduce_microstep: 4.77 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(0.0005, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.0589, device='cuda:0', grad_fn=) [2024-06-19 00:28:49,659] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.88 [2024-06-19 00:28:49,660] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.32 | bwd_microstep: 1623.18 | bwd_inner_microstep: 1617.62 | bwd_allreduce_microstep: 5.44 | step_microstep: 61.19 [2024-06-19 00:28:49,660] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6273.06 | bwd: 3301.50 | bwd_inner: 3291.09 | bwd_allreduce: 10.22 | step: 61.28 40%|███▉ | 79/200 [14:02<20:41, 10.26s/it] {'loss': 0.0773, 'learning_rate': 6.894700159171534e-05, 'epoch': 0.79} 40%|███▉ | 79/200 [14:02<20:41, 10.26s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1280) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1280, 6144]) tensor(1.3620, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.2839, device='cuda:0', grad_fn=) [2024-06-19 00:28:54,725] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3189.78 | bwd_microstep: 1778.99 | bwd_inner_microstep: 1774.14 | bwd_allreduce_microstep: 4.74 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2778, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.2079, device='cuda:0', grad_fn=) [2024-06-19 00:29:00,310] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:29:00,310] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3547.00 | bwd_microstep: 1885.35 | bwd_inner_microstep: 1879.84 | bwd_allreduce_microstep: 5.33 | step_microstep: 60.43 [2024-06-19 00:29:00,311] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6736.77 | bwd: 3664.33 | bwd_inner: 3654.03 | bwd_allreduce: 10.06 | step: 60.51 40%|████ | 80/200 [14:12<20:44, 10.37s/it] {'loss': 1.2459, 'learning_rate': 6.819524684817438e-05, 'epoch': 0.8} 40%|████ | 80/200 [14:12<20:44, 10.37s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.3901, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.3089, device='cuda:0', grad_fn=) [2024-06-19 00:29:05,824] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3542.98 | bwd_microstep: 1876.49 | bwd_inner_microstep: 1871.52 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2093, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.1465, device='cuda:0', grad_fn=) [2024-06-19 00:29:11,423] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.83 [2024-06-19 00:29:11,424] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3548.73 | bwd_microstep: 1894.31 | bwd_inner_microstep: 1888.86 | bwd_allreduce_microstep: 5.34 | step_microstep: 60.65 [2024-06-19 00:29:11,424] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7091.70 | bwd: 3770.79 | bwd_inner: 3760.43 | bwd_allreduce: 10.12 | step: 60.73 40%|████ | 81/200 [14:24<21:00, 10.60s/it] {'loss': 1.2277, 'learning_rate': 6.743872071649411e-05, 'epoch': 0.81} 40%|████ | 81/200 [14:24<21:00, 10.60s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0461, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(0.1115, device='cuda:0', grad_fn=) [2024-06-19 00:29:16,593] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3433.54 | bwd_microstep: 1653.40 | bwd_inner_microstep: 1648.50 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.4792, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.3891, device='cuda:0', grad_fn=) [2024-06-19 00:29:22,060] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:29:22,060] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3506.47 | bwd_microstep: 1810.38 | bwd_inner_microstep: 1804.74 | bwd_allreduce_microstep: 5.53 | step_microstep: 61.28 [2024-06-19 00:29:22,061] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6939.99 | bwd: 3463.77 | bwd_inner: 3453.25 | bwd_allreduce: 10.33 | step: 61.36 41%|████ | 82/200 [14:34<20:51, 10.61s/it] {'loss': 0.7503, 'learning_rate': 6.667762158254104e-05, 'epoch': 0.82} 41%|████ | 82/200 [14:34<20:51, 10.61s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(0.9056, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.8729, device='cuda:0', grad_fn=) [2024-06-19 00:29:26,613] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2834.77 | bwd_microstep: 1629.27 | bwd_inner_microstep: 1624.35 | bwd_allreduce_microstep: 4.82 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9543, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.9170, device='cuda:0', grad_fn=) [2024-06-19 00:29:32,089] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:29:32,089] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3499.13 | bwd_microstep: 1822.42 | bwd_inner_microstep: 1816.89 | bwd_allreduce_microstep: 5.36 | step_microstep: 60.51 [2024-06-19 00:29:32,090] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6333.89 | bwd: 3451.68 | bwd_inner: 3441.28 | bwd_allreduce: 10.16 | step: 60.60 42%|████▏ | 83/200 [14:44<20:20, 10.43s/it] {'loss': 0.8949, 'learning_rate': 6.59121490313722e-05, 'epoch': 0.83} 42%|████▏ | 83/200 [14:44<20:20, 10.43s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9646, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.9259, device='cuda:0', grad_fn=) [2024-06-19 00:29:37,484] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3492.17 | bwd_microstep: 1809.52 | bwd_inner_microstep: 1804.57 | bwd_allreduce_microstep: 4.78 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2875, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.2169, device='cuda:0', grad_fn=) [2024-06-19 00:29:42,959] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.86 [2024-06-19 00:29:42,959] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3500.86 | bwd_microstep: 1820.60 | bwd_inner_microstep: 1815.09 | bwd_allreduce_microstep: 5.39 | step_microstep: 60.42 [2024-06-19 00:29:42,960] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6993.01 | bwd: 3630.11 | bwd_inner: 3619.70 | bwd_allreduce: 10.16 | step: 60.50 42%|████▏ | 84/200 [14:55<20:25, 10.57s/it] {'loss': 1.0714, 'learning_rate': 6.514250379489753e-05, 'epoch': 0.84} 42%|████▏ | 84/200 [14:55<20:25, 10.57s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.5188, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.4254, device='cuda:0', grad_fn=) [2024-06-19 00:29:48,348] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3491.25 | bwd_microstep: 1804.42 | bwd_inner_microstep: 1799.31 | bwd_allreduce_microstep: 5.00 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0486, device='cuda:0', grad_fn=) tensor(0.5880, device='cuda:0', grad_fn=) tensor(0.1026, device='cuda:0', grad_fn=) [2024-06-19 00:29:53,481] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.79 [2024-06-19 00:29:53,481] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3394.37 | bwd_microstep: 1600.89 | bwd_inner_microstep: 1595.47 | bwd_allreduce_microstep: 5.31 | step_microstep: 60.22 [2024-06-19 00:29:53,482] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6885.60 | bwd: 3405.30 | bwd_inner: 3394.80 | bwd_allreduce: 10.32 | step: 60.31 42%|████▎ | 85/200 [15:06<20:13, 10.55s/it] {'loss': 0.764, 'learning_rate': 6.436888769924142e-05, 'epoch': 0.85} 42%|████▎ | 85/200 [15:06<20:13, 10.55s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0001, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.9582, device='cuda:0', grad_fn=) [2024-06-19 00:29:59,001] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3542.58 | bwd_microstep: 1881.71 | bwd_inner_microstep: 1876.80 | bwd_allreduce_microstep: 4.77 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1517, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0943, device='cuda:0', grad_fn=) [2024-06-19 00:30:04,590] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.83 [2024-06-19 00:30:04,590] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3543.89 | bwd_microstep: 1890.07 | bwd_inner_microstep: 1884.62 | bwd_allreduce_microstep: 5.37 | step_microstep: 60.34 [2024-06-19 00:30:04,591] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7086.45 | bwd: 3771.78 | bwd_inner: 3761.45 | bwd_allreduce: 10.15 | step: 60.42 43%|████▎ | 86/200 [15:17<20:21, 10.72s/it] {'loss': 1.0263, 'learning_rate': 6.359150361181715e-05, 'epoch': 0.86} 43%|████▎ | 86/200 [15:17<20:21, 10.72s/it]warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0330, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.9878, device='cuda:0', grad_fn=) [2024-06-19 00:30:09,435] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2932.60 | bwd_microstep: 1816.71 | bwd_inner_microstep: 1811.53 | bwd_allreduce_microstep: 5.05 | step_microstep: 0.09 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2631, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.1952, device='cuda:0', grad_fn=) [2024-06-19 00:30:15,001] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.85 [2024-06-19 00:30:15,002] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3540.29 | bwd_microstep: 1875.58 | bwd_inner_microstep: 1870.13 | bwd_allreduce_microstep: 5.28 | step_microstep: 60.50 [2024-06-19 00:30:15,002] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6472.88 | bwd: 3692.28 | bwd_inner: 3681.72 | bwd_allreduce: 10.32 | step: 60.59 44%|████▎ | 87/200 [15:27<20:00, 10.63s/it] {'loss': 1.0915, 'learning_rate': 6.281055538812861e-05, 'epoch': 0.87} 44%|████▎ | 87/200 [15:27<20:00, 10.63s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1269, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(1.0839, device='cuda:0', grad_fn=) [2024-06-19 00:30:20,395] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3492.48 | bwd_microstep: 1807.68 | bwd_inner_microstep: 1802.87 | bwd_allreduce_microstep: 4.74 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2676, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.1993, device='cuda:0', grad_fn=) [2024-06-19 00:30:25,842] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.88 [2024-06-19 00:30:25,843] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3495.46 | bwd_microstep: 1805.14 | bwd_inner_microstep: 1799.61 | bwd_allreduce_microstep: 5.42 | step_microstep: 60.39 [2024-06-19 00:30:25,843] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6987.93 | bwd: 3612.82 | bwd_inner: 3602.47 | bwd_allreduce: 10.18 | step: 60.47 44%|████▍ | 88/200 [15:38<19:57, 10.69s/it] {'loss': 1.1416, 'learning_rate': 6.202624781831268e-05, 'epoch': 0.88} 44%|████▍ | 88/200 [15:38<19:57, 10.69s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0756, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.0262, device='cuda:0', grad_fn=) [2024-06-19 00:30:31,364] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3538.96 | bwd_microstep: 1885.03 | bwd_inner_microstep: 1879.95 | bwd_allreduce_microstep: 4.97 | step_microstep: 0.14 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1224, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0680, device='cuda:0', grad_fn=) [2024-06-19 00:30:35,754] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.83 [2024-06-19 00:30:35,754] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2642.57 | bwd_microstep: 1600.31 | bwd_inner_microstep: 1594.77 | bwd_allreduce_microstep: 5.36 | step_microstep: 60.54 [2024-06-19 00:30:35,755] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6181.51 | bwd: 3485.33 | bwd_inner: 3474.78 | bwd_allreduce: 10.32 | step: 60.68 44%|████▍ | 89/200 [15:48<19:20, 10.46s/it] {'loss': 1.0471, 'learning_rate': 6.123878657343648e-05, 'epoch': 0.89} 44%|████▍ | 89/200 [15:48<19:20, 10.46s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.5192, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.4251, device='cuda:0', grad_fn=) [2024-06-19 00:30:41,280] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3543.59 | bwd_microstep: 1884.23 | bwd_inner_microstep: 1879.29 | bwd_allreduce_microstep: 4.76 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0261, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.0819, device='cuda:0', grad_fn=) [2024-06-19 00:30:46,502] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:30:46,503] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3431.21 | bwd_microstep: 1653.67 | bwd_inner_microstep: 1648.23 | bwd_allreduce_microstep: 5.33 | step_microstep: 60.28 [2024-06-19 00:30:46,503] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6974.79 | bwd: 3537.89 | bwd_inner: 3527.57 | bwd_allreduce: 10.10 | step: 60.36 45%|████▌ | 90/200 [15:59<19:19, 10.54s/it] {'loss': 0.7535, 'learning_rate': 6.044837815156377e-05, 'epoch': 0.9} 45%|████▌ | 90/200 [15:59<19:19, 10.54s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2180, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.1544, device='cuda:0', grad_fn=) [2024-06-19 00:30:52,034] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3543.60 | bwd_microstep: 1888.76 | bwd_inner_microstep: 1883.78 | bwd_allreduce_microstep: 4.81 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1824, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.1223, device='cuda:0', grad_fn=) [2024-06-19 00:30:57,598] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:30:57,599] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3540.05 | bwd_microstep: 1875.20 | bwd_inner_microstep: 1869.78 | bwd_allreduce_microstep: 5.31 | step_microstep: 60.29 [2024-06-19 00:30:57,599] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7083.63 | bwd: 3763.96 | bwd_inner: 3753.61 | bwd_allreduce: 10.12 | step: 60.37 46%|████▌ | 91/200 [16:10<19:27, 10.71s/it] {'loss': 1.1383, 'learning_rate': 5.9655229823604406e-05, 'epoch': 0.91} 46%|████▌ | 91/200 [16:10<19:27, 10.71s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.3060, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.2332, device='cuda:0', grad_fn=) [2024-06-19 00:31:03,023] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3504.61 | bwd_microstep: 1822.58 | bwd_inner_microstep: 1817.53 | bwd_allreduce_microstep: 4.93 | step_microstep: 0.09 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1310, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.0760, device='cuda:0', grad_fn=) [2024-06-19 00:31:08,640] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.91 [2024-06-19 00:31:08,641] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3551.59 | bwd_microstep: 1910.06 | bwd_inner_microstep: 1904.55 | bwd_allreduce_microstep: 5.35 | step_microstep: 60.96 [2024-06-19 00:31:08,641] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7056.20 | bwd: 3732.63 | bwd_inner: 3722.13 | bwd_allreduce: 10.30 | step: 61.05 46%|████▌ | 92/200 [16:21<19:27, 10.81s/it] {'loss': 1.1546, 'learning_rate': 5.885954957896115e-05, 'epoch': 0.92} 46%|████▌ | 92/200 [16:21<19:27, 10.81s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(1.3434, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.2669, device='cuda:0', grad_fn=) [2024-06-19 00:31:13,293] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2746.07 | bwd_microstep: 1805.35 | bwd_inner_microstep: 1800.47 | bwd_allreduce_microstep: 4.72 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(0.0332, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(0.0998, device='cuda:0', grad_fn=) [2024-06-19 00:31:15,785] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:31:15,786] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 1654.82 | bwd_microstep: 725.77 | bwd_inner_microstep: 720.39 | bwd_allreduce_microstep: 5.27 | step_microstep: 60.28 [2024-06-19 00:31:15,786] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 4400.87 | bwd: 2531.12 | bwd_inner: 2520.91 | bwd_allreduce: 10.00 | step: 60.36 46%|████▋ | 93/200 [16:28<17:18, 9.71s/it] {'loss': 0.6833, 'learning_rate': 5.8061546070987994e-05, 'epoch': 0.93} 46%|████▋ | 93/200 [16:28<17:18, 9.71s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0119, device='cuda:0', grad_fn=) tensor(0.7031, device='cuda:0', grad_fn=) tensor(0.0810, device='cuda:0', grad_fn=) [2024-06-19 00:31:20,984] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3432.03 | bwd_microstep: 1678.49 | bwd_inner_microstep: 1673.62 | bwd_allreduce_microstep: 4.76 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.4762, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(1.3982, device='cuda:0', grad_fn=) [2024-06-19 00:31:25,764] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.85 [2024-06-19 00:31:25,765] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2887.91 | bwd_microstep: 1743.99 | bwd_inner_microstep: 1738.45 | bwd_allreduce_microstep: 5.37 | step_microstep: 60.47 [2024-06-19 00:31:25,765] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6319.93 | bwd: 3422.47 | bwd_inner: 3412.12 | bwd_allreduce: 10.13 | step: 60.54 47%|████▋ | 94/200 [16:38<17:17, 9.79s/it] {'loss': 0.7396, 'learning_rate': 5.726142856227452e-05, 'epoch': 0.94} 47%|████▋ | 94/200 [16:38<17:17, 9.79s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.6736, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.6647, device='cuda:0', grad_fn=) [2024-06-19 00:31:31,148] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3488.50 | bwd_microstep: 1802.86 | bwd_inner_microstep: 1797.91 | bwd_allreduce_microstep: 4.77 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.4129, device='cuda:0', grad_fn=) tensor(0.6924, device='cuda:0', grad_fn=) tensor(1.3409, device='cuda:0', grad_fn=) [2024-06-19 00:31:36,602] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:31:36,602] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3498.20 | bwd_microstep: 1808.01 | bwd_inner_microstep: 1802.59 | bwd_allreduce_microstep: 5.31 | step_microstep: 60.46 [2024-06-19 00:31:36,603] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6986.69 | bwd: 3610.86 | bwd_inner: 3600.55 | bwd_allreduce: 10.06 | step: 60.54 48%|████▊ | 95/200 [16:49<17:41, 10.10s/it] {'loss': 1.0028, 'learning_rate': 5.645940686977033e-05, 'epoch': 0.95} 48%|████▊ | 95/200 [16:49<17:41, 10.10s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.6824, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.6723, device='cuda:0', grad_fn=) [2024-06-19 00:31:42,113] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3539.45 | bwd_microstep: 1875.74 | bwd_inner_microstep: 1870.74 | bwd_allreduce_microstep: 4.82 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.8739, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.8450, device='cuda:0', grad_fn=) [2024-06-19 00:31:47,673] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:31:47,673] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3535.87 | bwd_microstep: 1875.32 | bwd_inner_microstep: 1869.87 | bwd_allreduce_microstep: 5.28 | step_microstep: 60.18 [2024-06-19 00:31:47,674] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7075.30 | bwd: 3751.05 | bwd_inner: 3740.69 | bwd_allreduce: 10.08 | step: 60.25 48%|████▊ | 96/200 [17:00<18:01, 10.39s/it] {'loss': 0.7586, 'learning_rate': 5.565569130976422e-05, 'epoch': 0.96} 48%|████▊ | 96/200 [17:00<18:01, 10.39s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9918, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.9511, device='cuda:0', grad_fn=) [2024-06-19 00:31:53,079] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3504.65 | bwd_microstep: 1808.80 | bwd_inner_microstep: 1803.73 | bwd_allreduce_microstep: 4.96 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1572, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0993, device='cuda:0', grad_fn=) [2024-06-19 00:31:58,666] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.86 [2024-06-19 00:31:58,667] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3542.22 | bwd_microstep: 1890.69 | bwd_inner_microstep: 1885.24 | bwd_allreduce_microstep: 5.28 | step_microstep: 60.53 [2024-06-19 00:31:58,667] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7046.85 | bwd: 3699.48 | bwd_inner: 3689.03 | bwd_allreduce: 10.25 | step: 60.62 48%|████▊ | 97/200 [17:11<18:09, 10.57s/it] {'loss': 1.0252, 'learning_rate': 5.4850492642732406e-05, 'epoch': 0.97} 48%|████▊ | 97/200 [17:11<18:09, 10.57s/it]warning: The size of tensor a (0) must match the size of tensor b (1280) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1280, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0522, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.1052, device='cuda:0', grad_fn=) [2024-06-19 00:32:03,528] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3132.44 | bwd_microstep: 1644.48 | bwd_inner_microstep: 1639.57 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.6673, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.5584, device='cuda:0', grad_fn=) [2024-06-19 00:32:09,116] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:32:09,117] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3546.17 | bwd_microstep: 1886.12 | bwd_inner_microstep: 1880.54 | bwd_allreduce_microstep: 5.40 | step_microstep: 60.43 [2024-06-19 00:32:09,117] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6678.60 | bwd: 3530.59 | bwd_inner: 3520.16 | bwd_allreduce: 10.18 | step: 60.52 49%|████▉ | 98/200 [17:21<17:54, 10.54s/it] {'loss': 0.8318, 'learning_rate': 5.4044022018070214e-05, 'epoch': 0.98} 49%|████▉ | 98/200 [17:21<17:54, 10.54s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.4094, device='cuda:0', grad_fn=) tensor(0.6924, device='cuda:0', grad_fn=) tensor(1.3377, device='cuda:0', grad_fn=) [2024-06-19 00:32:14,511] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3494.08 | bwd_microstep: 1806.95 | bwd_inner_microstep: 1802.01 | bwd_allreduce_microstep: 4.83 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(1.7662, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.6477, device='cuda:0', grad_fn=) [2024-06-19 00:32:19,085] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:32:19,086] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2827.15 | bwd_microstep: 1603.12 | bwd_inner_microstep: 1597.62 | bwd_allreduce_microstep: 5.32 | step_microstep: 60.51 [2024-06-19 00:32:19,086] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6321.22 | bwd: 3410.06 | bwd_inner: 3399.68 | bwd_allreduce: 10.13 | step: 60.58 50%|████▉ | 99/200 [17:31<17:27, 10.37s/it] {'loss': 1.4927, 'learning_rate': 5.3236490918721794e-05, 'epoch': 0.99} 50%|████▉ | 99/200 [17:31<17:27, 10.37s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9994, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.9579, device='cuda:0', grad_fn=) [2024-06-19 00:32:24,599] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3541.13 | bwd_microstep: 1877.89 | bwd_inner_microstep: 1872.89 | bwd_allreduce_microstep: 4.81 | step_microstep: 0.07 please install petrel_client Replace train sampler!! petrel_client is not installed. Using PIL to load images. please install petrel_client Replace train sampler!! petrel_client is not installed. Using PIL to load images. please install petrel_client Replace train sampler!! petrel_client is not installed. Using PIL to load images. please install petrel_client Replace train sampler!! petrel_client is not installed. Using PIL to load images. warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2145, device='cuda:0', grad_fn=) tensor(0.5880, device='cuda:0', grad_fn=) tensor(1.1518, device='cuda:0', grad_fn=) [2024-06-19 00:32:30,926] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:32:30,927] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3539.96 | bwd_microstep: 1877.13 | bwd_inner_microstep: 1871.68 | bwd_allreduce_microstep: 5.34 | step_microstep: 60.46 [2024-06-19 00:32:30,927] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7081.04 | bwd: 3755.01 | bwd_inner: 3744.63 | bwd_allreduce: 10.14 | step: 60.54 50%|█████ | 100/200 [17:43<18:00, 10.81s/it] {'loss': 1.0549, 'learning_rate': 5.242811110572242e-05, 'epoch': 1.0} 50%|█████ | 100/200 [17:43<18:00, 10.81s/it][2024-06-19 00:32:33,668] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-06-19 00:32:39,419] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-06-19 00:32:45,214] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2024-06-19 00:32:50,970] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9970, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.9555, device='cuda:0', grad_fn=) [2024-06-19 00:32:59,987] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3549.60 | bwd_microstep: 1870.89 | bwd_inner_microstep: 1866.02 | bwd_allreduce_microstep: 4.77 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.3258, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.2514, device='cuda:0', grad_fn=) [2024-06-19 00:33:04,336] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:33:04,337] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2630.95 | bwd_microstep: 1574.27 | bwd_inner_microstep: 1568.91 | bwd_allreduce_microstep: 5.25 | step_microstep: 60.45 [2024-06-19 00:33:04,337] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6180.52 | bwd: 3445.15 | bwd_inner: 3434.94 | bwd_allreduce: 10.03 | step: 60.53 50%|█████ | 101/200 [18:16<29:01, 17.59s/it] {'loss': 1.1034, 'learning_rate': 5.1619094562667804e-05, 'epoch': 1.01} 50%|█████ | 101/200 [18:16<29:01, 17.59s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0245, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(0.9920, device='cuda:0', grad_fn=) [2024-06-19 00:33:09,723] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3494.39 | bwd_microstep: 1801.33 | bwd_inner_microstep: 1796.28 | bwd_allreduce_microstep: 4.94 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.3374, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.2618, device='cuda:0', grad_fn=) [2024-06-19 00:33:15,429] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.85 [2024-06-19 00:33:15,429] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3576.95 | bwd_microstep: 1966.38 | bwd_inner_microstep: 1960.94 | bwd_allreduce_microstep: 5.26 | step_microstep: 60.79 [2024-06-19 00:33:15,430] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7071.32 | bwd: 3767.70 | bwd_inner: 3757.28 | bwd_allreduce: 10.21 | step: 60.87 51%|█████ | 102/200 [18:28<25:32, 15.64s/it] {'loss': 1.1269, 'learning_rate': 5.080965344012508e-05, 'epoch': 1.02} 51%|█████ | 102/200 [18:28<25:32, 15.64s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0508, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.1035, device='cuda:0', grad_fn=) [2024-06-19 00:33:20,624] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3432.37 | bwd_microstep: 1677.94 | bwd_inner_microstep: 1673.03 | bwd_allreduce_microstep: 4.79 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9946, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.9529, device='cuda:0', grad_fn=) [2024-06-19 00:33:26,243] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:33:26,243] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3551.90 | bwd_microstep: 1908.40 | bwd_inner_microstep: 1903.00 | bwd_allreduce_microstep: 5.30 | step_microstep: 60.51 [2024-06-19 00:33:26,244] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6984.26 | bwd: 3586.33 | bwd_inner: 3576.05 | bwd_allreduce: 10.10 | step: 60.58 52%|█████▏ | 103/200 [18:38<22:56, 14.19s/it] {'loss': 0.5282, 'learning_rate': 5e-05, 'epoch': 1.03} 52%|█████▏ | 103/200 [18:38<22:56, 14.19s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0048, device='cuda:0', grad_fn=) tensor(0.8190, device='cuda:0', grad_fn=) tensor(0.0862, device='cuda:0', grad_fn=) [2024-06-19 00:33:31,410] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3430.34 | bwd_microstep: 1653.05 | bwd_inner_microstep: 1648.08 | bwd_allreduce_microstep: 4.86 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0917, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0404, device='cuda:0', grad_fn=) [2024-06-19 00:33:36,973] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:33:36,974] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3540.40 | bwd_microstep: 1872.81 | bwd_inner_microstep: 1867.41 | bwd_allreduce_microstep: 5.30 | step_microstep: 60.43 [2024-06-19 00:33:36,974] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6970.73 | bwd: 3525.86 | bwd_inner: 3515.50 | bwd_allreduce: 10.17 | step: 60.51 52%|█████▏ | 104/200 [18:49<21:02, 13.15s/it] {'loss': 0.5633, 'learning_rate': 4.919034655987493e-05, 'epoch': 1.04} 52%|█████▏ | 104/200 [18:49<21:02, 13.15s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9849, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(0.9560, device='cuda:0', grad_fn=) [2024-06-19 00:33:42,478] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3537.55 | bwd_microstep: 1873.50 | bwd_inner_microstep: 1868.51 | bwd_allreduce_microstep: 4.81 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0023, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.0599, device='cuda:0', grad_fn=) [2024-06-19 00:33:47,742] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:33:47,742] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3440.55 | bwd_microstep: 1682.04 | bwd_inner_microstep: 1676.57 | bwd_allreduce_microstep: 5.35 | step_microstep: 60.30 [2024-06-19 00:33:47,743] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6978.08 | bwd: 3555.53 | bwd_inner: 3545.14 | bwd_allreduce: 10.14 | step: 60.37 52%|█████▎ | 105/200 [19:00<19:41, 12.44s/it] {'loss': 0.508, 'learning_rate': 4.838090543733222e-05, 'epoch': 1.05} 52%|█████▎ | 105/200 [19:00<19:41, 12.44s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(1.1106, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.0577, device='cuda:0', grad_fn=) [2024-06-19 00:33:52,471] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2889.71 | bwd_microstep: 1747.33 | bwd_inner_microstep: 1742.45 | bwd_allreduce_microstep: 4.77 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0855, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(1.0466, device='cuda:0', grad_fn=) [2024-06-19 00:33:58,051] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.85 [2024-06-19 00:33:58,051] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3549.93 | bwd_microstep: 1877.96 | bwd_inner_microstep: 1872.37 | bwd_allreduce_microstep: 5.49 | step_microstep: 61.58 [2024-06-19 00:33:58,052] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6439.63 | bwd: 3625.29 | bwd_inner: 3614.83 | bwd_allreduce: 10.26 | step: 61.66 53%|█████▎ | 106/200 [19:10<18:29, 11.80s/it] {'loss': 1.0521, 'learning_rate': 4.7571888894277604e-05, 'epoch': 1.06} 53%|█████▎ | 106/200 [19:10<18:29, 11.80s/it]warning: The size of tensor a (0) must match the size of tensor b (1280) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1280, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.1453, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.1889, device='cuda:0', grad_fn=) [2024-06-19 00:34:02,876] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3123.53 | bwd_microstep: 1617.77 | bwd_inner_microstep: 1612.88 | bwd_allreduce_microstep: 4.79 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2101, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1469, device='cuda:0', grad_fn=) [2024-06-19 00:34:07,260] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.88 [2024-06-19 00:34:07,261] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2641.00 | bwd_microstep: 1599.12 | bwd_inner_microstep: 1593.67 | bwd_allreduce_microstep: 5.35 | step_microstep: 60.82 [2024-06-19 00:34:07,261] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 5764.52 | bwd: 3216.89 | bwd_inner: 3206.56 | bwd_allreduce: 10.15 | step: 60.90 54%|█████▎ | 107/200 [19:19<17:05, 11.02s/it] {'loss': 0.6679, 'learning_rate': 4.676350908127822e-05, 'epoch': 1.07} 54%|█████▎ | 107/200 [19:19<17:05, 11.02s/it]warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1301, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.0753, device='cuda:0', grad_fn=) [2024-06-19 00:34:11,587] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2638.94 | bwd_microstep: 1598.12 | bwd_inner_microstep: 1593.26 | bwd_allreduce_microstep: 4.75 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0175, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(0.0854, device='cuda:0', grad_fn=) [2024-06-19 00:34:16,838] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:34:16,838] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3433.15 | bwd_microstep: 1678.68 | bwd_inner_microstep: 1673.33 | bwd_allreduce_microstep: 5.24 | step_microstep: 60.19 [2024-06-19 00:34:16,839] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6072.07 | bwd: 3276.79 | bwd_inner: 3266.61 | bwd_allreduce: 10.00 | step: 60.27 54%|█████▍ | 108/200 [19:29<16:14, 10.59s/it] {'loss': 0.5803, 'learning_rate': 4.59559779819298e-05, 'epoch': 1.08} 54%|█████▍ | 108/200 [19:29<16:14, 10.59s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0849, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.1345, device='cuda:0', grad_fn=) [2024-06-19 00:34:22,041] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3435.45 | bwd_microstep: 1681.76 | bwd_inner_microstep: 1676.84 | bwd_allreduce_microstep: 4.81 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1635, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(1.1168, device='cuda:0', grad_fn=) [2024-06-19 00:34:27,473] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:34:27,474] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3486.77 | bwd_microstep: 1796.92 | bwd_inner_microstep: 1791.15 | bwd_allreduce_microstep: 5.66 | step_microstep: 62.25 [2024-06-19 00:34:27,474] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6922.20 | bwd: 3478.68 | bwd_inner: 3467.99 | bwd_allreduce: 10.48 | step: 62.33 55%|█████▍ | 109/200 [19:40<16:04, 10.60s/it] {'loss': 0.6257, 'learning_rate': 4.51495073572676e-05, 'epoch': 1.09} 55%|█████▍ | 109/200 [19:40<16:04, 10.60s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1463, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0895, device='cuda:0', grad_fn=) [2024-06-19 00:34:33,044] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3548.03 | bwd_microstep: 1919.61 | bwd_inner_microstep: 1914.62 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2266, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1617, device='cuda:0', grad_fn=) [2024-06-19 00:34:38,648] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:34:38,649] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3547.92 | bwd_microstep: 1899.89 | bwd_inner_microstep: 1894.54 | bwd_allreduce_microstep: 5.27 | step_microstep: 60.55 [2024-06-19 00:34:38,649] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7095.93 | bwd: 3819.49 | bwd_inner: 3809.20 | bwd_allreduce: 10.06 | step: 60.63 55%|█████▌ | 110/200 [19:51<16:09, 10.77s/it] {'loss': 1.1256, 'learning_rate': 4.434430869023579e-05, 'epoch': 1.1} 55%|█████▌ | 110/200 [19:51<16:09, 10.77s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1179, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0640, device='cuda:0', grad_fn=) [2024-06-19 00:34:44,039] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3491.37 | bwd_microstep: 1806.84 | bwd_inner_microstep: 1802.02 | bwd_allreduce_microstep: 4.75 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1608, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.1032, device='cuda:0', grad_fn=) [2024-06-19 00:34:49,528] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.83 [2024-06-19 00:34:49,529] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3510.74 | bwd_microstep: 1825.75 | bwd_inner_microstep: 1820.14 | bwd_allreduce_microstep: 5.48 | step_microstep: 60.69 [2024-06-19 00:34:49,530] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7002.10 | bwd: 3632.59 | bwd_inner: 3622.19 | bwd_allreduce: 10.24 | step: 60.77 56%|█████▌ | 111/200 [20:02<16:01, 10.81s/it] {'loss': 1.0836, 'learning_rate': 4.35405931302297e-05, 'epoch': 1.11} 56%|█████▌ | 111/200 [20:02<16:01, 10.81s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1675, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1086, device='cuda:0', grad_fn=) [2024-06-19 00:34:54,922] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3494.60 | bwd_microstep: 1805.20 | bwd_inner_microstep: 1800.21 | bwd_allreduce_microstep: 4.81 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9415, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.9052, device='cuda:0', grad_fn=) [2024-06-19 00:35:00,509] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.88 [2024-06-19 00:35:00,510] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3545.54 | bwd_microstep: 1888.57 | bwd_inner_microstep: 1882.92 | bwd_allreduce_microstep: 5.47 | step_microstep: 60.61 [2024-06-19 00:35:00,510] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7040.12 | bwd: 3693.76 | bwd_inner: 3683.22 | bwd_allreduce: 10.24 | step: 60.69 56%|█████▌ | 112/200 [20:13<15:55, 10.86s/it] {'loss': 1.0069, 'learning_rate': 4.27385714377255e-05, 'epoch': 1.12} 56%|█████▌ | 112/200 [20:13<15:55, 10.86s/it]warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0246, device='cuda:0', grad_fn=) tensor(0.7006, device='cuda:0', grad_fn=) tensor(0.0922, device='cuda:0', grad_fn=) [2024-06-19 00:35:05,040] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2831.91 | bwd_microstep: 1614.38 | bwd_inner_microstep: 1609.51 | bwd_allreduce_microstep: 4.76 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0164, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.0726, device='cuda:0', grad_fn=) [2024-06-19 00:35:10,308] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.85 [2024-06-19 00:35:10,309] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3439.83 | bwd_microstep: 1684.64 | bwd_inner_microstep: 1679.17 | bwd_allreduce_microstep: 5.36 | step_microstep: 60.42 [2024-06-19 00:35:10,309] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6271.74 | bwd: 3299.01 | bwd_inner: 3288.69 | bwd_allreduce: 10.14 | step: 60.50 56%|█████▋ | 113/200 [20:22<15:17, 10.54s/it] {'loss': 0.0824, 'learning_rate': 4.193845392901201e-05, 'epoch': 1.13} 56%|█████▋ | 113/200 [20:22<15:17, 10.54s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9526, device='cuda:0', grad_fn=) tensor(0.7006, device='cuda:0', grad_fn=) tensor(0.9274, device='cuda:0', grad_fn=) [2024-06-19 00:35:15,699] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3494.36 | bwd_microstep: 1804.27 | bwd_inner_microstep: 1799.40 | bwd_allreduce_microstep: 4.76 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(0.8762, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.8467, device='cuda:0', grad_fn=) [2024-06-19 00:35:20,268] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:35:20,268] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2828.06 | bwd_microstep: 1598.91 | bwd_inner_microstep: 1593.48 | bwd_allreduce_microstep: 5.32 | step_microstep: 60.21 [2024-06-19 00:35:20,269] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6322.40 | bwd: 3403.17 | bwd_inner: 3392.89 | bwd_allreduce: 10.09 | step: 60.29 57%|█████▋ | 114/200 [20:32<14:51, 10.37s/it] {'loss': 0.887, 'learning_rate': 4.114045042103887e-05, 'epoch': 1.14} 57%|█████▋ | 114/200 [20:32<14:51, 10.37s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.3005, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.2286, device='cuda:0', grad_fn=) [2024-06-19 00:35:25,777] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3538.16 | bwd_microstep: 1877.01 | bwd_inner_microstep: 1872.02 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1280) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1280, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.7770, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.7571, device='cuda:0', grad_fn=) [2024-06-19 00:35:30,897] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:35:30,898] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3192.50 | bwd_microstep: 1776.32 | bwd_inner_microstep: 1770.87 | bwd_allreduce_microstep: 5.34 | step_microstep: 60.48 [2024-06-19 00:35:30,898] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6730.64 | bwd: 3653.32 | bwd_inner: 3642.94 | bwd_allreduce: 10.13 | step: 60.56 57%|█████▊ | 115/200 [20:43<14:47, 10.45s/it] {'loss': 0.9928, 'learning_rate': 4.0344770176395606e-05, 'epoch': 1.15} 57%|█████▊ | 115/200 [20:43<14:47, 10.45s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0081, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(0.0769, device='cuda:0', grad_fn=) [2024-06-19 00:35:36,096] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3434.87 | bwd_microstep: 1680.16 | bwd_inner_microstep: 1675.25 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2262, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1614, device='cuda:0', grad_fn=) [2024-06-19 00:35:40,914] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:35:40,915] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2899.95 | bwd_microstep: 1763.93 | bwd_inner_microstep: 1758.26 | bwd_allreduce_microstep: 5.49 | step_microstep: 60.85 [2024-06-19 00:35:40,915] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6334.80 | bwd: 3444.08 | bwd_inner: 3433.56 | bwd_allreduce: 10.30 | step: 60.93 58%|█████▊ | 116/200 [20:53<14:26, 10.32s/it] {'loss': 0.6192, 'learning_rate': 3.955162184843625e-05, 'epoch': 1.16} 58%|█████▊ | 116/200 [20:53<14:26, 10.32s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9686, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(0.9414, device='cuda:0', grad_fn=) [2024-06-19 00:35:46,308] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3492.90 | bwd_microstep: 1807.34 | bwd_inner_microstep: 1802.34 | bwd_allreduce_microstep: 4.84 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0776, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0276, device='cuda:0', grad_fn=) [2024-06-19 00:35:51,019] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.83 [2024-06-19 00:35:51,019] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2748.37 | bwd_microstep: 1805.02 | bwd_inner_microstep: 1799.48 | bwd_allreduce_microstep: 5.36 | step_microstep: 60.60 [2024-06-19 00:35:51,020] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6241.25 | bwd: 3612.35 | bwd_inner: 3601.91 | bwd_allreduce: 10.19 | step: 60.68 58%|█████▊ | 117/200 [21:03<14:11, 10.25s/it] {'loss': 0.9845, 'learning_rate': 3.876121342656355e-05, 'epoch': 1.17} 58%|█████▊ | 117/200 [21:03<14:11, 10.25s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2928, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.2213, device='cuda:0', grad_fn=) [2024-06-19 00:35:56,541] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3540.12 | bwd_microstep: 1885.60 | bwd_inner_microstep: 1880.68 | bwd_allreduce_microstep: 4.77 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.8329, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.8078, device='cuda:0', grad_fn=) [2024-06-19 00:36:01,246] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.87 [2024-06-19 00:36:01,246] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2747.29 | bwd_microstep: 1805.22 | bwd_inner_microstep: 1799.69 | bwd_allreduce_microstep: 5.41 | step_microstep: 60.62 [2024-06-19 00:36:01,247] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6287.40 | bwd: 3690.81 | bwd_inner: 3680.43 | bwd_allreduce: 10.19 | step: 60.70 59%|█████▉ | 118/200 [21:13<14:00, 10.25s/it] {'loss': 1.0146, 'learning_rate': 3.7973752181687335e-05, 'epoch': 1.18} 59%|█████▉ | 118/200 [21:13<14:00, 10.25s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0870, device='cuda:0', grad_fn=) tensor(0.8190, device='cuda:0', grad_fn=) tensor(0.1602, device='cuda:0', grad_fn=) [2024-06-19 00:36:06,408] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3428.49 | bwd_microstep: 1652.10 | bwd_inner_microstep: 1647.02 | bwd_allreduce_microstep: 4.97 | step_microstep: 0.09 warning: The size of tensor a (0) must match the size of tensor b (1280) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1280, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1746, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1149, device='cuda:0', grad_fn=) [2024-06-19 00:36:11,364] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.77 [2024-06-19 00:36:11,364] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3140.26 | bwd_microstep: 1661.36 | bwd_inner_microstep: 1655.91 | bwd_allreduce_microstep: 5.34 | step_microstep: 60.67 [2024-06-19 00:36:11,365] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6568.72 | bwd: 3313.45 | bwd_inner: 3302.95 | bwd_allreduce: 10.32 | step: 60.76 60%|█████▉ | 119/200 [21:23<13:46, 10.21s/it] {'loss': 0.6376, 'learning_rate': 3.718944461187138e-05, 'epoch': 1.19} 60%|█████▉ | 119/200 [21:23<13:46, 10.21s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.3662, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(1.2992, device='cuda:0', grad_fn=) [2024-06-19 00:36:16,885] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3543.74 | bwd_microstep: 1876.65 | bwd_inner_microstep: 1871.69 | bwd_allreduce_microstep: 4.78 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0670, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(1.0299, device='cuda:0', grad_fn=) [2024-06-19 00:36:22,337] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:36:22,337] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3498.62 | bwd_microstep: 1806.51 | bwd_inner_microstep: 1801.05 | bwd_allreduce_microstep: 5.36 | step_microstep: 60.28 [2024-06-19 00:36:22,338] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7042.34 | bwd: 3683.15 | bwd_inner: 3672.79 | bwd_allreduce: 10.12 | step: 60.36 60%|██████ | 120/200 [21:34<13:54, 10.44s/it] {'loss': 1.1646, 'learning_rate': 3.640849638818286e-05, 'epoch': 1.2} 60%|██████ | 120/200 [21:34<13:54, 10.44s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.3422, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.3664, device='cuda:0', grad_fn=) [2024-06-19 00:36:27,506] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3432.06 | bwd_microstep: 1653.67 | bwd_inner_microstep: 1648.54 | bwd_allreduce_microstep: 5.01 | step_microstep: 0.09 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9998, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.9580, device='cuda:0', grad_fn=) [2024-06-19 00:36:33,102] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:36:33,102] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3547.04 | bwd_microstep: 1891.64 | bwd_inner_microstep: 1886.20 | bwd_allreduce_microstep: 5.34 | step_microstep: 60.64 [2024-06-19 00:36:33,103] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6979.08 | bwd: 3545.31 | bwd_inner: 3534.76 | bwd_allreduce: 10.36 | step: 60.74 60%|██████ | 121/200 [21:45<13:52, 10.54s/it] {'loss': 0.6622, 'learning_rate': 3.5631112300758595e-05, 'epoch': 1.21} 60%|██████ | 121/200 [21:45<13:52, 10.54s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0308, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0859, device='cuda:0', grad_fn=) [2024-06-19 00:36:38,301] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3436.28 | bwd_microstep: 1678.15 | bwd_inner_microstep: 1673.15 | bwd_allreduce_microstep: 4.83 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.1833, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(0.2349, device='cuda:0', grad_fn=) [2024-06-19 00:36:43,526] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.86 [2024-06-19 00:36:43,526] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3431.00 | bwd_microstep: 1654.07 | bwd_inner_microstep: 1648.57 | bwd_allreduce_microstep: 5.39 | step_microstep: 62.06 [2024-06-19 00:36:43,527] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6867.26 | bwd: 3332.22 | bwd_inner: 3321.77 | bwd_allreduce: 10.20 | step: 62.14 61%|██████ | 122/200 [21:56<13:39, 10.50s/it] {'loss': 0.1604, 'learning_rate': 3.4857496205102474e-05, 'epoch': 1.22} 61%|██████ | 122/200 [21:56<13:39, 10.50s/it]warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0056, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0632, device='cuda:0', grad_fn=) [2024-06-19 00:36:48,059] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.59 | bwd_microstep: 1615.27 | bwd_inner_microstep: 1610.18 | bwd_allreduce_microstep: 4.98 | step_microstep: 0.14 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0117, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.9684, device='cuda:0', grad_fn=) [2024-06-19 00:36:53,647] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:36:53,647] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3544.21 | bwd_microstep: 1887.30 | bwd_inner_microstep: 1881.89 | bwd_allreduce_microstep: 5.30 | step_microstep: 60.63 [2024-06-19 00:36:53,648] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6377.78 | bwd: 3502.57 | bwd_inner: 3492.09 | bwd_allreduce: 10.29 | step: 60.78 62%|██████▏ | 123/200 [22:06<13:19, 10.39s/it] {'loss': 0.5158, 'learning_rate': 3.408785096862782e-05, 'epoch': 1.23} 62%|██████▏ | 123/200 [22:06<13:19, 10.39s/it]warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(0.0031, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(0.0724, device='cuda:0', grad_fn=) [2024-06-19 00:36:56,416] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 1881.27 | bwd_microstep: 829.18 | bwd_inner_microstep: 824.31 | bwd_allreduce_microstep: 4.76 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1280) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1280, 6144]) tensor(1.2689, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1998, device='cuda:0', grad_fn=) [2024-06-19 00:37:01,551] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.83 [2024-06-19 00:37:01,551] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3194.96 | bwd_microstep: 1788.27 | bwd_inner_microstep: 1782.68 | bwd_allreduce_microstep: 5.42 | step_microstep: 60.61 [2024-06-19 00:37:01,552] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 5076.21 | bwd: 2617.44 | bwd_inner: 2607.03 | bwd_allreduce: 10.17 | step: 60.68 62%|██████▏ | 124/200 [22:14<12:12, 9.64s/it] {'loss': 0.6361, 'learning_rate': 3.332237841745898e-05, 'epoch': 1.24} 62%|██████▏ | 124/200 [22:14<12:12, 9.64s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2317, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.1667, device='cuda:0', grad_fn=) [2024-06-19 00:37:07,061] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3540.69 | bwd_microstep: 1875.94 | bwd_inner_microstep: 1870.76 | bwd_allreduce_microstep: 5.08 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2550, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.1879, device='cuda:0', grad_fn=) [2024-06-19 00:37:12,642] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:37:12,642] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3546.14 | bwd_microstep: 1882.90 | bwd_inner_microstep: 1877.35 | bwd_allreduce_microstep: 5.38 | step_microstep: 60.45 [2024-06-19 00:37:12,643] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7086.82 | bwd: 3758.83 | bwd_inner: 3748.15 | bwd_allreduce: 10.44 | step: 60.53 62%|██████▎ | 125/200 [22:25<12:35, 10.08s/it] {'loss': 1.1773, 'learning_rate': 3.2561279283505883e-05, 'epoch': 1.25} 62%|██████▎ | 125/200 [22:25<12:35, 10.08s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(1.1860, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1252, device='cuda:0', grad_fn=) [2024-06-19 00:37:17,385] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2892.62 | bwd_microstep: 1755.13 | bwd_inner_microstep: 1750.22 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1307, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0755, device='cuda:0', grad_fn=) [2024-06-19 00:37:22,866] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.85 [2024-06-19 00:37:22,867] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3502.85 | bwd_microstep: 1824.06 | bwd_inner_microstep: 1818.43 | bwd_allreduce_microstep: 5.55 | step_microstep: 60.90 [2024-06-19 00:37:22,867] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6395.46 | bwd: 3579.19 | bwd_inner: 3568.65 | bwd_allreduce: 10.36 | step: 60.97 63%|██████▎ | 126/200 [22:35<12:28, 10.12s/it] {'loss': 1.1003, 'learning_rate': 3.180475315182563e-05, 'epoch': 1.26} 63%|██████▎ | 126/200 [22:35<12:28, 10.12s/it]warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(0.0868, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(0.1481, device='cuda:0', grad_fn=) [2024-06-19 00:37:24,848] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 1435.38 | bwd_microstep: 501.19 | bwd_inner_microstep: 496.22 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1280) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1280, 6144]) tensor(0.9115, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.8781, device='cuda:0', grad_fn=) [2024-06-19 00:37:29,947] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.88 [2024-06-19 00:37:29,947] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3182.95 | bwd_microstep: 1768.74 | bwd_inner_microstep: 1763.22 | bwd_allreduce_microstep: 5.35 | step_microstep: 60.64 [2024-06-19 00:37:29,948] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 4618.31 | bwd: 2269.92 | bwd_inner: 2259.52 | bwd_allreduce: 10.12 | step: 60.72 64%|██████▎ | 127/200 [22:42<11:12, 9.21s/it] {'loss': 0.5131, 'learning_rate': 3.105299840828466e-05, 'epoch': 1.27} 64%|██████▎ | 127/200 [22:42<11:12, 9.21s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0946, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.0433, device='cuda:0', grad_fn=) [2024-06-19 00:37:35,456] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3537.70 | bwd_microstep: 1876.43 | bwd_inner_microstep: 1871.59 | bwd_allreduce_microstep: 4.76 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.8590, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.8310, device='cuda:0', grad_fn=) [2024-06-19 00:37:40,901] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.86 [2024-06-19 00:37:40,901] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3493.35 | bwd_microstep: 1803.51 | bwd_inner_microstep: 1797.94 | bwd_allreduce_microstep: 5.39 | step_microstep: 60.63 [2024-06-19 00:37:40,902] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7031.03 | bwd: 3679.93 | bwd_inner: 3669.56 | bwd_allreduce: 10.14 | step: 60.71 64%|██████▍ | 128/200 [22:53<11:40, 9.73s/it] {'loss': 0.9371, 'learning_rate': 3.0306212187535653e-05, 'epoch': 1.28} 64%|██████▍ | 128/200 [22:53<11:40, 9.73s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.7700, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(0.7627, device='cuda:0', grad_fn=) [2024-06-19 00:37:46,409] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3536.46 | bwd_microstep: 1875.88 | bwd_inner_microstep: 1870.79 | bwd_allreduce_microstep: 4.99 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0047, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0624, device='cuda:0', grad_fn=) [2024-06-19 00:37:51,669] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.79 [2024-06-19 00:37:51,670] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3438.21 | bwd_microstep: 1681.29 | bwd_inner_microstep: 1675.87 | bwd_allreduce_microstep: 5.30 | step_microstep: 59.75 [2024-06-19 00:37:51,671] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6974.65 | bwd: 3557.17 | bwd_inner: 3546.68 | bwd_allreduce: 10.30 | step: 59.84 64%|██████▍ | 129/200 [23:04<11:53, 10.04s/it] {'loss': 0.4125, 'learning_rate': 2.9564590321322207e-05, 'epoch': 1.29} 64%|██████▍ | 129/200 [23:04<11:53, 10.04s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0993, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0472, device='cuda:0', grad_fn=) [2024-06-19 00:37:57,177] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3537.28 | bwd_microstep: 1875.52 | bwd_inner_microstep: 1870.58 | bwd_allreduce_microstep: 4.77 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0697, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.0209, device='cuda:0', grad_fn=) [2024-06-19 00:38:02,790] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:38:02,790] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3551.80 | bwd_microstep: 1902.76 | bwd_inner_microstep: 1897.32 | bwd_allreduce_microstep: 5.33 | step_microstep: 60.19 [2024-06-19 00:38:02,791] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7089.06 | bwd: 3778.27 | bwd_inner: 3767.95 | bwd_allreduce: 10.08 | step: 60.27 65%|██████▌ | 130/200 [23:15<12:05, 10.37s/it] {'loss': 1.034, 'learning_rate': 2.882832728712551e-05, 'epoch': 1.3} 65%|██████▌ | 130/200 [23:15<12:05, 10.37s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0010, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.0594, device='cuda:0', grad_fn=) [2024-06-19 00:38:07,997] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3439.22 | bwd_microstep: 1682.34 | bwd_inner_microstep: 1677.18 | bwd_allreduce_microstep: 5.04 | step_microstep: 0.09 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1921, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1307, device='cuda:0', grad_fn=) [2024-06-19 00:38:13,578] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.83 [2024-06-19 00:38:13,579] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3544.21 | bwd_microstep: 1884.58 | bwd_inner_microstep: 1879.20 | bwd_allreduce_microstep: 5.26 | step_microstep: 60.32 [2024-06-19 00:38:13,579] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6983.41 | bwd: 3566.92 | bwd_inner: 3556.41 | bwd_allreduce: 10.31 | step: 60.42 66%|██████▌ | 131/200 [23:26<12:04, 10.49s/it] {'loss': 0.5951, 'learning_rate': 2.8097616157165883e-05, 'epoch': 1.31} 66%|██████▌ | 131/200 [23:26<12:04, 10.49s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0803, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(0.1419, device='cuda:0', grad_fn=) [2024-06-19 00:38:18,776] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3435.51 | bwd_microstep: 1677.33 | bwd_inner_microstep: 1672.48 | bwd_allreduce_microstep: 4.74 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.4910, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.5004, device='cuda:0', grad_fn=) [2024-06-19 00:38:24,334] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.88 [2024-06-19 00:38:24,335] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3538.39 | bwd_microstep: 1872.16 | bwd_inner_microstep: 1866.70 | bwd_allreduce_microstep: 5.36 | step_microstep: 60.24 [2024-06-19 00:38:24,336] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6973.88 | bwd: 3549.49 | bwd_inner: 3539.19 | bwd_allreduce: 10.11 | step: 60.32 66%|██████▌ | 132/200 [23:36<11:58, 10.57s/it] {'loss': 0.3212, 'learning_rate': 2.737264854777306e-05, 'epoch': 1.32} 66%|██████▌ | 132/200 [23:36<11:58, 10.57s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0061, device='cuda:0', grad_fn=) tensor(0.7039, device='cuda:0', grad_fn=) tensor(0.0759, device='cuda:0', grad_fn=) [2024-06-19 00:38:29,542] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3444.21 | bwd_microstep: 1678.03 | bwd_inner_microstep: 1672.93 | bwd_allreduce_microstep: 4.92 | step_microstep: 0.14 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2807, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.2108, device='cuda:0', grad_fn=) [2024-06-19 00:38:34,121] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:38:34,121] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2705.26 | bwd_microstep: 1725.30 | bwd_inner_microstep: 1719.85 | bwd_allreduce_microstep: 5.29 | step_microstep: 60.15 [2024-06-19 00:38:34,122] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6149.46 | bwd: 3403.33 | bwd_inner: 3392.88 | bwd_allreduce: 10.21 | step: 60.29 66%|██████▋ | 133/200 [23:46<11:32, 10.34s/it] {'loss': 0.6433, 'learning_rate': 2.6653614569137968e-05, 'epoch': 1.33} 66%|██████▋ | 133/200 [23:46<11:32, 10.34s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0056, device='cuda:0', grad_fn=) tensor(0.7006, device='cuda:0', grad_fn=) tensor(0.0751, device='cuda:0', grad_fn=) [2024-06-19 00:38:39,325] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3438.46 | bwd_microstep: 1680.20 | bwd_inner_microstep: 1675.29 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.4083, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.3260, device='cuda:0', grad_fn=) [2024-06-19 00:38:44,781] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:38:44,781] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3498.41 | bwd_microstep: 1809.60 | bwd_inner_microstep: 1804.21 | bwd_allreduce_microstep: 5.28 | step_microstep: 60.31 [2024-06-19 00:38:44,782] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6936.86 | bwd: 3489.80 | bwd_inner: 3479.52 | bwd_allreduce: 10.09 | step: 60.40 67%|██████▋ | 134/200 [23:57<11:28, 10.43s/it] {'loss': 0.7006, 'learning_rate': 2.5940702775459747e-05, 'epoch': 1.34} 67%|██████▋ | 134/200 [23:57<11:28, 10.43s/it]warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9569, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.9190, device='cuda:0', grad_fn=) [2024-06-19 00:38:49,638] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2938.99 | bwd_microstep: 1818.99 | bwd_inner_microstep: 1814.08 | bwd_allreduce_microstep: 4.81 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(1.3391, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.2630, device='cuda:0', grad_fn=) [2024-06-19 00:38:54,361] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:38:54,362] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2753.91 | bwd_microstep: 1811.56 | bwd_inner_microstep: 1806.13 | bwd_allreduce_microstep: 5.32 | step_microstep: 60.27 [2024-06-19 00:38:54,362] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 5692.88 | bwd: 3630.54 | bwd_inner: 3620.22 | bwd_allreduce: 10.14 | step: 60.35 68%|██████▊ | 135/200 [24:06<11:01, 10.18s/it] {'loss': 1.091, 'learning_rate': 2.5234100115500643e-05, 'epoch': 1.35} 68%|██████▊ | 135/200 [24:06<11:01, 10.18s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.1349, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.1793, device='cuda:0', grad_fn=) [2024-06-19 00:38:59,569] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3440.67 | bwd_microstep: 1681.25 | bwd_inner_microstep: 1676.39 | bwd_allreduce_microstep: 4.79 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.7350, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.7196, device='cuda:0', grad_fn=) [2024-06-19 00:39:05,019] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.85 [2024-06-19 00:39:05,019] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3498.04 | bwd_microstep: 1804.96 | bwd_inner_microstep: 1799.35 | bwd_allreduce_microstep: 5.51 | step_microstep: 60.52 [2024-06-19 00:39:05,020] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6938.70 | bwd: 3486.21 | bwd_inner: 3475.73 | bwd_allreduce: 10.31 | step: 60.59 68%|██████▊ | 136/200 [24:17<11:00, 10.32s/it] {'loss': 0.4494, 'learning_rate': 2.4533991883561868e-05, 'epoch': 1.36} 68%|██████▊ | 136/200 [24:17<11:00, 10.32s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0063, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.0635, device='cuda:0', grad_fn=) [2024-06-19 00:39:10,145] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3410.88 | bwd_microstep: 1629.48 | bwd_inner_microstep: 1624.56 | bwd_allreduce_microstep: 4.82 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.8095, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(0.7982, device='cuda:0', grad_fn=) [2024-06-19 00:39:15,604] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:39:15,605] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3501.21 | bwd_microstep: 1809.04 | bwd_inner_microstep: 1803.69 | bwd_allreduce_microstep: 5.28 | step_microstep: 60.30 [2024-06-19 00:39:15,605] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6912.07 | bwd: 3438.52 | bwd_inner: 3428.24 | bwd_allreduce: 10.11 | step: 60.38 68%|██████▊ | 137/200 [24:28<10:55, 10.40s/it] {'loss': 0.4309, 'learning_rate': 2.3840561670893496e-05, 'epoch': 1.37} 68%|██████▊ | 137/200 [24:28<10:55, 10.40s/it]warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.7922, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.7712, device='cuda:0', grad_fn=) [2024-06-19 00:39:20,262] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2753.84 | bwd_microstep: 1806.49 | bwd_inner_microstep: 1801.42 | bwd_allreduce_microstep: 4.89 | step_microstep: 0.14 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1908, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1295, device='cuda:0', grad_fn=) [2024-06-19 00:39:25,981] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.83 [2024-06-19 00:39:25,982] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3581.61 | bwd_microstep: 1971.59 | bwd_inner_microstep: 1966.05 | bwd_allreduce_microstep: 5.43 | step_microstep: 61.01 [2024-06-19 00:39:25,982] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6335.44 | bwd: 3778.07 | bwd_inner: 3767.52 | bwd_allreduce: 10.33 | step: 61.16 69%|██████▉ | 138/200 [24:38<10:44, 10.39s/it] {'loss': 0.9503, 'learning_rate': 2.315399131755081e-05, 'epoch': 1.38} 69%|██████▉ | 138/200 [24:38<10:44, 10.39s/it]warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0678, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(1.0306, device='cuda:0', grad_fn=) [2024-06-19 00:39:30,489] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2697.28 | bwd_microstep: 1720.34 | bwd_inner_microstep: 1715.46 | bwd_allreduce_microstep: 4.77 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.3834, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.3028, device='cuda:0', grad_fn=) [2024-06-19 00:39:36,200] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:39:36,201] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3582.32 | bwd_microstep: 1968.17 | bwd_inner_microstep: 1962.59 | bwd_allreduce_microstep: 5.40 | step_microstep: 60.59 [2024-06-19 00:39:36,202] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6279.58 | bwd: 3688.50 | bwd_inner: 3678.10 | bwd_allreduce: 10.16 | step: 60.67 70%|██████▉ | 139/200 [24:48<10:30, 10.34s/it] {'loss': 1.1667, 'learning_rate': 2.2474460864709824e-05, 'epoch': 1.39} 70%|██████▉ | 139/200 [24:48<10:30, 10.34s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.8518, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.8245, device='cuda:0', grad_fn=) [2024-06-19 00:39:41,615] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3505.21 | bwd_microstep: 1814.95 | bwd_inner_microstep: 1810.05 | bwd_allreduce_microstep: 4.79 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1280) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1280, 6144]) tensor(1.5815, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.4811, device='cuda:0', grad_fn=) [2024-06-19 00:39:46,564] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:39:46,565] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3138.87 | bwd_microstep: 1662.54 | bwd_inner_microstep: 1657.12 | bwd_allreduce_microstep: 5.32 | step_microstep: 60.16 [2024-06-19 00:39:46,565] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6644.07 | bwd: 3477.48 | bwd_inner: 3467.18 | bwd_allreduce: 10.12 | step: 60.24 70%|███████ | 140/200 [24:59<10:20, 10.35s/it] {'loss': 1.1528, 'learning_rate': 2.180214850745467e-05, 'epoch': 1.4} 70%|███████ | 140/200 [24:59<10:20, 10.35s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(1.0776, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.0280, device='cuda:0', grad_fn=) [2024-06-19 00:39:51,094] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2708.62 | bwd_microstep: 1728.99 | bwd_inner_microstep: 1723.91 | bwd_allreduce_microstep: 4.97 | step_microstep: 0.09 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0057, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(0.0747, device='cuda:0', grad_fn=) [2024-06-19 00:39:56,353] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.83 [2024-06-19 00:39:56,353] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3439.22 | bwd_microstep: 1677.92 | bwd_inner_microstep: 1672.48 | bwd_allreduce_microstep: 5.34 | step_microstep: 60.99 [2024-06-19 00:39:56,354] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6147.83 | bwd: 3406.90 | bwd_inner: 3396.40 | bwd_allreduce: 10.31 | step: 61.08 70%|███████ | 141/200 [25:08<10:00, 10.18s/it] {'loss': 0.5514, 'learning_rate': 2.1137230548049043e-05, 'epoch': 1.41} 70%|███████ | 141/200 [25:08<10:00, 10.18s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0767, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.0275, device='cuda:0', grad_fn=) [2024-06-19 00:40:01,893] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3547.92 | bwd_microstep: 1892.11 | bwd_inner_microstep: 1887.06 | bwd_allreduce_microstep: 4.88 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0394, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.9933, device='cuda:0', grad_fn=) [2024-06-19 00:40:07,356] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.83 [2024-06-19 00:40:07,357] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3502.27 | bwd_microstep: 1812.81 | bwd_inner_microstep: 1807.30 | bwd_allreduce_microstep: 5.34 | step_microstep: 60.19 [2024-06-19 00:40:07,357] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7050.17 | bwd: 3704.92 | bwd_inner: 3694.44 | bwd_allreduce: 10.19 | step: 60.27 71%|███████ | 142/200 [25:19<10:04, 10.43s/it] {'loss': 1.0104, 'learning_rate': 2.0479881349703883e-05, 'epoch': 1.42} 71%|███████ | 142/200 [25:19<10:04, 10.43s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.4469, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.3600, device='cuda:0', grad_fn=) [2024-06-19 00:40:12,885] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3546.01 | bwd_microstep: 1885.85 | bwd_inner_microstep: 1880.84 | bwd_allreduce_microstep: 4.83 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(1.1395, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0834, device='cuda:0', grad_fn=) [2024-06-19 00:40:17,581] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:40:17,582] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2746.34 | bwd_microstep: 1797.02 | bwd_inner_microstep: 1791.59 | bwd_allreduce_microstep: 5.31 | step_microstep: 60.35 [2024-06-19 00:40:17,582] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6292.34 | bwd: 3682.86 | bwd_inner: 3672.49 | bwd_allreduce: 10.15 | step: 60.43 72%|███████▏ | 143/200 [25:30<09:50, 10.37s/it] {'loss': 1.2217, 'learning_rate': 1.983027329085377e-05, 'epoch': 1.43} 72%|███████▏ | 143/200 [25:30<09:50, 10.37s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.4314, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(1.3579, device='cuda:0', grad_fn=) [2024-06-19 00:40:22,964] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3496.52 | bwd_microstep: 1795.88 | bwd_inner_microstep: 1790.96 | bwd_allreduce_microstep: 4.75 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(1.0751, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0254, device='cuda:0', grad_fn=) [2024-06-19 00:40:27,663] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:40:27,663] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2745.41 | bwd_microstep: 1799.77 | bwd_inner_microstep: 1794.33 | bwd_allreduce_microstep: 5.33 | step_microstep: 59.99 [2024-06-19 00:40:27,664] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6241.91 | bwd: 3595.65 | bwd_inner: 3585.34 | bwd_allreduce: 10.09 | step: 60.07 72%|███████▏ | 144/200 [25:40<09:35, 10.28s/it] {'loss': 1.1916, 'learning_rate': 1.9188576719953633e-05, 'epoch': 1.44} 72%|███████▏ | 144/200 [25:40<09:35, 10.28s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0079, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.0649, device='cuda:0', grad_fn=) [2024-06-19 00:40:32,794] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3415.92 | bwd_microstep: 1628.84 | bwd_inner_microstep: 1623.81 | bwd_allreduce_microstep: 4.86 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0344, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.9887, device='cuda:0', grad_fn=) [2024-06-19 00:40:37,382] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:40:37,383] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2706.31 | bwd_microstep: 1730.17 | bwd_inner_microstep: 1724.63 | bwd_allreduce_microstep: 5.37 | step_microstep: 60.10 [2024-06-19 00:40:37,383] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6122.21 | bwd: 3359.01 | bwd_inner: 3348.53 | bwd_allreduce: 10.21 | step: 60.17 72%|███████▎ | 145/200 [25:50<09:16, 10.11s/it] {'loss': 0.5268, 'learning_rate': 1.8554959910807775e-05, 'epoch': 1.45} 72%|███████▎ | 145/200 [25:50<09:16, 10.11s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(0.2541, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(0.2984, device='cuda:0', grad_fn=) [2024-06-19 00:40:41,891] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2833.28 | bwd_microstep: 1592.71 | bwd_inner_microstep: 1587.65 | bwd_allreduce_microstep: 4.95 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1024) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1024, 6144]) tensor(1.0408, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(1.0064, device='cuda:0', grad_fn=) [2024-06-19 00:40:46,630] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:40:46,631] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2975.65 | bwd_microstep: 1619.15 | bwd_inner_microstep: 1613.72 | bwd_allreduce_microstep: 5.32 | step_microstep: 60.53 [2024-06-19 00:40:46,631] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 5808.92 | bwd: 3211.85 | bwd_inner: 3201.37 | bwd_allreduce: 10.28 | step: 60.62 73%|███████▎ | 146/200 [25:59<08:52, 9.85s/it] {'loss': 0.6524, 'learning_rate': 1.7929589018443016e-05, 'epoch': 1.46} 73%|███████▎ | 146/200 [25:59<08:52, 9.85s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(1.0077, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.9654, device='cuda:0', grad_fn=) [2024-06-19 00:40:51,150] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2699.58 | bwd_microstep: 1728.00 | bwd_inner_microstep: 1723.10 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(0.0015, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0595, device='cuda:0', grad_fn=) [2024-06-19 00:40:55,746] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.88 [2024-06-19 00:40:55,747] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2835.46 | bwd_microstep: 1620.13 | bwd_inner_microstep: 1614.54 | bwd_allreduce_microstep: 5.41 | step_microstep: 60.43 [2024-06-19 00:40:55,747] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 5535.02 | bwd: 3348.12 | bwd_inner: 3337.69 | bwd_allreduce: 10.19 | step: 60.51 74%|███████▎ | 147/200 [26:08<08:30, 9.63s/it] {'loss': 0.5125, 'learning_rate': 1.7312628035537387e-05, 'epoch': 1.47} 74%|███████▎ | 147/200 [26:08<08:30, 9.63s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1495, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.0927, device='cuda:0', grad_fn=) [2024-06-19 00:41:01,138] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3493.26 | bwd_microstep: 1805.64 | bwd_inner_microstep: 1800.50 | bwd_allreduce_microstep: 4.99 | step_microstep: 0.13 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0040, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.0614, device='cuda:0', grad_fn=) [2024-06-19 00:41:06,405] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:41:06,406] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3441.56 | bwd_microstep: 1683.33 | bwd_inner_microstep: 1677.89 | bwd_allreduce_microstep: 5.32 | step_microstep: 60.47 [2024-06-19 00:41:06,406] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6934.80 | bwd: 3488.96 | bwd_inner: 3478.46 | bwd_allreduce: 10.32 | step: 60.61 74%|███████▍ | 148/200 [26:19<08:36, 9.94s/it] {'loss': 0.577, 'learning_rate': 1.6704238749415957e-05, 'epoch': 1.48} 74%|███████▍ | 148/200 [26:19<08:36, 9.94s/it]warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1152, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0615, device='cuda:0', grad_fn=) [2024-06-19 00:41:11,039] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2744.64 | bwd_microstep: 1794.34 | bwd_inner_microstep: 1789.39 | bwd_allreduce_microstep: 4.78 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1958, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.1344, device='cuda:0', grad_fn=) [2024-06-19 00:41:15,744] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:41:15,744] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2748.14 | bwd_microstep: 1801.68 | bwd_inner_microstep: 1796.28 | bwd_allreduce_microstep: 5.30 | step_microstep: 60.28 [2024-06-19 00:41:15,745] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 5492.76 | bwd: 3596.01 | bwd_inner: 3585.71 | bwd_allreduce: 10.06 | step: 60.37 74%|███████▍ | 149/200 [26:28<08:17, 9.76s/it] {'loss': 1.098, 'learning_rate': 1.6104580699624837e-05, 'epoch': 1.49} 74%|███████▍ | 149/200 [26:28<08:17, 9.76s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9571, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.9195, device='cuda:0', grad_fn=) [2024-06-19 00:41:21,139] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3495.52 | bwd_microstep: 1806.58 | bwd_inner_microstep: 1801.53 | bwd_allreduce_microstep: 4.95 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1024) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1024, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9262, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.8914, device='cuda:0', grad_fn=) [2024-06-19 00:41:26,108] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:41:26,109] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3040.56 | bwd_microstep: 1774.32 | bwd_inner_microstep: 1768.82 | bwd_allreduce_microstep: 5.32 | step_microstep: 60.12 [2024-06-19 00:41:26,109] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6536.06 | bwd: 3580.90 | bwd_inner: 3570.40 | bwd_allreduce: 10.27 | step: 60.19 75%|███████▌ | 150/200 [26:38<08:17, 9.94s/it] {'loss': 0.9055, 'learning_rate': 1.5513811136094787e-05, 'epoch': 1.5} 75%|███████▌ | 150/200 [26:38<08:17, 9.94s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.6546, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.6470, device='cuda:0', grad_fn=) [2024-06-19 00:41:31,623] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3544.24 | bwd_microstep: 1875.15 | bwd_inner_microstep: 1870.15 | bwd_allreduce_microstep: 4.82 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1427, device='cuda:0', grad_fn=) tensor(0.7006, device='cuda:0', grad_fn=) tensor(1.0985, device='cuda:0', grad_fn=) [2024-06-19 00:41:37,195] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.86 [2024-06-19 00:41:37,196] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3542.16 | bwd_microstep: 1879.99 | bwd_inner_microstep: 1874.36 | bwd_allreduce_microstep: 5.52 | step_microstep: 60.45 [2024-06-19 00:41:37,196] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7086.39 | bwd: 3755.13 | bwd_inner: 3744.56 | bwd_allreduce: 10.32 | step: 60.53 76%|███████▌ | 151/200 [26:49<08:23, 10.28s/it] {'loss': 0.8727, 'learning_rate': 1.4932084977905042e-05, 'epoch': 1.51} 76%|███████▌ | 151/200 [26:49<08:23, 10.28s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9869, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.9463, device='cuda:0', grad_fn=) [2024-06-19 00:41:42,741] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3548.80 | bwd_microstep: 1896.60 | bwd_inner_microstep: 1891.62 | bwd_allreduce_microstep: 4.82 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9003, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.8687, device='cuda:0', grad_fn=) [2024-06-19 00:41:48,317] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.85 [2024-06-19 00:41:48,318] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3545.80 | bwd_microstep: 1879.23 | bwd_inner_microstep: 1873.74 | bwd_allreduce_microstep: 5.32 | step_microstep: 60.29 [2024-06-19 00:41:48,318] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7094.58 | bwd: 3775.82 | bwd_inner: 3765.47 | bwd_allreduce: 10.15 | step: 60.37 76%|███████▌ | 152/200 [27:00<08:25, 10.54s/it] {'loss': 0.9075, 'learning_rate': 1.4359554772658552e-05, 'epoch': 1.52} 76%|███████▌ | 152/200 [27:00<08:25, 10.54s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.4986, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.4066, device='cuda:0', grad_fn=) [2024-06-19 00:41:53,845] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3546.85 | bwd_microstep: 1883.77 | bwd_inner_microstep: 1878.72 | bwd_allreduce_microstep: 4.93 | step_microstep: 0.13 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9593, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.9215, device='cuda:0', grad_fn=) [2024-06-19 00:41:59,322] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:41:59,323] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3504.25 | bwd_microstep: 1819.96 | bwd_inner_microstep: 1814.34 | bwd_allreduce_microstep: 5.51 | step_microstep: 61.22 [2024-06-19 00:41:59,323] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7051.09 | bwd: 3703.72 | bwd_inner: 3693.07 | bwd_allreduce: 10.45 | step: 61.36 76%|███████▋ | 153/200 [27:11<08:21, 10.68s/it] {'loss': 1.1641, 'learning_rate': 1.3796370656478935e-05, 'epoch': 1.53} 76%|███████▋ | 153/200 [27:11<08:21, 10.68s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0040, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0618, device='cuda:0', grad_fn=) [2024-06-19 00:42:04,526] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3441.01 | bwd_microstep: 1678.07 | bwd_inner_microstep: 1673.15 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0910, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0397, device='cuda:0', grad_fn=) [2024-06-19 00:42:10,118] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:42:10,119] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3546.63 | bwd_microstep: 1890.81 | bwd_inner_microstep: 1885.39 | bwd_allreduce_microstep: 5.33 | step_microstep: 60.22 [2024-06-19 00:42:10,119] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6987.63 | bwd: 3568.87 | bwd_inner: 3558.56 | bwd_allreduce: 10.14 | step: 60.29 77%|███████▋ | 154/200 [27:22<08:12, 10.71s/it] {'loss': 0.5507, 'learning_rate': 1.3242680314639993e-05, 'epoch': 1.54} 77%|███████▋ | 154/200 [27:22<08:12, 10.71s/it]warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0385, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0928, device='cuda:0', grad_fn=) [2024-06-19 00:42:14,442] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2648.23 | bwd_microstep: 1591.91 | bwd_inner_microstep: 1586.92 | bwd_allreduce_microstep: 4.82 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1218, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0674, device='cuda:0', grad_fn=) [2024-06-19 00:42:19,909] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:42:19,910] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3498.53 | bwd_microstep: 1817.13 | bwd_inner_microstep: 1811.59 | bwd_allreduce_microstep: 5.37 | step_microstep: 60.22 [2024-06-19 00:42:19,910] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6146.74 | bwd: 3409.03 | bwd_inner: 3398.60 | bwd_allreduce: 10.15 | step: 60.30 78%|███████▊ | 155/200 [27:32<07:49, 10.44s/it] {'loss': 0.5801, 'learning_rate': 1.2698628942837699e-05, 'epoch': 1.55} 78%|███████▊ | 155/200 [27:32<07:49, 10.44s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9976, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.9560, device='cuda:0', grad_fn=) [2024-06-19 00:42:25,313] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3504.72 | bwd_microstep: 1805.97 | bwd_inner_microstep: 1800.89 | bwd_allreduce_microstep: 4.95 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.5242, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.4299, device='cuda:0', grad_fn=) [2024-06-19 00:42:30,930] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:42:30,930] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3555.02 | bwd_microstep: 1902.25 | bwd_inner_microstep: 1896.68 | bwd_allreduce_microstep: 5.40 | step_microstep: 60.74 [2024-06-19 00:42:30,931] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7059.75 | bwd: 3708.21 | bwd_inner: 3697.64 | bwd_allreduce: 10.32 | step: 60.83 78%|███████▊ | 156/200 [27:43<07:46, 10.61s/it] {'loss': 1.1929, 'learning_rate': 1.2164359209115234e-05, 'epoch': 1.56} 78%|███████▊ | 156/200 [27:43<07:46, 10.61s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.4109, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.3279, device='cuda:0', grad_fn=) [2024-06-19 00:42:36,452] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3544.70 | bwd_microstep: 1881.76 | bwd_inner_microstep: 1876.70 | bwd_allreduce_microstep: 4.89 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1560, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.0986, device='cuda:0', grad_fn=) [2024-06-19 00:42:41,918] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.89 [2024-06-19 00:42:41,918] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3501.31 | bwd_microstep: 1813.22 | bwd_inner_microstep: 1807.59 | bwd_allreduce_microstep: 5.53 | step_microstep: 61.17 [2024-06-19 00:42:41,919] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7045.98 | bwd: 3694.98 | bwd_inner: 3684.34 | bwd_allreduce: 10.40 | step: 61.25 78%|███████▊ | 157/200 [27:54<07:41, 10.72s/it] {'loss': 1.2133, 'learning_rate': 1.1640011216450691e-05, 'epoch': 1.57} 78%|███████▊ | 157/200 [27:54<07:41, 10.72s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.3381, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.2625, device='cuda:0', grad_fn=) [2024-06-19 00:42:47,475] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3551.41 | bwd_microstep: 1905.74 | bwd_inner_microstep: 1900.79 | bwd_allreduce_microstep: 4.78 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0165, device='cuda:0', grad_fn=) tensor(0.7006, device='cuda:0', grad_fn=) tensor(0.0849, device='cuda:0', grad_fn=) [2024-06-19 00:42:52,738] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:42:52,739] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3442.56 | bwd_microstep: 1679.51 | bwd_inner_microstep: 1673.96 | bwd_allreduce_microstep: 5.38 | step_microstep: 60.38 [2024-06-19 00:42:52,739] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6993.95 | bwd: 3585.25 | bwd_inner: 3574.84 | bwd_allreduce: 10.13 | step: 60.46 79%|███████▉ | 158/200 [28:05<07:31, 10.75s/it] {'loss': 0.6737, 'learning_rate': 1.1125722466017547e-05, 'epoch': 1.58} 79%|███████▉ | 158/200 [28:05<07:31, 10.75s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1184, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.0647, device='cuda:0', grad_fn=) [2024-06-19 00:42:58,252] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3541.40 | bwd_microstep: 1877.90 | bwd_inner_microstep: 1872.92 | bwd_allreduce_microstep: 4.81 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1280) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1280, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1020, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.0503, device='cuda:0', grad_fn=) [2024-06-19 00:43:03,191] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:43:03,192] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3137.58 | bwd_microstep: 1656.55 | bwd_inner_microstep: 1651.13 | bwd_allreduce_microstep: 5.31 | step_microstep: 59.95 [2024-06-19 00:43:03,192] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6678.96 | bwd: 3534.45 | bwd_inner: 3524.10 | bwd_allreduce: 10.13 | step: 60.03 80%|███████▉ | 159/200 [28:15<07:17, 10.66s/it] {'loss': 1.0575, 'learning_rate': 1.0621627821127289e-05, 'epoch': 1.59} 80%|███████▉ | 159/200 [28:15<07:17, 10.66s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0330, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0878, device='cuda:0', grad_fn=) [2024-06-19 00:43:08,359] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3433.91 | bwd_microstep: 1651.65 | bwd_inner_microstep: 1646.68 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0302, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.9856, device='cuda:0', grad_fn=) [2024-06-19 00:43:13,963] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.87 [2024-06-19 00:43:13,964] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3554.55 | bwd_microstep: 1892.61 | bwd_inner_microstep: 1886.98 | bwd_allreduce_microstep: 5.52 | step_microstep: 61.49 [2024-06-19 00:43:13,964] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6988.45 | bwd: 3544.26 | bwd_inner: 3533.71 | bwd_allreduce: 10.30 | step: 61.57 80%|████████ | 160/200 [28:26<07:07, 10.70s/it] {'loss': 0.5367, 'learning_rate': 1.012785947186397e-05, 'epoch': 1.6} 80%|████████ | 160/200 [28:26<07:07, 10.70s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.8932, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.8623, device='cuda:0', grad_fn=) [2024-06-19 00:43:19,368] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3499.05 | bwd_microstep: 1812.68 | bwd_inner_microstep: 1807.71 | bwd_allreduce_microstep: 4.86 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9238, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.8896, device='cuda:0', grad_fn=) [2024-06-19 00:43:24,975] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.83 [2024-06-19 00:43:24,975] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3548.91 | bwd_microstep: 1900.80 | bwd_inner_microstep: 1895.29 | bwd_allreduce_microstep: 5.36 | step_microstep: 60.53 [2024-06-19 00:43:24,976] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7047.93 | bwd: 3713.47 | bwd_inner: 3703.05 | bwd_allreduce: 10.23 | step: 60.61 80%|████████ | 161/200 [28:37<07:00, 10.79s/it] {'loss': 0.876, 'learning_rate': 9.644546900419533e-06, 'epoch': 1.61} 80%|████████ | 161/200 [28:37<07:00, 10.79s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(0.7359, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.7204, device='cuda:0', grad_fn=) [2024-06-19 00:43:29,619] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2748.98 | bwd_microstep: 1800.71 | bwd_inner_microstep: 1795.75 | bwd_allreduce_microstep: 4.79 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1139, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.0606, device='cuda:0', grad_fn=) [2024-06-19 00:43:35,220] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:43:35,221] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3548.56 | bwd_microstep: 1897.10 | bwd_inner_microstep: 1891.56 | bwd_allreduce_microstep: 5.43 | step_microstep: 60.59 [2024-06-19 00:43:35,221] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6297.53 | bwd: 3697.80 | bwd_inner: 3687.35 | bwd_allreduce: 10.23 | step: 60.66 81%|████████ | 162/200 [28:47<06:43, 10.63s/it] {'loss': 0.8905, 'learning_rate': 9.171816847139448e-06, 'epoch': 1.62} 81%|████████ | 162/200 [28:47<06:43, 10.63s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0008, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.0586, device='cuda:0', grad_fn=) [2024-06-19 00:43:40,431] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3443.14 | bwd_microstep: 1680.96 | bwd_inner_microstep: 1676.07 | bwd_allreduce_microstep: 4.79 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2140, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.1510, device='cuda:0', grad_fn=) [2024-06-19 00:43:46,038] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:43:46,038] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3553.20 | bwd_microstep: 1896.46 | bwd_inner_microstep: 1891.07 | bwd_allreduce_microstep: 5.32 | step_microstep: 60.04 [2024-06-19 00:43:46,039] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6996.32 | bwd: 3577.41 | bwd_inner: 3567.13 | bwd_allreduce: 10.13 | step: 60.12 82%|████████▏ | 163/200 [28:58<06:35, 10.68s/it] {'loss': 0.6048, 'learning_rate': 8.70979327728718e-06, 'epoch': 1.63} 82%|████████▏ | 163/200 [28:58<06:35, 10.68s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.8346, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.8093, device='cuda:0', grad_fn=) [2024-06-19 00:43:51,434] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3494.87 | bwd_microstep: 1807.59 | bwd_inner_microstep: 1802.69 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0083, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(0.0774, device='cuda:0', grad_fn=) [2024-06-19 00:43:56,575] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:43:56,576] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3402.81 | bwd_microstep: 1599.73 | bwd_inner_microstep: 1594.29 | bwd_allreduce_microstep: 5.34 | step_microstep: 60.41 [2024-06-19 00:43:56,576] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6897.66 | bwd: 3407.32 | bwd_inner: 3396.99 | bwd_allreduce: 10.15 | step: 60.49 82%|████████▏ | 164/200 [29:09<06:23, 10.64s/it] {'loss': 0.4433, 'learning_rate': 8.25859734853645e-06, 'epoch': 1.64} 82%|████████▏ | 164/200 [29:09<06:23, 10.64s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2878, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(1.2290, device='cuda:0', grad_fn=) [2024-06-19 00:44:02,145] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3556.53 | bwd_microstep: 1913.81 | bwd_inner_microstep: 1908.85 | bwd_allreduce_microstep: 4.81 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0861, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0353, device='cuda:0', grad_fn=) [2024-06-19 00:44:06,745] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:44:06,746] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2714.74 | bwd_microstep: 1731.23 | bwd_inner_microstep: 1725.56 | bwd_allreduce_microstep: 5.49 | step_microstep: 60.60 [2024-06-19 00:44:06,746] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6271.25 | bwd: 3645.03 | bwd_inner: 3634.49 | bwd_allreduce: 10.30 | step: 60.68 82%|████████▎ | 165/200 [29:19<06:07, 10.50s/it] {'loss': 1.1322, 'learning_rate': 7.81834737919978e-06, 'epoch': 1.65} 82%|████████▎ | 165/200 [29:19<06:07, 10.50s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.8087, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(0.7978, device='cuda:0', grad_fn=) [2024-06-19 00:44:12,142] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3496.19 | bwd_microstep: 1807.73 | bwd_inner_microstep: 1802.71 | bwd_allreduce_microstep: 4.84 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.5584, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.4607, device='cuda:0', grad_fn=) [2024-06-19 00:44:17,593] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.89 [2024-06-19 00:44:17,593] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3497.07 | bwd_microstep: 1805.25 | bwd_inner_microstep: 1799.68 | bwd_allreduce_microstep: 5.46 | step_microstep: 60.45 [2024-06-19 00:44:17,594] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6993.24 | bwd: 3612.97 | bwd_inner: 3602.45 | bwd_allreduce: 10.29 | step: 60.53 83%|████████▎ | 166/200 [29:30<06:00, 10.60s/it] {'loss': 1.1292, 'learning_rate': 7.389158817201542e-06, 'epoch': 1.66} 83%|████████▎ | 166/200 [29:30<06:00, 10.60s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0037, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0614, device='cuda:0', grad_fn=) [2024-06-19 00:44:22,762] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3434.50 | bwd_microstep: 1652.44 | bwd_inner_microstep: 1647.41 | bwd_allreduce_microstep: 4.85 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1933, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.1324, device='cuda:0', grad_fn=) [2024-06-19 00:44:28,214] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.85 [2024-06-19 00:44:28,214] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3498.55 | bwd_microstep: 1804.40 | bwd_inner_microstep: 1798.80 | bwd_allreduce_microstep: 5.42 | step_microstep: 60.22 [2024-06-19 00:44:28,215] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6933.03 | bwd: 3456.83 | bwd_inner: 3446.30 | bwd_allreduce: 10.23 | step: 60.30 84%|████████▎ | 167/200 [29:40<05:50, 10.61s/it] {'loss': 0.5969, 'learning_rate': 6.9711442098037375e-06, 'epoch': 1.67} 84%|████████▎ | 167/200 [29:40<05:50, 10.61s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9627, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.9242, device='cuda:0', grad_fn=) [2024-06-19 00:44:33,788] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3556.15 | bwd_microstep: 1917.63 | bwd_inner_microstep: 1912.66 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.7200, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.7058, device='cuda:0', grad_fn=) [2024-06-19 00:44:39,398] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:44:39,398] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3550.49 | bwd_microstep: 1901.76 | bwd_inner_microstep: 1896.23 | bwd_allreduce_microstep: 5.35 | step_microstep: 60.21 [2024-06-19 00:44:39,399] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7106.63 | bwd: 3819.38 | bwd_inner: 3808.98 | bwd_allreduce: 10.12 | step: 60.29 84%|████████▍ | 168/200 [29:52<05:45, 10.78s/it] {'loss': 0.815, 'learning_rate': 6.564413174092443e-06, 'epoch': 1.68} 84%|████████▍ | 168/200 [29:52<05:45, 10.78s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2952, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(1.2353, device='cuda:0', grad_fn=) [2024-06-19 00:44:44,795] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3497.88 | bwd_microstep: 1806.73 | bwd_inner_microstep: 1801.84 | bwd_allreduce_microstep: 4.78 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0878, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.0368, device='cuda:0', grad_fn=) [2024-06-19 00:44:49,498] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:44:49,499] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2747.45 | bwd_microstep: 1800.61 | bwd_inner_microstep: 1795.11 | bwd_allreduce_microstep: 5.33 | step_microstep: 60.18 [2024-06-19 00:44:49,499] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6245.31 | bwd: 3607.33 | bwd_inner: 3597.00 | bwd_allreduce: 10.09 | step: 60.26 84%|████████▍ | 169/200 [30:02<05:27, 10.58s/it] {'loss': 1.1361, 'learning_rate': 6.16907236823262e-06, 'epoch': 1.69} 84%|████████▍ | 169/200 [30:02<05:27, 10.58s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0037, device='cuda:0', grad_fn=) tensor(0.7039, device='cuda:0', grad_fn=) tensor(0.0737, device='cuda:0', grad_fn=) [2024-06-19 00:44:54,672] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3435.29 | bwd_microstep: 1651.57 | bwd_inner_microstep: 1646.46 | bwd_allreduce_microstep: 4.99 | step_microstep: 0.09 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.7315, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.7162, device='cuda:0', grad_fn=) [2024-06-19 00:45:00,156] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:45:00,157] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3503.36 | bwd_microstep: 1825.83 | bwd_inner_microstep: 1820.33 | bwd_allreduce_microstep: 5.39 | step_microstep: 60.45 [2024-06-19 00:45:00,157] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6938.64 | bwd: 3477.39 | bwd_inner: 3466.80 | bwd_allreduce: 10.39 | step: 60.54 85%|████████▌ | 170/200 [30:12<05:18, 10.60s/it] {'loss': 0.3949, 'learning_rate': 5.785225463498828e-06, 'epoch': 1.7} 85%|████████▌ | 170/200 [30:12<05:18, 10.60s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2709, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.2023, device='cuda:0', grad_fn=) [2024-06-19 00:45:05,677] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3543.04 | bwd_microstep: 1880.48 | bwd_inner_microstep: 1875.52 | bwd_allreduce_microstep: 4.82 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9550, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.9176, device='cuda:0', grad_fn=) [2024-06-19 00:45:11,133] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.85 [2024-06-19 00:45:11,134] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3497.87 | bwd_microstep: 1808.37 | bwd_inner_microstep: 1802.93 | bwd_allreduce_microstep: 5.31 | step_microstep: 60.57 [2024-06-19 00:45:11,134] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7040.89 | bwd: 3688.84 | bwd_inner: 3678.50 | bwd_allreduce: 10.14 | step: 60.65 86%|████████▌ | 171/200 [30:23<05:10, 10.71s/it] {'loss': 1.0599, 'learning_rate': 5.412973117089287e-06, 'epoch': 1.71} 86%|████████▌ | 171/200 [30:23<05:10, 10.71s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0040, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0618, device='cuda:0', grad_fn=) [2024-06-19 00:45:16,338] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3440.99 | bwd_microstep: 1678.40 | bwd_inner_microstep: 1673.41 | bwd_allreduce_microstep: 4.84 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.2602, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.2923, device='cuda:0', grad_fn=) [2024-06-19 00:45:21,601] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:45:21,602] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3440.84 | bwd_microstep: 1681.19 | bwd_inner_microstep: 1675.64 | bwd_allreduce_microstep: 5.38 | step_microstep: 60.52 [2024-06-19 00:45:21,602] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6881.81 | bwd: 3359.58 | bwd_inner: 3349.13 | bwd_allreduce: 10.21 | step: 60.61 86%|████████▌ | 172/200 [30:34<04:57, 10.64s/it] {'loss': 0.177, 'learning_rate': 5.05241294573024e-06, 'epoch': 1.72} 86%|████████▌ | 172/200 [30:34<04:57, 10.64s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1704, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1112, device='cuda:0', grad_fn=) [2024-06-19 00:45:27,002] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3498.25 | bwd_microstep: 1808.19 | bwd_inner_microstep: 1803.24 | bwd_allreduce_microstep: 4.79 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0034, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.9612, device='cuda:0', grad_fn=) [2024-06-19 00:45:32,474] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:45:32,474] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3500.27 | bwd_microstep: 1819.88 | bwd_inner_microstep: 1814.32 | bwd_allreduce_microstep: 5.39 | step_microstep: 60.73 [2024-06-19 00:45:32,475] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6998.50 | bwd: 3628.07 | bwd_inner: 3617.64 | bwd_allreduce: 10.16 | step: 60.81 86%|████████▋ | 173/200 [30:45<04:49, 10.71s/it] {'loss': 1.0362, 'learning_rate': 4.703639500077656e-06, 'epoch': 1.73} 86%|████████▋ | 173/200 [30:45<04:49, 10.71s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0022, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(0.0720, device='cuda:0', grad_fn=) [2024-06-19 00:45:37,557] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3402.43 | bwd_microstep: 1598.22 | bwd_inner_microstep: 1593.28 | bwd_allreduce_microstep: 4.84 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.6325, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.6274, device='cuda:0', grad_fn=) [2024-06-19 00:45:42,999] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:45:43,000] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3492.01 | bwd_microstep: 1803.36 | bwd_inner_microstep: 1797.91 | bwd_allreduce_microstep: 5.31 | step_microstep: 60.22 [2024-06-19 00:45:43,000] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6894.42 | bwd: 3401.57 | bwd_inner: 3391.24 | bwd_allreduce: 10.15 | step: 60.30 87%|████████▋ | 174/200 [30:55<04:37, 10.65s/it] {'loss': 0.3497, 'learning_rate': 4.366744239922998e-06, 'epoch': 1.74} 87%|████████▋ | 174/200 [30:55<04:37, 10.65s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.6656, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.6568, device='cuda:0', grad_fn=) [2024-06-19 00:45:48,401] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3499.18 | bwd_microstep: 1809.53 | bwd_inner_microstep: 1804.46 | bwd_allreduce_microstep: 4.96 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.3221, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(1.2599, device='cuda:0', grad_fn=) [2024-06-19 00:45:53,967] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.85 [2024-06-19 00:45:53,967] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3541.85 | bwd_microstep: 1875.15 | bwd_inner_microstep: 1869.71 | bwd_allreduce_microstep: 5.33 | step_microstep: 60.92 [2024-06-19 00:45:53,968] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7041.02 | bwd: 3684.68 | bwd_inner: 3674.19 | bwd_allreduce: 10.30 | step: 61.00 88%|████████▊ | 175/200 [31:06<04:28, 10.75s/it] {'loss': 0.9584, 'learning_rate': 4.041815510209396e-06, 'epoch': 1.75} 88%|████████▊ | 175/200 [31:06<04:28, 10.75s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.3841, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.3038, device='cuda:0', grad_fn=) [2024-06-19 00:45:59,364] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3498.95 | bwd_microstep: 1803.16 | bwd_inner_microstep: 1798.31 | bwd_allreduce_microstep: 4.78 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(0.5720, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.5726, device='cuda:0', grad_fn=) [2024-06-19 00:46:02,201] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:46:02,202] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 1762.88 | bwd_microstep: 949.01 | bwd_inner_microstep: 943.52 | bwd_allreduce_microstep: 5.38 | step_microstep: 60.44 [2024-06-19 00:46:02,202] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 5261.81 | bwd: 2752.17 | bwd_inner: 2741.82 | bwd_allreduce: 10.18 | step: 60.52 88%|████████▊ | 176/200 [31:14<03:59, 9.99s/it] {'loss': 0.9382, 'learning_rate': 3.728938517864794e-06, 'epoch': 1.76} 88%|████████▊ | 176/200 [31:14<03:59, 9.99s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.7206, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(0.7185, device='cuda:0', grad_fn=) [2024-06-19 00:46:07,589] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3493.17 | bwd_microstep: 1802.10 | bwd_inner_microstep: 1797.18 | bwd_allreduce_microstep: 4.81 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2154, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.1517, device='cuda:0', grad_fn=) [2024-06-19 00:46:13,169] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:46:13,170] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3546.08 | bwd_microstep: 1882.81 | bwd_inner_microstep: 1877.34 | bwd_allreduce_microstep: 5.36 | step_microstep: 60.29 [2024-06-19 00:46:13,170] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7039.22 | bwd: 3684.90 | bwd_inner: 3674.53 | bwd_allreduce: 10.18 | step: 60.37 88%|████████▊ | 177/200 [31:25<03:56, 10.29s/it] {'loss': 0.9351, 'learning_rate': 3.4281953094578877e-06, 'epoch': 1.77} 88%|████████▊ | 177/200 [31:25<03:56, 10.29s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1965, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.1353, device='cuda:0', grad_fn=) [2024-06-19 00:46:18,563] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3495.47 | bwd_microstep: 1805.68 | bwd_inner_microstep: 1800.80 | bwd_allreduce_microstep: 4.77 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) tensor(1.6224, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.5183, device='cuda:0', grad_fn=) [2024-06-19 00:46:22,919] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.78 [2024-06-19 00:46:22,920] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2638.41 | bwd_microstep: 1576.84 | bwd_inner_microstep: 1571.42 | bwd_allreduce_microstep: 5.31 | step_microstep: 60.13 [2024-06-19 00:46:22,920] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6133.86 | bwd: 3382.51 | bwd_inner: 3372.23 | bwd_allreduce: 10.09 | step: 60.21 89%|████████▉ | 178/200 [31:35<03:42, 10.13s/it] {'loss': 1.3268, 'learning_rate': 3.1396647496828247e-06, 'epoch': 1.78} 89%|████████▉ | 178/200 [31:35<03:42, 10.13s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0094, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0666, device='cuda:0', grad_fn=) [2024-06-19 00:46:28,120] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3437.72 | bwd_microstep: 1678.74 | bwd_inner_microstep: 1673.82 | bwd_allreduce_microstep: 4.81 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9212, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.8869, device='cuda:0', grad_fn=) [2024-06-19 00:46:33,600] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:46:33,601] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3503.58 | bwd_microstep: 1823.65 | bwd_inner_microstep: 1818.07 | bwd_allreduce_microstep: 5.40 | step_microstep: 60.32 [2024-06-19 00:46:33,601] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6941.28 | bwd: 3502.38 | bwd_inner: 3491.95 | bwd_allreduce: 10.19 | step: 60.40 90%|████████▉ | 179/200 [31:46<03:36, 10.29s/it] {'loss': 0.4768, 'learning_rate': 2.8634225006782865e-06, 'epoch': 1.79} 90%|████████▉ | 179/200 [31:46<03:36, 10.29s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0143, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0710, device='cuda:0', grad_fn=) [2024-06-19 00:46:38,801] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3439.62 | bwd_microstep: 1677.23 | bwd_inner_microstep: 1672.13 | bwd_allreduce_microstep: 5.00 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0026, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.0608, device='cuda:0', grad_fn=) [2024-06-19 00:46:43,175] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:46:43,176] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2643.42 | bwd_microstep: 1592.04 | bwd_inner_microstep: 1586.61 | bwd_allreduce_microstep: 5.33 | step_microstep: 60.72 [2024-06-19 00:46:43,176] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6083.02 | bwd: 3269.27 | bwd_inner: 3258.74 | bwd_allreduce: 10.33 | step: 60.80 90%|█████████ | 180/200 [31:55<03:21, 10.08s/it] {'loss': 0.0659, 'learning_rate': 2.5995410021864787e-06, 'epoch': 1.8} 90%|█████████ | 180/200 [31:55<03:21, 10.08s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0030, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.0611, device='cuda:0', grad_fn=) [2024-06-19 00:46:48,383] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3442.09 | bwd_microstep: 1680.53 | bwd_inner_microstep: 1675.49 | bwd_allreduce_microstep: 4.86 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0313, device='cuda:0', grad_fn=) tensor(0.8148, device='cuda:0', grad_fn=) tensor(0.1097, device='cuda:0', grad_fn=) [2024-06-19 00:46:53,610] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:46:53,611] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3435.51 | bwd_microstep: 1653.29 | bwd_inner_microstep: 1647.91 | bwd_allreduce_microstep: 5.31 | step_microstep: 60.08 [2024-06-19 00:46:53,611] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6877.58 | bwd: 3333.81 | bwd_inner: 3323.43 | bwd_allreduce: 10.16 | step: 60.16 90%|█████████ | 181/200 [32:06<03:13, 10.18s/it] {'loss': 0.0854, 'learning_rate': 2.3480894525569562e-06, 'epoch': 1.81} 90%|█████████ | 181/200 [32:06<03:13, 10.18s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9338, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.8986, device='cuda:0', grad_fn=) [2024-06-19 00:46:59,003] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3495.96 | bwd_microstep: 1804.70 | bwd_inner_microstep: 1799.78 | bwd_allreduce_microstep: 4.83 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2346, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.1693, device='cuda:0', grad_fn=) [2024-06-19 00:47:04,487] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.83 [2024-06-19 00:47:04,487] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3504.88 | bwd_microstep: 1825.33 | bwd_inner_microstep: 1819.77 | bwd_allreduce_microstep: 5.39 | step_microstep: 60.48 [2024-06-19 00:47:04,488] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7000.82 | bwd: 3630.03 | bwd_inner: 3619.58 | bwd_allreduce: 10.21 | step: 60.56 91%|█████████ | 182/200 [32:17<03:07, 10.39s/it] {'loss': 1.0339, 'learning_rate': 2.1091337906006482e-06, 'epoch': 1.82} 91%|█████████ | 182/200 [32:17<03:07, 10.39s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(1.4620, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(1.3858, device='cuda:0', grad_fn=) [2024-06-19 00:47:09,218] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2890.66 | bwd_microstep: 1749.44 | bwd_inner_microstep: 1744.61 | bwd_allreduce_microstep: 4.73 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0162, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0727, device='cuda:0', grad_fn=) [2024-06-19 00:47:14,485] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.82 [2024-06-19 00:47:14,486] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3442.86 | bwd_microstep: 1683.56 | bwd_inner_microstep: 1677.53 | bwd_allreduce_microstep: 5.92 | step_microstep: 60.93 [2024-06-19 00:47:14,486] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6333.51 | bwd: 3433.00 | bwd_inner: 3422.15 | bwd_allreduce: 10.66 | step: 61.01 92%|█████████▏| 183/200 [32:27<02:54, 10.27s/it] {'loss': 0.7292, 'learning_rate': 1.8827366782984913e-06, 'epoch': 1.83} 92%|█████████▏| 183/200 [32:27<02:54, 10.27s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.8071, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.7845, device='cuda:0', grad_fn=) [2024-06-19 00:47:19,878] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3494.60 | bwd_microstep: 1805.15 | bwd_inner_microstep: 1800.15 | bwd_allreduce_microstep: 4.83 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(1.4683, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.3796, device='cuda:0', grad_fn=) [2024-06-19 00:47:24,494] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:47:24,495] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2839.06 | bwd_microstep: 1632.35 | bwd_inner_microstep: 1626.83 | bwd_allreduce_microstep: 5.34 | step_microstep: 60.09 [2024-06-19 00:47:24,495] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6333.64 | bwd: 3437.49 | bwd_inner: 3427.06 | bwd_allreduce: 10.13 | step: 60.17 92%|█████████▏| 184/200 [32:37<02:43, 10.19s/it] {'loss': 1.082, 'learning_rate': 1.6689574843694433e-06, 'epoch': 1.84} 92%|█████████▏| 184/200 [32:37<02:43, 10.19s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.3659, device='cuda:0', grad_fn=) tensor(0.7006, device='cuda:0', grad_fn=) tensor(1.2994, device='cuda:0', grad_fn=) [2024-06-19 00:47:30,118] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3573.03 | bwd_microstep: 1949.97 | bwd_inner_microstep: 1944.81 | bwd_allreduce_microstep: 4.98 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2796, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.2098, device='cuda:0', grad_fn=) [2024-06-19 00:47:34,477] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:47:34,478] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2641.98 | bwd_microstep: 1573.88 | bwd_inner_microstep: 1568.32 | bwd_allreduce_microstep: 5.38 | step_microstep: 60.60 [2024-06-19 00:47:34,478] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6214.99 | bwd: 3523.84 | bwd_inner: 3513.22 | bwd_allreduce: 10.33 | step: 60.68 92%|█████████▎| 185/200 [32:47<02:31, 10.13s/it] {'loss': 1.2546, 'learning_rate': 1.4678522687020413e-06, 'epoch': 1.85} 92%|█████████▎| 185/200 [32:47<02:31, 10.13s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.2788, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.2091, device='cuda:0', grad_fn=) [2024-06-19 00:47:39,893] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3501.33 | bwd_microstep: 1818.07 | bwd_inner_microstep: 1813.04 | bwd_allreduce_microstep: 4.85 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0288, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.9844, device='cuda:0', grad_fn=) [2024-06-19 00:47:45,483] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.88 [2024-06-19 00:47:45,484] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3546.17 | bwd_microstep: 1889.73 | bwd_inner_microstep: 1884.05 | bwd_allreduce_microstep: 5.51 | step_microstep: 60.76 [2024-06-19 00:47:45,484] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7047.49 | bwd: 3707.79 | bwd_inner: 3697.17 | bwd_allreduce: 10.32 | step: 60.84 93%|█████████▎| 186/200 [32:58<02:25, 10.39s/it] {'loss': 1.0967, 'learning_rate': 1.2794737676536994e-06, 'epoch': 1.86} 93%|█████████▎| 186/200 [32:58<02:25, 10.39s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0036, device='cuda:0', grad_fn=) tensor(0.7006, device='cuda:0', grad_fn=) tensor(0.0733, device='cuda:0', grad_fn=) [2024-06-19 00:47:50,689] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3439.67 | bwd_microstep: 1679.96 | bwd_inner_microstep: 1675.04 | bwd_allreduce_microstep: 4.82 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1328, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.0776, device='cuda:0', grad_fn=) [2024-06-19 00:47:56,302] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:47:56,303] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3554.10 | bwd_microstep: 1902.60 | bwd_inner_microstep: 1897.16 | bwd_allreduce_microstep: 5.34 | step_microstep: 60.06 [2024-06-19 00:47:56,303] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6993.75 | bwd: 3582.56 | bwd_inner: 3572.20 | bwd_allreduce: 10.17 | step: 60.14 94%|█████████▎| 187/200 [33:08<02:16, 10.52s/it] {'loss': 0.5755, 'learning_rate': 1.1038713802214717e-06, 'epoch': 1.87} 94%|█████████▎| 187/200 [33:08<02:16, 10.52s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0118, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.9684, device='cuda:0', grad_fn=) [2024-06-19 00:48:01,700] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3498.35 | bwd_microstep: 1808.17 | bwd_inner_microstep: 1803.28 | bwd_allreduce_microstep: 4.78 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.3855, device='cuda:0', grad_fn=) tensor(0.6998, device='cuda:0', grad_fn=) tensor(0.4170, device='cuda:0', grad_fn=) [2024-06-19 00:48:07,058] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:48:07,059] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3477.17 | bwd_microstep: 1736.59 | bwd_inner_microstep: 1731.17 | bwd_allreduce_microstep: 5.32 | step_microstep: 60.47 [2024-06-19 00:48:07,059] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6975.51 | bwd: 3544.74 | bwd_inner: 3534.46 | bwd_allreduce: 10.10 | step: 60.55 94%|█████████▍| 188/200 [33:19<02:07, 10.59s/it] {'loss': 0.6927, 'learning_rate': 9.410911550880475e-07, 'epoch': 1.88} 94%|█████████▍| 188/200 [33:19<02:07, 10.59s/it]warning: The size of tensor a (0) must match the size of tensor b (256) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([256, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1009, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.0489, device='cuda:0', grad_fn=) [2024-06-19 00:48:11,583] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2702.55 | bwd_microstep: 1726.16 | bwd_inner_microstep: 1721.16 | bwd_allreduce_microstep: 4.82 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1200, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.0661, device='cuda:0', grad_fn=) [2024-06-19 00:48:17,181] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.83 [2024-06-19 00:48:17,181] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3552.58 | bwd_microstep: 1889.92 | bwd_inner_microstep: 1884.28 | bwd_allreduce_microstep: 5.50 | step_microstep: 60.37 [2024-06-19 00:48:17,182] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6255.12 | bwd: 3616.07 | bwd_inner: 3605.52 | bwd_allreduce: 10.30 | step: 60.44 94%|█████████▍| 189/200 [33:29<01:54, 10.45s/it] {'loss': 1.0575, 'learning_rate': 7.911757785462881e-07, 'epoch': 1.89} 94%|█████████▍| 189/200 [33:29<01:54, 10.45s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0004, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.9582, device='cuda:0', grad_fn=) [2024-06-19 00:48:22,718] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3545.86 | bwd_microstep: 1891.39 | bwd_inner_microstep: 1886.39 | bwd_allreduce_microstep: 4.82 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.0156, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.9722, device='cuda:0', grad_fn=) [2024-06-19 00:48:28,288] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.85 [2024-06-19 00:48:28,289] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3543.58 | bwd_microstep: 1876.95 | bwd_inner_microstep: 1871.39 | bwd_allreduce_microstep: 5.39 | step_microstep: 60.53 [2024-06-19 00:48:28,289] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7089.43 | bwd: 3768.34 | bwd_inner: 3757.87 | bwd_allreduce: 10.17 | step: 60.61 95%|█████████▌| 190/200 [33:40<01:46, 10.65s/it] {'loss': 0.9652, 'learning_rate': 6.54164563305465e-07, 'epoch': 1.9} 95%|█████████▌| 190/200 [33:40<01:46, 10.65s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9643, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.9260, device='cuda:0', grad_fn=) [2024-06-19 00:48:33,677] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3494.44 | bwd_microstep: 1801.98 | bwd_inner_microstep: 1796.98 | bwd_allreduce_microstep: 4.82 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.5209, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.5267, device='cuda:0', grad_fn=) [2024-06-19 00:48:39,260] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.86 [2024-06-19 00:48:39,261] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3547.05 | bwd_microstep: 1883.82 | bwd_inner_microstep: 1878.31 | bwd_allreduce_microstep: 5.40 | step_microstep: 60.55 [2024-06-19 00:48:39,261] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7041.47 | bwd: 3685.79 | bwd_inner: 3675.33 | bwd_allreduce: 10.20 | step: 60.64 96%|█████████▌| 191/200 [33:51<01:36, 10.75s/it] {'loss': 0.7263, 'learning_rate': 5.300934381821998e-07, 'epoch': 1.91} 96%|█████████▌| 191/200 [33:51<01:36, 10.75s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.7903, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.7694, device='cuda:0', grad_fn=) [2024-06-19 00:48:44,666] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3499.35 | bwd_microstep: 1812.97 | bwd_inner_microstep: 1808.08 | bwd_allreduce_microstep: 4.79 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0019, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.0602, device='cuda:0', grad_fn=) [2024-06-19 00:48:49,927] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.79 [2024-06-19 00:48:49,928] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3442.71 | bwd_microstep: 1679.06 | bwd_inner_microstep: 1673.64 | bwd_allreduce_microstep: 5.30 | step_microstep: 60.15 [2024-06-19 00:48:49,928] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6942.04 | bwd: 3492.02 | bwd_inner: 3481.73 | bwd_allreduce: 10.10 | step: 60.22 96%|█████████▌| 192/200 [34:02<01:25, 10.72s/it] {'loss': 0.4148, 'learning_rate': 4.189949386787462e-07, 'epoch': 1.92} 96%|█████████▌| 192/200 [34:02<01:25, 10.72s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.3267, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(1.2637, device='cuda:0', grad_fn=) [2024-06-19 00:48:55,317] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3495.12 | bwd_microstep: 1802.80 | bwd_inner_microstep: 1797.90 | bwd_allreduce_microstep: 4.80 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.6355, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.6304, device='cuda:0', grad_fn=) [2024-06-19 00:49:00,769] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.84 [2024-06-19 00:49:00,770] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3497.37 | bwd_microstep: 1807.48 | bwd_inner_microstep: 1802.05 | bwd_allreduce_microstep: 5.32 | step_microstep: 60.22 [2024-06-19 00:49:00,770] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6992.46 | bwd: 3610.27 | bwd_inner: 3599.96 | bwd_allreduce: 10.13 | step: 60.30 96%|█████████▋| 193/200 [34:13<01:15, 10.76s/it] {'loss': 0.9471, 'learning_rate': 3.208981984511195e-07, 'epoch': 1.93} 96%|█████████▋| 193/200 [34:13<01:15, 10.76s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9920, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(0.9506, device='cuda:0', grad_fn=) [2024-06-19 00:49:06,179] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3501.87 | bwd_microstep: 1812.26 | bwd_inner_microstep: 1807.21 | bwd_allreduce_microstep: 4.94 | step_microstep: 0.09 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1195, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(1.0661, device='cuda:0', grad_fn=) [2024-06-19 00:49:11,639] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:49:11,640] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3499.14 | bwd_microstep: 1812.29 | bwd_inner_microstep: 1806.87 | bwd_allreduce_microstep: 5.31 | step_microstep: 60.68 [2024-06-19 00:49:11,640] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7001.00 | bwd: 3624.54 | bwd_inner: 3614.08 | bwd_allreduce: 10.26 | step: 60.77 97%|█████████▋| 194/200 [34:24<01:04, 10.79s/it] {'loss': 1.0083, 'learning_rate': 2.3582894166930268e-07, 'epoch': 1.94} 97%|█████████▋| 194/200 [34:24<01:04, 10.79s/it]warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.1117, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.1590, device='cuda:0', grad_fn=) [2024-06-19 00:49:16,135] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2824.29 | bwd_microstep: 1589.20 | bwd_inner_microstep: 1584.29 | bwd_allreduce_microstep: 4.81 | step_microstep: 0.08 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0015, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0595, device='cuda:0', grad_fn=) [2024-06-19 00:49:21,401] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:49:21,401] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3440.26 | bwd_microstep: 1684.74 | bwd_inner_microstep: 1679.38 | bwd_allreduce_microstep: 5.26 | step_microstep: 60.23 [2024-06-19 00:49:21,402] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6264.53 | bwd: 3273.93 | bwd_inner: 3263.68 | bwd_allreduce: 10.07 | step: 60.31 98%|█████████▊| 195/200 [34:34<00:52, 10.48s/it] {'loss': 0.1093, 'learning_rate': 1.6380947627153143e-07, 'epoch': 1.95} 98%|█████████▊| 195/200 [34:34<00:52, 10.48s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.9206, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.8867, device='cuda:0', grad_fn=) [2024-06-19 00:49:27,102] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3710.82 | bwd_microstep: 1890.78 | bwd_inner_microstep: 1885.85 | bwd_allreduce_microstep: 4.82 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.8483, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.8216, device='cuda:0', grad_fn=) [2024-06-19 00:49:32,675] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.81 [2024-06-19 00:49:32,676] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3543.12 | bwd_microstep: 1878.79 | bwd_inner_microstep: 1873.39 | bwd_allreduce_microstep: 5.29 | step_microstep: 60.67 [2024-06-19 00:49:32,676] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7253.91 | bwd: 3769.56 | bwd_inner: 3759.25 | bwd_allreduce: 10.12 | step: 60.75 98%|█████████▊| 196/200 [34:45<00:42, 10.72s/it] {'loss': 0.8541, 'learning_rate': 1.0485868811441757e-07, 'epoch': 1.96} 98%|█████████▊| 196/200 [34:45<00:42, 10.72s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.8063, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.7838, device='cuda:0', grad_fn=) [2024-06-19 00:49:38,067] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3495.26 | bwd_microstep: 1805.64 | bwd_inner_microstep: 1800.65 | bwd_allreduce_microstep: 4.82 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) tensor(1.3581, device='cuda:0', grad_fn=) tensor(0.5782, device='cuda:0', grad_fn=) tensor(1.2801, device='cuda:0', grad_fn=) [2024-06-19 00:49:42,981] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.79 [2024-06-19 00:49:42,982] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2937.50 | bwd_microstep: 1823.47 | bwd_inner_microstep: 1818.02 | bwd_allreduce_microstep: 5.35 | step_microstep: 60.21 [2024-06-19 00:49:42,982] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6432.74 | bwd: 3629.11 | bwd_inner: 3618.72 | bwd_allreduce: 10.15 | step: 60.29 98%|█████████▊| 197/200 [34:55<00:31, 10.60s/it] {'loss': 1.032, 'learning_rate': 5.899203602046655e-08, 'epoch': 1.97} 98%|█████████▊| 197/200 [34:55<00:31, 10.60s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.1671, device='cuda:0', grad_fn=) tensor(0.6965, device='cuda:0', grad_fn=) tensor(1.1200, device='cuda:0', grad_fn=) [2024-06-19 00:49:48,371] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3497.02 | bwd_microstep: 1802.62 | bwd_inner_microstep: 1797.74 | bwd_allreduce_microstep: 4.72 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.3160, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.2425, device='cuda:0', grad_fn=) [2024-06-19 00:49:53,977] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.90 [2024-06-19 00:49:53,978] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3549.36 | bwd_microstep: 1898.92 | bwd_inner_microstep: 1892.95 | bwd_allreduce_microstep: 5.84 | step_microstep: 63.33 [2024-06-19 00:49:53,978] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7046.37 | bwd: 3701.54 | bwd_inner: 3690.76 | bwd_allreduce: 10.57 | step: 63.41 99%|█████████▉| 198/200 [35:06<00:21, 10.72s/it] {'loss': 1.1813, 'learning_rate': 2.6221547724253337e-08, 'epoch': 1.98} 99%|█████████▉| 198/200 [35:06<00:21, 10.72s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0021, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.0601, device='cuda:0', grad_fn=) [2024-06-19 00:49:59,187] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3442.35 | bwd_microstep: 1681.06 | bwd_inner_microstep: 1676.25 | bwd_allreduce_microstep: 4.74 | step_microstep: 0.07 warning: The size of tensor a (0) must match the size of tensor b (768) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([768, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.0019, device='cuda:0', grad_fn=) tensor(0.5847, device='cuda:0', grad_fn=) tensor(0.0601, device='cuda:0', grad_fn=) [2024-06-19 00:50:03,781] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.80 [2024-06-19 00:50:03,781] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 2837.18 | bwd_microstep: 1616.91 | bwd_inner_microstep: 1611.44 | bwd_allreduce_microstep: 5.36 | step_microstep: 60.24 [2024-06-19 00:50:03,782] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 6279.52 | bwd: 3297.96 | bwd_inner: 3287.68 | bwd_allreduce: 10.12 | step: 60.32 100%|█████████▉| 199/200 [35:16<00:10, 10.44s/it] {'loss': 0.0601, 'learning_rate': 6.5558167183898955e-09, 'epoch': 1.99} 100%|█████████▉| 199/200 [35:16<00:10, 10.44s/it]warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(1.4640, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(1.3758, device='cuda:0', grad_fn=) [2024-06-19 00:50:09,164] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3494.66 | bwd_microstep: 1798.46 | bwd_inner_microstep: 1793.53 | bwd_allreduce_microstep: 4.82 | step_microstep: 0.07 please install petrel_client Replace train sampler!! petrel_client is not installed. Using PIL to load images. please install petrel_client Replace train sampler!! petrel_client is not installed. Using PIL to load images. please install petrel_client Replace train sampler!! petrel_client is not installed. Using PIL to load images. please install petrel_client Replace train sampler!! petrel_client is not installed. Using PIL to load images. warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) warning: The size of tensor a (0) must match the size of tensor b (1792) at non-singleton dimension 0, input_embeds[selected].shape=torch.Size([0, 6144]), vit_embeds.shape=torch.Size([1792, 6144]) tensor(0.8882, device='cuda:0', grad_fn=) tensor(0.5814, device='cuda:0', grad_fn=) tensor(0.8575, device='cuda:0', grad_fn=) [2024-06-19 00:50:15,612] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | optimizer_step: 1.87 [2024-06-19 00:50:15,612] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd_microstep: 3548.07 | bwd_microstep: 1897.51 | bwd_inner_microstep: 1891.79 | bwd_allreduce_microstep: 5.57 | step_microstep: 61.67 [2024-06-19 00:50:15,613] [INFO] [logging.py:96:log_dist] [Rank 0] time (ms) | fwd: 7042.68 | bwd: 3695.96 | bwd_inner: 3685.38 | bwd_allreduce: 10.40 | step: 61.75 100%|██████████| 200/200 [35:28<00:00, 10.86s/it] {'loss': 1.1166, 'learning_rate': 0.0, 'epoch': 2.0} 100%|██████████| 200/200 [35:28<00:00, 10.86s/it][INFO|trainer.py:1962] 2024-06-19 00:50:15,625 >> Training completed. Do not forget to share your model on huggingface.co/models =) {'train_runtime': 2128.2481, 'train_samples_per_second': 0.376, 'train_steps_per_second': 0.094, 'train_loss': 0.8767247783765196, 'epoch': 2.0} 100%|██████████| 200/200 [35:28<00:00, 10.86s/it] 100%|██████████| 200/200 [35:28<00:00, 10.64s/it] [INFO|trainer.py:2936] 2024-06-19 00:50:43,672 >> Saving model checkpoint to ckpts/baseline2_2_epochs/ [INFO|configuration_utils.py:473] 2024-06-19 00:50:43,676 >> Configuration saved in ckpts/baseline2_2_epochs/config.json [INFO|configuration_utils.py:594] 2024-06-19 00:50:43,677 >> Configuration saved in ckpts/baseline2_2_epochs/generation_config.json [INFO|modeling_utils.py:2501] 2024-06-19 00:51:21,878 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 11 checkpoint shards. You can find where each parameters has been saved in the index located at ckpts/baseline2_2_epochs/model.safetensors.index.json. [INFO|tokenization_utils_base.py:2433] 2024-06-19 00:51:21,880 >> tokenizer config file saved in ckpts/baseline2_2_epochs/tokenizer_config.json [INFO|tokenization_utils_base.py:2442] 2024-06-19 00:51:21,881 >> Special tokens file saved in ckpts/baseline2_2_epochs/special_tokens_map.json [INFO|tokenization_utils_base.py:2493] 2024-06-19 00:51:21,881 >> added tokens file saved in ckpts/baseline2_2_epochs/added_tokens.json ***** train metrics ***** epoch = 2.0 train_loss = 0.8767 train_runtime = 0:35:28.24 train_samples = 400 train_samples_per_second = 0.376 train_steps_per_second = 0.094