Hugging Face's logo

stabilityai
/
stablelm-2-zephyr-1_6b

Text Generation
Transformers
Safetensors
GGUF
English
stablelm
causal-lm
conversational
Inference Endpoints
Model card Files Community
19

RuntimeError: FlashAttention only support fp16 and bf16 data type

#5
by interstellarninja - opened
Discussion
interstellarninja

I'm getting a runtime error while attempting to finetune with axolotl with flash_attention: true in the config.

here's the full stack trace:

  0%|                                                                                      | 0/2418 [00:00<?, ?it/s][2024-01-23 12:20:53,446] [INFO] [axolotl.utils.samplers.multipack._len_est:178] [PID:9523] [RANK:0] packing_efficiency_estimate: 0.77 total_num_tokens per device: 5142267
Traceback (most recent call last):
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/interstellarninja/ai_projects/axolotl/src/axolotl/cli/train.py", line 43, in <module>
    fire.Fire(do_cli)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/interstellarninja/ai_projects/axolotl/src/axolotl/cli/train.py", line 39, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
  File "/home/interstellarninja/ai_projects/axolotl/src/axolotl/train.py", line 149, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/transformers/trainer.py", line 1534, in train
    return inner_training_loop(
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/transformers/trainer.py", line 1860, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/transformers/trainer.py", line 2737, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/interstellarninja/ai_projects/axolotl/src/axolotl/core/trainer_builder.py", line 329, in compute_loss
    return super().compute_loss(model, inputs, return_outputs=return_outputs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/transformers/trainer.py", line 2760, in compute_loss
    outputs = model(**inputs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/accelerate/utils/operations.py", line 687, in forward
    return model_forward(*args, **kwargs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/accelerate/utils/operations.py", line 675, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
    return func(*args, **kwargs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/peft/peft_model.py", line 1071, in forward
    return self.base_model(
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/peft/tuners/tuners_utils.py", line 108, in forward
    return self.model.forward(*args, **kwargs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/interstellarninja/.cache/huggingface/modules/transformers_modules/stabilityai/stablelm-2-zephyr-1_6b/589adbfdd913d96282d43411c87a996f1bc7b000/modeling_stablelm_epoch.py", line 818, in forward
    outputs = self.model(
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/interstellarninja/ai_projects/axolotl/src/axolotl/monkeypatch/stablelm_attn_hijack_flash.py", line 366, in stablelm_model_forward
    layer_outputs = torch.utils.checkpoint.checkpoint(
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 249, in checkpoint
    return CheckpointFunction.apply(function, preserve, *args)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 107, in forward
    outputs = run_function(*args)
  File "/home/interstellarninja/ai_projects/axolotl/src/axolotl/monkeypatch/stablelm_attn_hijack_flash.py", line 362, in custom_forward
    return module(*inputs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/interstellarninja/ai_projects/axolotl/src/axolotl/monkeypatch/stablelm_attn_hijack_flash.py", line 224, in decoder_layer_forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/interstellarninja/ai_projects/axolotl/src/axolotl/monkeypatch/stablelm_attn_hijack_flash.py", line 159, in flashattn_attn
    output = flash_attn_varlen_qkvpacked_func(
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 887, in flash_attn_varlen_qkvpacked_func
    return FlashAttnVarlenQKVPackedFunc.apply(
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 288, in forward
    out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = _flash_attn_varlen_forward(
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 85, in _flash_attn_varlen_forward
    out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = flash_attn_cuda.varlen_fwd(
RuntimeError: FlashAttention only support fp16 and bf16 data type
Traceback (most recent call last):
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/interstellarninja/ai_projects/axolotl/src/axolotl/cli/train.py", line 43, in <module>
    fire.Fire(do_cli)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/interstellarninja/ai_projects/axolotl/src/axolotl/cli/train.py", line 39, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
  File "/home/interstellarninja/ai_projects/axolotl/src/axolotl/train.py", line 149, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/transformers/trainer.py", line 1534, in train
    return inner_training_loop(
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/transformers/trainer.py", line 1860, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/transformers/trainer.py", line 2737, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/interstellarninja/ai_projects/axolotl/src/axolotl/core/trainer_builder.py", line 329, in compute_loss
    return super().compute_loss(model, inputs, return_outputs=return_outputs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/transformers/trainer.py", line 2760, in compute_loss
    outputs = model(**inputs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/accelerate/utils/operations.py", line 687, in forward
    return model_forward(*args, **kwargs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/accelerate/utils/operations.py", line 675, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
    return func(*args, **kwargs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/peft/peft_model.py", line 1071, in forward
    return self.base_model(
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/peft/tuners/tuners_utils.py", line 108, in forward
    return self.model.forward(*args, **kwargs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/interstellarninja/.cache/huggingface/modules/transformers_modules/stabilityai/stablelm-2-zephyr-1_6b/589adbfdd913d96282d43411c87a996f1bc7b000/modeling_stablelm_epoch.py", line 818, in forward
    outputs = self.model(
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/interstellarninja/ai_projects/axolotl/src/axolotl/monkeypatch/stablelm_attn_hijack_flash.py", line 366, in stablelm_model_forward
    layer_outputs = torch.utils.checkpoint.checkpoint(
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 249, in checkpoint
    return CheckpointFunction.apply(function, preserve, *args)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 107, in forward
    outputs = run_function(*args)
  File "/home/interstellarninja/ai_projects/axolotl/src/axolotl/monkeypatch/stablelm_attn_hijack_flash.py", line 362, in custom_forward
    return module(*inputs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/interstellarninja/ai_projects/axolotl/src/axolotl/monkeypatch/stablelm_attn_hijack_flash.py", line 224, in decoder_layer_forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/interstellarninja/ai_projects/axolotl/src/axolotl/monkeypatch/stablelm_attn_hijack_flash.py", line 159, in flashattn_attn
    output = flash_attn_varlen_qkvpacked_func(
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 887, in flash_attn_varlen_qkvpacked_func
    return FlashAttnVarlenQKVPackedFunc.apply(
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 288, in forward
    out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = _flash_attn_varlen_forward(
  File "/home/interstellarninja/miniconda3/envs/sft-finetune/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 85, in _flash_attn_varlen_forward
    out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = flash_attn_cuda.varlen_fwd(
RuntimeError: FlashAttention only support fp16 and bf16 data type```

Sign up or log in to comment