m7mdal7aj commited on
Commit
b52c17b
1 Parent(s): 0824054

Update my_model/fine_tuner/fine_tuning_data_handler.py

Browse files
my_model/fine_tuner/fine_tuning_data_handler.py CHANGED
@@ -9,7 +9,7 @@ from my_model.LLAMA2.LLAMA2_model import Llama2ModelManager
9
 
10
  class FinetuningDataHandler:
11
  """
12
- A class dedicated to handling data for fine-tuning language models. It manages loading,
13
  inspecting, preparing, and splitting the dataset, specifically designed to filter out
14
  data samples exceeding a specified token count limit. This is crucial for models with
15
  token count constraints and it helps control the level of GPU RAM tolerance based on the number of tokens,
 
9
 
10
  class FinetuningDataHandler:
11
  """
12
+ A class dedicated to handling data for fine-tuning LLaMA-2 Chat models. It manages loading,
13
  inspecting, preparing, and splitting the dataset, specifically designed to filter out
14
  data samples exceeding a specified token count limit. This is crucial for models with
15
  token count constraints and it helps control the level of GPU RAM tolerance based on the number of tokens,