Update my_model/fine_tuner/fine_tuning_data_handler.py
Browse files
my_model/fine_tuner/fine_tuning_data_handler.py
CHANGED
@@ -9,7 +9,7 @@ from my_model.LLAMA2.LLAMA2_model import Llama2ModelManager
|
|
9 |
|
10 |
class FinetuningDataHandler:
|
11 |
"""
|
12 |
-
A class dedicated to handling data for fine-tuning
|
13 |
inspecting, preparing, and splitting the dataset, specifically designed to filter out
|
14 |
data samples exceeding a specified token count limit. This is crucial for models with
|
15 |
token count constraints and it helps control the level of GPU RAM tolerance based on the number of tokens,
|
|
|
9 |
|
10 |
class FinetuningDataHandler:
|
11 |
"""
|
12 |
+
A class dedicated to handling data for fine-tuning LLaMA-2 Chat models. It manages loading,
|
13 |
inspecting, preparing, and splitting the dataset, specifically designed to filter out
|
14 |
data samples exceeding a specified token count limit. This is crucial for models with
|
15 |
token count constraints and it helps control the level of GPU RAM tolerance based on the number of tokens,
|