Upload folder using huggingface_hub

README.md

@@ -1,3 +1,202 @@
----
-license: mit
----
+---
+library_name: peft
+base_model: recogna-nlp/internlm-chatbode-7b
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.10.0

adapter_config.json

@@ -0,0 +1,33 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "recogna-nlp/internlm-chatbode-7b",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 256,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "w2",
+    "wo",
+    "output",
+    "w1",
+    "wqkv",
+    "w3"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aaf41a237c5a0dae7489721c71eb3042c56c51b9452fa148897b67be6f287829
+size 1257556050

xtuner_config.py

@@ -0,0 +1,212 @@
+SYSTEM = ''
+accumulative_counts = 16
+alpaca_en = dict(
+    dataset=dict(
+        data_files=
+        '/petrobr/parceirosbr/home/luis.afonso/.cache/huggingface/hub/datasets--recogna-nlp--adalberto_dataset/snapshots/0b21870f7cec193508fd22d81be533ca240ee8b1/train.json',
+        path='json',
+        type='datasets.load_dataset'),
+    dataset_map_fn='xtuner.dataset.map_fns.adalberto_map_fn',
+    max_length=2048,
+    pack_to_max_length=False,
+    remove_unused_columns=True,
+    shuffle_before_pack=True,
+    template_map_fn=dict(
+        template='xtuner.utils.PROMPT_TEMPLATE.internlm2_chat',
+        type='xtuner.dataset.map_fns.template_map_fn_factory'),
+    tokenizer=dict(
+        padding_side='right',
+        pretrained_model_name_or_path=
+        '/petrobr/parceirosbr/home/luis.afonso/.cache/huggingface/hub/models--recogna-nlp--internlm-chatbode-7b/snapshots/b3acf1fa06aa64d49a6f725864316f35dcc885e2',
+        trust_remote_code=True,
+        type='transformers.AutoTokenizer.from_pretrained'),
+    type='xtuner.dataset.process_hf_dataset',
+    use_varlen_attn=False)
+alpaca_en_path = '/petrobr/parceirosbr/home/luis.afonso/.cache/huggingface/hub/datasets--recogna-nlp--adalberto_dataset/snapshots/0b21870f7cec193508fd22d81be533ca240ee8b1/train.json'
+batch_size = 3
+betas = (
+    0.9,
+    0.999,
+)
+custom_hooks = [
+    dict(
+        tokenizer=dict(
+            padding_side='right',
+            pretrained_model_name_or_path=
+            '/petrobr/parceirosbr/home/luis.afonso/.cache/huggingface/hub/models--recogna-nlp--internlm-chatbode-7b/snapshots/b3acf1fa06aa64d49a6f725864316f35dcc885e2',
+            trust_remote_code=True,
+            type='transformers.AutoTokenizer.from_pretrained'),
+        type='xtuner.engine.hooks.DatasetInfoHook'),
+    dict(
+        evaluation_inputs=[
+            'O que é um bode?',
+            'Qual a capital da França?',
+            'Você pode me explicar o Teorema de Pitágoras com um exemplo, por favor?',
+            'Olá, tudo bem? Estou procurando livros de ficção científica para ler, você teria sugestões para mim?',
+            'Resolva a equação de segundo grau x² - x - 30 = 0',
+            'Escreva um código em python para calcular x^y usando divisão e conquista.',
+        ],
+        every_n_iters=500,
+        prompt_template='xtuner.utils.PROMPT_TEMPLATE.internlm2_chat',
+        system='',
+        tokenizer=dict(
+            padding_side='right',
+            pretrained_model_name_or_path=
+            '/petrobr/parceirosbr/home/luis.afonso/.cache/huggingface/hub/models--recogna-nlp--internlm-chatbode-7b/snapshots/b3acf1fa06aa64d49a6f725864316f35dcc885e2',
+            trust_remote_code=True,
+            type='transformers.AutoTokenizer.from_pretrained'),
+        type='xtuner.engine.hooks.EvaluateChatHook'),
+]
+dataloader_num_workers = 0
+default_hooks = dict(
+    checkpoint=dict(
+        by_epoch=False,
+        interval=500,
+        max_keep_ckpts=2,
+        type='mmengine.hooks.CheckpointHook'),
+    logger=dict(
+        interval=10,
+        log_metric_by_epoch=False,
+        type='mmengine.hooks.LoggerHook'),
+    param_scheduler=dict(type='mmengine.hooks.ParamSchedulerHook'),
+    sampler_seed=dict(type='mmengine.hooks.DistSamplerSeedHook'),
+    timer=dict(type='mmengine.hooks.IterTimerHook'))
+env_cfg = dict(
+    cudnn_benchmark=False,
+    dist_cfg=dict(backend='nccl'),
+    mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0))
+evaluation_freq = 500
+evaluation_inputs = [
+    'O que é um bode?',
+    'Qual a capital da França?',
+    'Você pode me explicar o Teorema de Pitágoras com um exemplo, por favor?',
+    'Olá, tudo bem? Estou procurando livros de ficção científica para ler, você teria sugestões para mim?',
+    'Resolva a equação de segundo grau x² - x - 30 = 0',
+    'Escreva um código em python para calcular x^y usando divisão e conquista.',
+]
+launcher = 'none'
+load_from = None
+log_level = 'INFO'
+log_processor = dict(by_epoch=False)
+lr = 0.0002
+max_epochs = 1
+max_length = 2048
+max_norm = 1
+model = dict(
+    llm=dict(
+        attn_implementation='eager',
+        pretrained_model_name_or_path=
+        '/petrobr/parceirosbr/home/luis.afonso/.cache/huggingface/hub/models--recogna-nlp--internlm-chatbode-7b/snapshots/b3acf1fa06aa64d49a6f725864316f35dcc885e2',
+        quantization_config=dict(
+            bnb_4bit_compute_dtype='torch.float16',
+            bnb_4bit_quant_type='nf4',
+            bnb_4bit_use_double_quant=True,
+            llm_int8_has_fp16_weight=False,
+            llm_int8_threshold=6.0,
+            load_in_4bit=True,
+            load_in_8bit=False,
+            type='transformers.BitsAndBytesConfig'),
+        torch_dtype='torch.float16',
+        trust_remote_code=True,
+        type='transformers.AutoModelForCausalLM.from_pretrained'),
+    lora=dict(
+        bias='none',
+        lora_alpha=128,
+        lora_dropout=0.1,
+        r=256,
+        task_type='CAUSAL_LM',
+        type='peft.LoraConfig'),
+    type='xtuner.model.SupervisedFinetune',
+    use_varlen_attn=False)
+optim_type = 'torch.optim.AdamW'
+optim_wrapper = dict(
+    optimizer=dict(
+        betas=(
+            0.9,
+            0.999,
+        ),
+        lr=0.0002,
+        type='torch.optim.AdamW',
+        weight_decay=0),
+    type='DeepSpeedOptimWrapper')
+pack_to_max_length = False
+param_scheduler = [
+    dict(
+        begin=0,
+        by_epoch=True,
+        convert_to_iter_based=True,
+        end=0.03,
+        start_factor=1e-05,
+        type='mmengine.optim.LinearLR'),
+    dict(
+        begin=0.03,
+        by_epoch=True,
+        convert_to_iter_based=True,
+        end=1,
+        eta_min=0.0,
+        type='mmengine.optim.CosineAnnealingLR'),
+]
+pretrained_model_name_or_path = '/petrobr/parceirosbr/home/luis.afonso/.cache/huggingface/hub/models--recogna-nlp--internlm-chatbode-7b/snapshots/b3acf1fa06aa64d49a6f725864316f35dcc885e2'
+prompt_template = 'xtuner.utils.PROMPT_TEMPLATE.internlm2_chat'
+randomness = dict(deterministic=False, seed=None)
+resume = False
+runner_type = 'FlexibleRunner'
+save_steps = 500
+save_total_limit = 2
+strategy = dict(
+    config=dict(
+        bf16=dict(enabled=False),
+        fp16=dict(enabled=True, initial_scale_power=16),
+        gradient_accumulation_steps='auto',
+        gradient_clipping='auto',
+        train_micro_batch_size_per_gpu='auto',
+        zero_allow_untested_optimizer=True,
+        zero_force_ds_cpu_optimizer=False,
+        zero_optimization=dict(overlap_comm=True, stage=2)),
+    exclude_frozen_parameters=True,
+    gradient_accumulation_steps=16,
+    gradient_clipping=1,
+    sequence_parallel_size=1,
+    train_micro_batch_size_per_gpu=3,
+    type='xtuner.engine.DeepSpeedStrategy')
+tokenizer = dict(
+    padding_side='right',
+    pretrained_model_name_or_path=
+    '/petrobr/parceirosbr/home/luis.afonso/.cache/huggingface/hub/models--recogna-nlp--internlm-chatbode-7b/snapshots/b3acf1fa06aa64d49a6f725864316f35dcc885e2',
+    trust_remote_code=True,
+    type='transformers.AutoTokenizer.from_pretrained')
+train_cfg = dict(max_epochs=1, type='xtuner.engine.runner.TrainLoop')
+train_dataloader = dict(
+    batch_size=3,
+    collate_fn=dict(
+        type='xtuner.dataset.collate_fns.default_collate_fn',
+        use_varlen_attn=False),
+    dataset=dict(
+        dataset=dict(
+            data_files=
+            '/petrobr/parceirosbr/home/luis.afonso/.cache/huggingface/hub/datasets--recogna-nlp--adalberto_dataset/snapshots/0b21870f7cec193508fd22d81be533ca240ee8b1/train.json',
+            path='json',
+            type='datasets.load_dataset'),
+        dataset_map_fn='xtuner.dataset.map_fns.adalberto_map_fn',
+        max_length=2048,
+        pack_to_max_length=False,
+        remove_unused_columns=True,
+        shuffle_before_pack=True,
+        template_map_fn=dict(
+            template='xtuner.utils.PROMPT_TEMPLATE.internlm2_chat',
+            type='xtuner.dataset.map_fns.template_map_fn_factory'),
+        tokenizer=dict(
+            padding_side='right',
+            pretrained_model_name_or_path=
+            '/petrobr/parceirosbr/home/luis.afonso/.cache/huggingface/hub/models--recogna-nlp--internlm-chatbode-7b/snapshots/b3acf1fa06aa64d49a6f725864316f35dcc885e2',
+            trust_remote_code=True,
+            type='transformers.AutoTokenizer.from_pretrained'),
+        type='xtuner.dataset.process_hf_dataset',
+        use_varlen_attn=False),
+    num_workers=0,
+    sampler=dict(shuffle=True, type='mmengine.dataset.DefaultSampler'))
+use_varlen_attn = False
+visualizer = None
+warmup_ratio = 0.03
+weight_decay = 0
+work_dir = './work_dirs/internlm2_chat_7b_qlora_adalberto'