--- language: - en license: apache-2.0 library_name: transformers tags: - medical - trl - trainer datasets: - ShieldX/manovyadh-3.5k metrics: - accuracy thumbnail: https://huggingface.co/ShieldX/manovyadh-1.1B-v1-chat/blob/main/manovyadh.png pipeline_tag: text-generation base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0 widget: - text: '###SYSTEM: You are an AI assistant that helps people cope with stress and improve their mental health. User will tell you about their feelings and challenges. Your task is to listen empathetically and offer helpful suggestions. While responding, think about the user’s needs and goals and show compassion and support ###USER: I don''t know how to tell someone how I feel about them. How can I get better at expressing how I feel?? ###ASSISTANT: ' model-index: - name: manovyadh-1.1B-v1-chat results: - task: type: text-generation dataset: name: ai2_arc type: arc metrics: - type: pass@1 value: 35.92 name: pass@1 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard name: Open LLM Leaderboard - task: type: text-generation dataset: name: hellaswag type: hellaswag metrics: - type: pass@1 value: 60.03 name: pass@1 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard name: Open LLM Leaderboard - task: type: text-generation dataset: name: truthful_qa type: truthful_qa metrics: - type: pass@1 value: 39.17 name: pass@1 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard name: Open LLM Leaderboard - task: type: text-generation dataset: name: winogrande type: winogrande metrics: - type: pass@1 value: 61.09 name: pass@1 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics: - type: acc_norm value: 35.92 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ShieldX/manovyadh-1.1B-v1-chat name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics: - type: acc_norm value: 60.03 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ShieldX/manovyadh-1.1B-v1-chat name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics: - type: acc value: 25.82 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ShieldX/manovyadh-1.1B-v1-chat name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics: - type: mc2 value: 39.17 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ShieldX/manovyadh-1.1B-v1-chat name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics: - type: acc value: 61.09 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ShieldX/manovyadh-1.1B-v1-chat name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics: - type: acc value: 1.74 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ShieldX/manovyadh-1.1B-v1-chat name: Open LLM Leaderboard --- # Uploaded model - **Developed by:** ShieldX - **License:** apache-2.0 - **Finetuned from model :** TinyLlama/TinyLlama-1.1B-Chat-v1.0 # ShieldX/manovyadh-1.1B-v1 Introducing ManoVyadh, A finetuned version of TinyLlama 1.1B Chat on Mental Health Counselling Dataset. BongLlama # Model Details ## Model Description ManoVyadh is a LLM for mental health counselling. # Uses ## Direct Use - base model for further finetuning - for fun ## Downstream Use - can be deployed with api - used to create webapp or app to show demo ## Out-of-Scope Use - cannot be used for production purpose - not to be applied in real life health purpose - cannot be used to generate text for research or academic purposes # Usage ``` # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig tokenizer = AutoTokenizer.from_pretrained("ShieldX/manovyadh-1.1B-v1-chat") model = AutoModelForCausalLM.from_pretrained("ShieldX/manovyadh-1.1B-v1-chat").to("cuda") config = AutoConfig.from_pretrained("ShieldX/manovyadh-1.1B-v1-chat") def format_prompt(q): return f"""###SYSTEM: You are an AI assistant that helps people cope with stress and improve their mental health. User will tell you about their feelings and challenges. Your task is to listen empathetically and offer helpful suggestions. While responding, think about the user’s needs and goals and show compassion and support ###USER: {q} ###ASSISTANT:""" prompt = format_prompt("I've never been able to talk with my parents. My parents are in their sixties while I am a teenager. I love both of them but not their personalities. I feel that they do not take me seriously whenever I talk about a serious event in my life. If my dad doesn’t believe me, then my mom goes along with my dad and acts like she doesn’t believe me either. I’m a pansexual, but I can’t trust my own parents. I've fought depression and won; however, stress and anxiety are killing me. I feel that my friends don't listen to me. I know they have their own problems, which I do my best to help with. But they don't always try to help me with mine, when I really need them. I feel as if my childhood has been taken from me. I feel as if I have no one whom I can trust.") import torch from transformers import GenerationConfig, TextStreamer from time import perf_counter # Check for GPU availability if torch.cuda.is_available(): device = "cuda" else: device = "cpu" # Move model and inputs to the GPU (if available) model.to(device) inputs = tokenizer(prompt, return_tensors="pt").to(device) streamer = TextStreamer(tokenizer) generation_config = GenerationConfig( penalty_alpha=0.6, do_sample=True, top_k=5, temperature=0.5, repetition_penalty=1.2, max_new_tokens=256, streamer=streamer, pad_token_id=tokenizer.eos_token_id ) start_time = perf_counter() outputs = model.generate(**inputs, generation_config=generation_config) print(tokenizer.decode(outputs[0], skip_special_tokens=True)) output_time = perf_counter() - start_time print(f"Time taken for inference: {round(output_time, 2)} seconds") ``` # Bias, Risks, and Limitations Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. # Training Details # Model Examination We will be further finetuning this model on large dataset to see how it performs # Environmental Impact Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). - **Hardware Type:** 1 X Tesla T4 - **Hours used:** 0.48 - **Cloud Provider:** Google Colab - **Compute Region:** India # Technical Specifications ## Model Architecture and Objective Finetuned on Tiny-Llama 1.1B Chat model ### Hardware 1 X Tesla T4 # training This model is a fine-tuned version of [TinyLlama/TinyLlama-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0) on [ShieldX/manovyadh-3.5k](https://huggingface.co/datasets/ShieldX/manovyadh-3.5k) dataset. It achieves the following results on the evaluation set: - Loss: 1.8587 ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2.5e-05 - train_batch_size: 2 - eval_batch_size: 8 - seed: 42 - gradient_accumulation_steps: 4 - total_train_batch_size: 8 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - training_steps: 400 - mixed_precision_training: Native AMP - ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:-----:|:----:|:---------------:| | 2.5894 | 0.01 | 5 | 2.5428 | | 2.5283 | 0.02 | 10 | 2.5240 | | 2.5013 | 0.03 | 15 | 2.5033 | | 2.378 | 0.05 | 20 | 2.4770 | | 2.3735 | 0.06 | 25 | 2.4544 | | 2.3894 | 0.07 | 30 | 2.4335 | | 2.403 | 0.08 | 35 | 2.4098 | | 2.3719 | 0.09 | 40 | 2.3846 | | 2.3691 | 0.1 | 45 | 2.3649 | | 2.3088 | 0.12 | 50 | 2.3405 | | 2.3384 | 0.13 | 55 | 2.3182 | | 2.2577 | 0.14 | 60 | 2.2926 | | 2.245 | 0.15 | 65 | 2.2702 | | 2.1389 | 0.16 | 70 | 2.2457 | | 2.1482 | 0.17 | 75 | 2.2176 | | 2.1567 | 0.18 | 80 | 2.1887 | | 2.1533 | 0.2 | 85 | 2.1616 | | 2.0629 | 0.21 | 90 | 2.1318 | | 2.1068 | 0.22 | 95 | 2.0995 | | 2.0196 | 0.23 | 100 | 2.0740 | | 2.062 | 0.24 | 105 | 2.0461 | | 1.9436 | 0.25 | 110 | 2.0203 | | 1.9348 | 0.26 | 115 | 1.9975 | | 1.8803 | 0.28 | 120 | 1.9747 | | 1.9108 | 0.29 | 125 | 1.9607 | | 1.7826 | 0.3 | 130 | 1.9506 | | 1.906 | 0.31 | 135 | 1.9374 | | 1.8745 | 0.32 | 140 | 1.9300 | | 1.8634 | 0.33 | 145 | 1.9232 | | 1.8561 | 0.35 | 150 | 1.9183 | | 1.8371 | 0.36 | 155 | 1.9147 | | 1.8006 | 0.37 | 160 | 1.9106 | | 1.8941 | 0.38 | 165 | 1.9069 | | 1.8456 | 0.39 | 170 | 1.9048 | | 1.8525 | 0.4 | 175 | 1.9014 | | 1.8475 | 0.41 | 180 | 1.8998 | | 1.8255 | 0.43 | 185 | 1.8962 | | 1.9358 | 0.44 | 190 | 1.8948 | | 1.758 | 0.45 | 195 | 1.8935 | | 1.7859 | 0.46 | 200 | 1.8910 | | 1.8412 | 0.47 | 205 | 1.8893 | | 1.835 | 0.48 | 210 | 1.8875 | | 1.8739 | 0.49 | 215 | 1.8860 | | 1.9397 | 0.51 | 220 | 1.8843 | | 1.8187 | 0.52 | 225 | 1.8816 | | 1.8174 | 0.53 | 230 | 1.8807 | | 1.8 | 0.54 | 235 | 1.8794 | | 1.7736 | 0.55 | 240 | 1.8772 | | 1.7429 | 0.56 | 245 | 1.8778 | | 1.8024 | 0.58 | 250 | 1.8742 | | 1.8431 | 0.59 | 255 | 1.8731 | | 1.7692 | 0.6 | 260 | 1.8706 | | 1.8084 | 0.61 | 265 | 1.8698 | | 1.7602 | 0.62 | 270 | 1.8705 | | 1.7751 | 0.63 | 275 | 1.8681 | | 1.7403 | 0.64 | 280 | 1.8672 | | 1.8078 | 0.66 | 285 | 1.8648 | | 1.8464 | 0.67 | 290 | 1.8648 | | 1.7853 | 0.68 | 295 | 1.8651 | | 1.8546 | 0.69 | 300 | 1.8643 | | 1.8319 | 0.7 | 305 | 1.8633 | | 1.7908 | 0.71 | 310 | 1.8614 | | 1.738 | 0.72 | 315 | 1.8625 | | 1.8868 | 0.74 | 320 | 1.8630 | | 1.7744 | 0.75 | 325 | 1.8621 | | 1.8292 | 0.76 | 330 | 1.8609 | | 1.7905 | 0.77 | 335 | 1.8623 | | 1.7652 | 0.78 | 340 | 1.8610 | | 1.8371 | 0.79 | 345 | 1.8611 | | 1.7024 | 0.81 | 350 | 1.8593 | | 1.7328 | 0.82 | 355 | 1.8593 | | 1.7376 | 0.83 | 360 | 1.8606 | | 1.747 | 0.84 | 365 | 1.8601 | | 1.7777 | 0.85 | 370 | 1.8602 | | 1.8701 | 0.86 | 375 | 1.8598 | | 1.7165 | 0.87 | 380 | 1.8579 | | 1.779 | 0.89 | 385 | 1.8588 | | 1.8536 | 0.9 | 390 | 1.8583 | | 1.7263 | 0.91 | 395 | 1.8582 | | 1.7983 | 0.92 | 400 | 1.8587 | ### Framework versions - PEFT 0.7.1 - Transformers 4.37.1 - Pytorch 2.1.0+cu121 - Datasets 2.16.1 - Tokenizers 0.15.1 # Citation **BibTeX:** ``` @misc{ShieldX/manovyadh-1.1B-v1-chat, url={[https://huggingface.co/ShieldX/manovyadh-1.1B-v1-chat](https://huggingface.co/ShieldX/manovyadh-1.1B-v1-chat)}, title={ManoVyadh}, author={Rohan Shaw}, year={2024}, month={Jan} } ``` # Model Card Authors ShieldX a.k.a Rohan Shaw # Model Card Contact email : rohanshaw.dev@gmail.com # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_ShieldX__manovyadh-1.1B-v1-chat) | Metric |Value| |---------------------------------|----:| |Avg. |37.30| |AI2 Reasoning Challenge (25-Shot)|35.92| |HellaSwag (10-Shot) |60.03| |MMLU (5-Shot) |25.82| |TruthfulQA (0-shot) |39.17| |Winogrande (5-shot) |61.09| |GSM8k (5-shot) | 1.74|