Hugging Face's logo

SciPhi
/
SciPhi-Self-RAG-Mistral-7B-32k

Text Generation
Transformers
PyTorch
mistral
text-generation-inference
Inference Endpoints
Model card Files Community
9
SciPhi-Self-RAG-Mistral-7B-32k / README.md
emrgnt-cmplxty's picture
emrgnt-cmplxty
Update README.md
f5af0a1
preview code
|
raw
history blame
2.71 kB
metadata
license: mit

SciPhi-Self-RAG-Mistral-7B-32k Model Card

SciPhi-Self-RAG-Mistral-7B-32k is a Large Language Model (LLM) fine-tuned from Mistral-7B-v0.1. This model underwent the fine-tuning process described in the SciPhi-Mistral-7B-32k model card. It then underwent further fine-tuning on the recently released self-rag dataset. Other RAG-related instruct datasets were mixed in during this process in an effort to keep the tone of the current model. This model benchmarks well but needs further tuning to be an excellent conversationalist.

SciPhi-AI is available via a free hosted API, though the exposed model can vary. Currently, SciPhi-Self-RAG-Mistral-7B-32k is available. More details can be found in the docs here.

Model Architecture

Base Model: Mistral-7B-v0.1

Architecture Features:

  • Transformer-based model
  • Grouped-Query Attention
  • Sliding-Window Attention
  • Byte-fallback BPE tokenizer

Built with Axolotl

References

  1. Lian, W., Goodson, B., Wang, G., Pentland, E., Cook, A., Vong, C., & Teknium. (2023). MistralOrca: Mistral-7B Model Instruct-tuned on Filtered OpenOrcaV1 GPT-4 Dataset. HuggingFace repository. Link
  2. Mukherjee, S., Mitra, A., Jawahar, G., Agarwal, S., Palangi, H., & Awadallah, A. (2023). Orca: Progressive Learning from Complex Explanation Traces of GPT-4. arXiv preprint arXiv:2306.02707.
  3. Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., & Roberts, A. (2023). The Flan Collection: Designing Data and Methods for Effective Instruction Tuning. arXiv preprint arXiv:2301.13688.
  4. Mistral AI. (2023). Model Card for Mistral-7B-v0.1. The Mistral-7B-v0.1 Large Language Model (LLM) is a pretrained generative text model with 7 billion parameters. Mistral-7B-v0.1 outperforms Llama 2 13B on all benchmarks tested. For full details, please refer to the paper and release blog post. Model Architecture: Transformer with Grouped-Query Attention, Sliding-Window Attention, and Byte-fallback BPE tokenizer. Link

Acknowledgements

Thank you to the AI Alignment Lab, vikp, jph00 and others who contributed to this work.