Kabumbus commited on
Commit
fec5b7e
1 Parent(s): 3a32033

Update README.md

Browse files

To play with it in pytorch or GGML/.Net [start here](https://github.com/NethermindEth/Mpt-Instruct-DotNet-S)

## Training
Finetuned for CSharp [mosaicml/mpt-7b-instruct](https://huggingface.co/mosaicml/mpt-7b-instruct). Max context length is restricted to 1024 tokens.

'Loss': 0.256045166015625 on 300k CSharp-related records
'Loss': 0.095714599609375 on 50k specific short prompts

## Sources
data contained (most data was around 500 tokens long < 1000, except large code files):
- codeparrot/github-code C# ("mit", "Apache-2.0", "Bsd-3-clause", "Bsd-2-clause", "Cc0-1.0", "Unlicense", "isc")
- raw data Plain .cs files randomly cut at the 60-80% in the instruction, and we ask the network to continue last 40-20% (76k)
- documented static functions 72k
- SO 5q_5answer + 5q_5best (CC BY-SA 4.0) 70k
- Dotnet wiki (30k, rendered out from [github repo](https://github.com/microsoft/dotnet), see also removed, GPT-4 generated short question to each file)
- All NM Static Functions and Tests (from [nethermind client repo](https://github.com/NethermindEth/nethermind) documented and described via GPT-4 (4k)
- GPT-4 questions, GPT-3.5 answers for CSharp: Short Q->Code, Explain Code X > Step-By-Step (35k)
- GPT-4 questions, GPT-3.5 answers for nethermind client interface `IEthRpcModule `: Short Q->Code, Explain Code X -> Step-By-Step (7k)

## Contents
- HF compatible model
- GGML compatible quantisations (f16, q8, q5)

Files changed (1) hide show
  1. README.md +9 -0
README.md CHANGED
@@ -1,3 +1,12 @@
1
  ---
2
  license: cc-by-sa-3.0
 
 
 
 
 
 
 
 
 
3
  ---
 
1
  ---
2
  license: cc-by-sa-3.0
3
+ language:
4
+ - en
5
+ pipeline_tag: text-generation
6
+ tags:
7
+ - csharp
8
+ - mpt
9
+ - instruct
10
+ - 7b
11
+ - llm
12
  ---