Goekdeniz-Guelmez/J.O.S.I.E.3-Beta1.1-slerp-must-reshape

Model card of JOSIE_Beta_1.1_7B

This is my Token customized Isaak-Carter/JOSIE_Beta-1-7B-slerp model

[Origional Model][https://huggingface.co/Isaak-Carter/JOSIE_Beta-1-7B-slerp]

This is based on Isaak-Carter/JOSIE_Beta-1-7B-slerp model with added custom special Tokens. This wil most likely be my next Model, trained on my own Dataset.

New added Special Tokens

'<|functions|>',
'<|system|>',
'<|gökdeniz|>',
'<|user|>',
'<|josie|>',
'<|assistant|>',
'<|function_call|>',
'<|function_response|>',
'<|image|>',
'<|long_term_memory|>',
'<|short_term_memory|>',
'<|home_state|>',
'<|current_states|>',
'<|context|>'

New BOS and EOS Tokens

BOS = '<|startoftext|>'
EOS = '<|endoftext|>'

Model Architecture:

MistralForCausalLM(
  (model): MistralModel(
    (embed_tokens): Embedding(32016, 4096)
    (layers): ModuleList(
      (0-31): 32 x MistralDecoderLayer(
        (self_attn): MistralSdpaAttention(
          (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (k_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (v_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (rotary_emb): MistralRotaryEmbedding()
        )
        (mlp): MistralMLP(
          (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (up_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (down_proj): Linear(in_features=14336, out_features=4096, bias=False)
          (act_fn): SiLU()
        )
        (input_layernorm): MistralRMSNorm()
        (post_attention_layernorm): MistralRMSNorm()
      )
    )
    (norm): MistralRMSNorm()
  )
  (lm_head): Linear(in_features=4096, out_features=32016, bias=False)
)