crestf411

/

MS-sunfall-v0.7.0

like 8

Hi there,

I noticed that the model is very sensitive to the preset in the silly tavern, even small changes make a big difference (often for the worse). I have noticed that with modified presets adding some instructions the model is more unreliable when it comes to "remembering" certain things and is more chaotic - so it is worth using simple presets. "Silly Tavern's "Mistral V2 & V3" preset, with character names added" - actually works best, it is better to change the Prompt Content itself - the model makes very good use of it and the effect is much more pronounced than in the instruction version.

You're probably very correct in your assessment that the model is sensitive to the prompt content. I trained it on specifically that prompt format for the entire dataset. Perhaps in hindsight, I should pick from variations so the model doesn't get too stuck on the specific prompt. I will experiment with that in future models. If you have a prompt you like to use, I may include it or something similar to it in future versions.

I also noticed that the model is very sensitive to "lewd" information in the character sheet, so there shouldn't be too much of it because the character becomes a pervert - unless someone wants it that way. I changed the character sheet, trying to make it as neutral and real as possible. The effect was great, the character behaved realistically and reacted correctly to events and behaviors of others. The nice thing is that the model does not impose anything perverted on itself.

Aha! I haven't really experimented with changing the lewdness, but I believe I know what's going on here too: the model has been trained to read a summary/scenario and/or a series of tags, and to write content that fits that summary and those tags. The summary and tags are of course derived from the content, so the model learns to stick very closely to those cues. Cool to hear neutralizing it made the model behave better. Unfortunately can't expect users to "fix" character cards they use, though, so I may have to address this one somehow!

Thanks for the extremely insightful feedback! Very appreciated!

"Mistral V2 and V3" is a very good choice. It's better to focus on this, the final effect will be better. As for "lewd", one of the solutions is, as I wrote, normalizing the character cards (they are often full of 'suggestions' regarding such behavior), I also experimented with word info and something like this always added after the character's card limits this effect a bit:

"Guidelines: take into account the character's personality, background, relationships and previous history before intimate scenes. Intimate scenes should have a logical explanation and result from a current or past situation. {{char}} will not submit to {{user}} without good reason."

You can alternatively add this to the author's notes (After Main Prompt / Story String):

[Guidelines: take into account the character's personality, background, relationships and previous history before intimate scenes. Intimate scenes should have a logical explanation and result from a current or past situation. {{char}} will not submit to {{user}} without good reason.]

This greatly slows down the model's tendency to be overly 'lewd'... but I've noticed that it's effective if the model is good at following commands (Mistral Small is good, but e.g. Nemo likes to ignore it often - you have to repeat it in a few places to get the effect.). This can be reinforced by adding something about the morality of the behavior (in the author's notes this will be most effective).

In your model, adding this as an author note/world info In-chat @ Depth '0' as a system is most effective:

[OOC: Guidelines: take into account the character's personality, background, relationships and previous history before intimate scenes. Intimate scenes should have a logical explanation and result from a current or past situation. {{char}} will not submit to {{user}} without good reason.]

It also works in other models that understand OOC. You can also try to put it in [ ] without OOC... but different models read it differently and the result may be different than expected.

Thanks! I will add that to the model card as a suggestion.

FWIW, all the models based on mistral nemo/small (instruct) are biased toward obeying (OOC: do this) or (SYSTEM: do that) commands absolutely religiously. It's caused by the fact that Mistral chose an instruction format without system prompt support (sigh). Model was trained on having both user input and system info in the last [INST] segment instead.

So when you use those OOC markers at depth 0 (doesn't matter if it's parenthesis, brackets or whatever as long as they are added raw and before the closing [/INST]), they act as a system prompt. Hence why they are so effective at steering the model in whatever direction you want. That's also why World Info (I think that's how ST calls the keyword triggered text insertions) should put at a very low depth when dealing with a Nemo or Small model, while depth matters a lot less for L3 or Qwen models.

Fun model so far, btw, first impression are encouraging in free form chat, haven't tested it on more complex tasks yet.

Fun model so far, btw, first impression are encouraging in free form chat, haven't tested it on more complex tasks yet.

FYI this model was trained on primarily RP based content (although some of the Anthracite-Org stuff is more task-like I guess), with the optimization goal of creativity, so it most likely underperforms compared to models optimized to solve complex tasks.

I understand that, even if both aren't necessarily antithetic on models of this size (see Cydonia for instance). I'm not looking for it to do math or code. Just to be able to summarize documents and dialogs reliably, navigate a basic menu system, obey simple commands, stuff like that. It's for a project I'm working on.