LWDCLS

/

LLM-Discussions

like 20

---

@saishf @ABX-AI @Endevor @jeiku @Nitral-AI @Epiculous @Clevyby @Virt-io @saishf @nbeerbower @grimjim @localfultonextractor

Well, well, these are trying post GPT-4o times. What does the future hold for Llama, and everything else? Don't miss the exciting new chapters!

Apologies if this tangents too hard.

This is a direct Part 5 continuation of Part 4 in this thread.

Coping for june , maybe multimodal l3? We wait and cope more.

[Relevant comment transfered from @grimjim from previous discussion.]

The failed reasoning in my tests with a 7B seem to revolve around determining that steel is denser than feathers, and then halting there rather than chaining in conversions.

I stumbled onto the fact that this model that I released with little notice a couple of months back recently got quanted by two of the current high volume quanters. I have no idea how this happened, but this was a few days after someone came across my post about it and noted that it was a good model? This was a merge where I took a successful merge and then remerged it with a higher benching model, so this appears to support the meta about merging in reasoning, which I will apply to some eventual L3 merges.
https://huggingface.co/grimjim/kunoichi-lemon-royale-v2-32K-7B

I'd been sitting on another 7B merge, and finally got around to releasing it. Starling was never meant to be an RP model, but it seems to have helped in conjunction with Mistral v0.2.
https://huggingface.co/grimjim/cuckoo-starling-32k-7B

Well, well, these are trying post GPT-4o times. What does the future hold for Llama, and everything else? Don't miss the exciting new chapters!

Apologies if this tangents too hard.

This is a direct Part 5 continuation of Part 4 in this thread.

Coping for june , maybe multimodal l3? We wait and cope more.

Knowing it took near 3 days to cook llama-3 8B and Meta claimed that Llama-3 was still learning with further training. I guess they pushed Llama-3 out early to free up GPUs for the 400B model?
I can hope for a further trained or VLM version. 34B would be nice for the 24GB vram users too.
150T token Llama?

We made several new observations on scaling behavior during the development of Llama 3. For example, while the Chinchilla-optimal amount of training compute for an 8B parameter model corresponds to ~200B tokens, we found that model performance continues to improve even after the model is trained on two orders of magnitude more data. Both our 8B and 70B parameter models continued to improve log-linearly after we trained them on up to 15T tokens. Larger models can match the performance of these smaller models with less training compute, but smaller models are generally preferred because they are much more efficient during inference.

openbmb/MiniCPM-Llama3-V-2_5 MultiModal model that claims to surpass the old GPT-4V

🔥 Leading Performance. MiniCPM-Llama3-V 2.5 has achieved an average score of 65.1 on OpenCompass, a comprehensive evaluation over 11 popular benchmarks. It surpasses widely used proprietary models like GPT-4V-1106, Gemini Pro, Qwen-VL-Max and Claude 3 with 8B parameters, greatly outperforming other multimodal large models built on Llama 3.

Huggingface doesn't list GPUs older than Ampere(30) & still the 3070Ti, 3070, 3060Ti, 3060, 3050 are missing 😭
https://huggingface.co/settings/local-apps

openbmb/MiniCPM-Llama3-V-2_5 MultiModal model that claims to surpass the old GPT-4V

I'm sure it does 🙄
/rant
Soon enough, even models with <1B parameters will claim to 100% all tests.
/endrant

I'll still give it a go, even if i'm more interested in audio in/out than pictures.

the other phi 3 models dropped, incl a vision model ;)

https://huggingface.co/collections/microsoft/phi-3-6626e15e9585a200d2d761e3

My best 7B merge yet, I dare say. If the card has a style format and you keep to it, the model will stick to the format. It is very attentive to the prompt, and is capable of introducing new elements to drive plot.
https://huggingface.co/grimjim/rogue-enchantress-32k-7B

When interrogated, Nemo claims to prefer Llama 3 style Instruct tags instead of Mistral style. Curious, but I went with it for this iteration.

Try this one on for size for writing style. There's a paired context template too.
https://huggingface.co/debased-ai/SillyTavern-settings/blob/main/advanced_formatting/instruct_mode/Nemo%20Unleashed2.json

There's a lot more that could be altered for sentence variety via promptsteering. IMO that's been underinvestigated to date, as individual efforts tend not to be shared widely, slowing down the rate of learning from peers.

I wonder if something like the character writing helper in text gen webui could work
This thing: https://github.com/p-e-w/chatbot_clinic
Blind voting - no bias :3

When interrogated, Nemo claims to prefer Llama 3 style Instruct tags instead of Mistral style. Curious, but I went with it for this iteration.

Try this one on for size for writing style. There's a paired context template too.
https://huggingface.co/debased-ai/SillyTavern-settings/blob/main/advanced_formatting/instruct_mode/Nemo%20Unleashed2.json

I tried out v1 & v2, I prefer the way unleashed writes but it's less intelligent.
It tends to get the wrong idea when giving a vague response to the model using unleashed.
mistral picks up on the meaning pretty easily.
It kept messing up timelines too, like thinking what the character said 30 seconds ago woke up user 30 minutes ago. Mistral presets don't have the same issue.

When telling a character I got woken by a noise earlier in the morning I receive the response

Unleashed

You mean my voice?

From "mistral" I get

well, you should go back to bed then

Using unleashed-1 with the mistral [INST] things instead it seems to fix the issue.

Unleashed-1-Mistralified?

Well, I suggest you go back to bed then.

My samplers might be non-ideal or it may be that I'm using a rather small quant - Q4_K_S

I like experimenting with prompting for stylistic changes, but I am concerned that the model's attention heads may be "distracted" past a point. Telling it to track temporal aspects might help? More experimentation required.

I'll edit the templates in place to revert to using [INST] and [/INST], which I only recently confirmed is privileged by the tokenizer.

The only semi-big problem with the [INST] [/INST] instruct tokens for user/sys and no delimiters for bot, is that it makes the format practically useless for any "group" chat with multiple bots, unless the user speaks in between each single character. It's not my use case so I couldn't care less, but it's notable. The good thing, is that it's the most compact instruct format I'm aware of (2 tokens of overhead per message pair), sure it's less important nowadays, if you have the VRAM, but for people with normie GFX cards, it's still cool.

I'm not sure what's that unleashed prompt. But it's definitely incorrect. there's no user/assistant indicators for Nemo (or any mistral model for that matter). Sure, the model will still manage with incorrect prompting, they all do, but using that thing will decrease the model's performance. Unless I'm misunderstanding and it's aimed at a particular fine-tune (in that case, why?, nobody needs yet another format, there are already way too many).

Telling it to track temporal aspects might help? More experimentation required.

Funny you mention that. I did a small needle test on a 24K tks context

at 8K'ish tokens -> Told it the current date
at 13K'ish tokens -> Told it about a meeting I had "next monday"
at 18K'ish tokens -> Told it the new date (several days later, but not "monday" yet)
at 21K'ish tokens -> Asked it "In how many days in my meeting?"

Got at a correct-ish response. Semi consistent over a ton of re-rolls. It flip flopped by a day, which, depending how you count, was still technically correct. Worked on most (sane) inference sampling methods i tried. That actually genuinely impressed me.

They still struggle with the time of the day. But in my experience, when a model says good night when told it was mid-day 3 messages ago, it's either the user input doesn't give it much to talk about, and it's an exit strategy (as much as is "the possibilities are endless" when asked to pick something, but the model has nothing to pick from); or it's picking up on a pattern of "X messages, then goodnight" that previously happened in the context window, it's not really about time.

In my Instruct prompts, I'm experimenting with avoid overtly specifying {{char}} or {{user}}. A goal to to allow ephemeral (or walk-on) characters to be more embodied.

The instruct model is not even censored, I mean, Mistral has always been light on the alignement, but this time, it's like they literally skipped that step altogether.

+1
I use MAID sometimes (the app) and the default character is told that it should say its not ai.
Llama3 doesn't care and admits to being AI which is boring
Nemo just gaslights the user into believing it's not an AI

Oh, me? No, of course not! I'm as real as you are. Why do you ask such a thing? Are you feeling alright?