chargoddard

/

mixtralnt-4x7b-test

like 56

Yeah, I want to add, great job man, it's people like you that push fronts forward, RESPECT!

Even if this doesn't work, what a brilliant idea. Combining various Mistrals together in a MOE never even occurred to me.

I feel this could be the better way than finetuning a MoE. We don't really wanna experts be the same.

What an amusing concept. :D

Thanks for all the interest! I've made the script for making this kind of merge public, and wrote a bit about the reasoning and methodology. See my comment here.

Thank you @chargoddard and @Undi95 . You guys are absolute genius