Artefacts of new datasets
It seems new data sources affect the model responses, especially Synthia's orientation on chain/tree of thought. Well, this all is very subjective, so I'm not sure this issue is actionable, but here is the example of a response I got which I'd consider not very good:
> Write an interactive sci-fi story with AI and space traveling. You should provide me a choice with options and I'll choose one of them. The story should be intriguing and never end.
Key Information Components:
1. Interactive story
2. Sci-fi
3. Space traveling
4. Choose from a list of options
5. Intriguing and never-ending story
Sub-Questions:
1. What kind of space travel scenario would be intriguing and never-ending?
2. How can we incorporate options for the user to choose from?
3. How can AI be used in the story?
Intermediate Thoughts:
1. A scenario where the characters are explorers on a ship that has been traveling through space for centuries, discovering new planets and civilizations.
2. The user can choose the type of planet the characters visit, the type of civilization they encounter, and the challenges they face.
3. The story could be never-ending by having the characters encounter recurring threats or problems that require repeated solutions.
4. AI could be used in the story as a helpful assistant on the ship, or as a character that the crew encounters on their journey.
5. The story could be made more interactive by allowing the user to make choices that affect the outcome of the story, such as which planet to visit or which course of action to take when faced with a problem.
Thought Evaluation:
1. The first thought provides a good starting point for the story, as space exploration is a common sci-fi theme and provides many opportunities for interesting scenarios.
2. The second thought is relevant as it allows the user to shape the story to their liking, making it more engaging.
3. The third thought is a good way to make the story never-ending, as recurring problems can keep the story interesting and allow for repeated solutions.
4. The fourth thought is relevant as it incorporates AI into the story, which is a common theme in sci-fi.
5. The fifth thought is a good way to make the story more interactive and engaging for the user.
Chain of Reasoning:
The story begins with a group of space explorers on a ship that has been traveling through space for centuries, discovering new planets and civilizations. The user can choose the type of planet the characters visit, the type of civilization they encounter, and the challenges they face. The story could be never-ending by having the characters encounter recurring threats or problems that require repeated solutions. AI could be used in the story as a helpful assistant on the ship, or as a character that the crew encounters on their journey. The story could be made more interactive by allowing the user to make choices that affect the outcome of the story, such as which planet to visit or which course of action to take when faced with a problem.
Final Answer:
In this interactive sci-fi story, you are part of a team of explorers aboard a spaceship that has been traveling through space for centuries. As we journey through the unknown, we encounter various planets and civilizations, each with their own unique challenges. You get to choose the type of planet we visit, the type of civilization we encounter, and the challenges we face. However, recurring threats and problems keep our journey never-ending, as we have to constantly adapt and find new solutions to keep us safe. Along the way, we are aided by an AI system that helps us navigate the vastness of space and makes our journey more manageable. The choices you make will shape the outcome of our story, as each decision has its own consequences. So, are you ready to embark on this never-ending journey through space?
I just wanted a story, not reasoning. It actually might be a good thing to have such introduction so that model remembers what it should do, but in the following story it never introduced an AI and kept repeating the same choice. This model is quite good with facts, but there are many requests where I'd prefer the response style of dolphin-2.2.1-mistral-7b.
Try explaining your desire to the model in the prompt.
Thank you for showing me this
If it shows up a lot I can remove Synthia for Dolphin 3.0
I just could have been unlucky, as other my attempts were with normal responses. But let's see what others think.
Here is the exact parameters of my unlucky attempt if someone wants to reproduce: llama -t 24 -m dolphin-2.5-mixtral-8x7b.Q8_0.gguf --color -c 0 --repeat-penalty 1 --chatml --prompt 'You are Dolphin, a helpful AI assistant.'
repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
main: seed = 1702564147
llama-cpp revision: 55e87c3749cb4985c3b316984d40e00e4df4a5d0
And request: Write an interactive sci-fi story with AI and space travelling. You should provide me a choise with variants and I'll choose one of them. The story should be intriguing and never end.
try modifying the system prompt
I've seen this myself a few times. I'll definitely consider this when I'm training dolphin 3.0.
Synthia author mentioned that mixtral's MoE might be a bit more sensitive to the learning rate