If 4-bit = 8GB model would 8-bit = 16GB? And how much better would it be?
#40
by
Wildstar50
- opened
Curious, per the title. Would an 8-bit version of this model perform noticeably better and still fit into a 3090 / 4090?
there are no 8 bit models you just load 16 bit in 8 bit mode
but 8 bit is very slow for some reason. response quality is slightly better.
rather than using 13b in 8 bit mode you better use 30b in 4 bit mode or even 3 bit mode
Excellent, thanks!
Wildstar50
changed discussion status to
closed