Spaces:
Running
on
CPU Upgrade
Model deleted from Pending
Hi,
I have a new kind of model that's quite large, called dnhkng/Large.
As it's beyond the 100B parameter limit for BFloat16, so I uploaded a bitsandbytes 4bit version (dnhkng/Large-bnb-4bit) for testing on the Leaderboard. In my personal tests, this model does very well, and fits in 50Gb on an H100.
However, I see it was just deleted from the Pending list! Is there a reason for this? Should I just resubmit?
The model was generated with new technique, and I think it should be tested, even if it's handicapped to 4-bit. The BFloat16 performs better, but I understand if you don't want to test it, as it needs 3x H100s for inference.
But the 4bit model runs on the standard transformers code, and runs on 50Gb VRAM, so should be fine for Leaderboard submission.
I would also be happy to sponsor the full run using the BFloat16 model.
OK, I have resubmitted dnhkng/Large-bnb-4bit
If there are any issues with this model, please let me know so I can correct them for a proper submission.
And it's gone, again!
Could someone at least explain what the issue is?
@clefourrier Is there a problem with the number of parameters? I would have thought if it fits on an 80GB card, the absolute number of params is not relevant.
Hi @dnhkng ,
Please start by reading our FAQ (in our documentation, linked both at the top and in the submit tab).
Notably,
- when you report a problem with a model, we need you to point to the request file so we can investigate
- you should avoid re-submitting models when they don't seem to work, as it's adding useless strain on our system (when evals go through, your model will be evaluated two times, which is a waste of compute).
- please be patient - you opened this issue less than 24h ago and already sent 4 messages in it + tagged a maintainer. We are looking at all issues daily but are not necessarily in the same time zone as you are.
Sorry for the spam! I was making updated posts to track what was happening as I tried things, but I went overboard 😅.
I didn't mean to be annoying; you are doing the community a great service, thanks for the effort!
The submission request is here:
https://huggingface.co/datasets/open-llm-leaderboard/requests/resolve/main/dnhkng/Large-bnb-4bit_eval_request_False_4bit_Original.json
UPDATE:
I've renamed the model just now, so the new technique for creating it is in the name:
dnhkng/RYS-Huge-bnb-4bit
Once the results are here, I will include them in the paper 😃
Update: The model failed to run. When someone has time, please let me know if its something I can fix!
It appeared to be an issue from our side, but everything is fixed and I've relaunched your model – hope it will be fine :)
I close this discussion, please, ping me here if you encounter any other problems with this exact model or start a new one
Hi
@dnhkng
,
I tried to re-run your model but it's no longer available.
It looks like you renamed it to dnhkng/RYS-Huge-bnb-4bit so I relaunched this one. Please leave your models public/don't rename them when we're investigating evaluations next time please.
I'm not having much luck. My other model, RYS-XLarge just failed too:
Hi! Network error, passed it to pending again
BTW, dnhkng/RYS-Huge-bnb-4bit also still shows up as failed.
Hi!
Yes, I had relaunched it manually.
I have deleted my model dnhkng/RYS-Huge-bnb-4bit (I made some errors when it was created, which is why it scored badly), but its still on the leaderboard. Could this be removed please?
I'm not sure how to do it myself.
Hi!
You're not supposed to do it yourself, you're supposed to ask here ^^ - is the above request file the correct one?
Yes, dnhkng/RYS-Huge-bnb-4bit_eval_request_False_4bit_Original.json. is the one that should be deleted :)
I see my new model worked pretty well. I selected it based on a new dataset I made of under 100 samples. I heard you on the Latent Spaces podcast so I thought you might be interested in new datasets.
Thanks, changed its status to DELETED, it should be removed from display at the next leaderboard restart.
Closing the issue, but cc
@alozowski
: we should add the info on how to delete a model in the FAQ.
Thanks for having listened to the Latent Spaces podcast! I'm indeed working on new evaluation datasets atm, you can send me an email at [email protected]