huggingface/bloom_demo · racist response

Thanks for reporting this completion, it's important to know what prompts can trigger unwanted replies!

I've tried stress-testing variations of the prompt you used without getting comparable results so it's likely a rare completion, but good to know it's a possibility even if remote.

We do not have a block list for the outputs, we thought it would introduce more problems than it fixed. We have written extensively about the trade-offs in developing the corpus and model, you can read more in the following paper and linked works: BLOOM paper. In the corpus, the word "negre" is mostly present in scientific articles on HAL (a repository of open French scientific articles and theses) and in a different historical context - but also in the OpenSubtitles. You can get more information about how a completion relates to the training dataset by using the ROOTS exploration tool.