Is there a paper describing this method somewhere?
Hi,
This looks quite interesting. Is there a paper/blog/article giving details on how this model was trained and tested?
Thanks!
Salut Jean-Philippe, we are still working on a publication. I have invited you via LinkedIn to share the draft with you.
The model description explains how it is trained, and blind testing is done by measuring WER score/rank correlations,
and the resulting transcription quality in WER when the model is used to select the best among many ASR hypotheses.
The current model is reliable for intra-sample quality ranking of outputs from multiple ASR models or model versions.
Hence, it shall be possible to use it to approximate a/b testing on production data streams without having references.
We will also have an improved version of this model (especially in terms of inter-sample ranking) available in aiXplain.
The new model will also have the potential for helping with challenge set selection from the production data streams.
Hello! I'm using this model to quickly estimate the quality of a large number of ASR transcripts, and it seems to be working well! I would also be keenly interested in seeing any publication associated with the model.
Hello Steven, you can find the ICASSP and InterSpeech publications below:
https://arxiv.org/abs/2306.13114
https://arxiv.org/abs/2306.12577
Here is a more recent model (the second paper) that is sometimes better.
https://github.com/aixplain/NoRefER
Please let me know in case you would have any questions. Have a nice day.