nithinraok commited on
Commit
df9cc09
1 Parent(s): 74ec759

Update README.md

Browse files

Add Fairness evaluation

Files changed (1) hide show
  1. README.md +24 -0
README.md CHANGED
@@ -270,6 +270,28 @@ The following tables summarizes the performance of the available models in this
270
 
271
  These are greedy WER numbers without external LM. More details on evaluation can be found at [HuggingFace ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard)
272
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
273
  ## NVIDIA Riva: Deployment
274
 
275
  [NVIDIA Riva](https://developer.nvidia.com/riva), is an accelerated speech AI SDK deployable on-prem, in all clouds, multi-cloud, hybrid, on edge, and embedded.
@@ -295,6 +317,8 @@ Check out [Riva live demo](https://developer.nvidia.com/riva#demos).
295
 
296
  [6] [HuggingFace ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard)
297
 
 
 
298
 
299
  ## Licence
300
 
 
270
 
271
  These are greedy WER numbers without external LM. More details on evaluation can be found at [HuggingFace ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard)
272
 
273
+
274
+ ## Model Fairness Evaluation
275
+
276
+ As presented in paper "Towards Measuring Fairness in AI: the Casual Conversations Dataset" for Fairness evaluation of Models, we evaluated parakeet-tdt-1.1b on the CausalConversations-v1
277
+ dataset and results are reported as follows:
278
+
279
+ ### Gender Bias:
280
+
281
+ | Gender | Male | Female | N/A | Other |
282
+ | :--- | :--- | :--- | :--- | :--- |
283
+ | Num utterances | 19325 | 24532 | 926 | 33 |
284
+ | % WER | 19.56 | 16.87 | 21.45 | 38.88 |
285
+
286
+ ### Age Bias:
287
+
288
+ | Age Group | $(18-30)$ | $(31-45)$ | $(46-85)$ | $(1-100)$ |
289
+ | :--- | :--- | :--- | :--- | :--- |
290
+ | Num utterances | 15956 | 14585 | 13349 | 43890 |
291
+ | % WER | 17.95 | 18.28 | 17.95 | 18.06 |
292
+
293
+
294
+
295
  ## NVIDIA Riva: Deployment
296
 
297
  [NVIDIA Riva](https://developer.nvidia.com/riva), is an accelerated speech AI SDK deployable on-prem, in all clouds, multi-cloud, hybrid, on edge, and embedded.
 
317
 
318
  [6] [HuggingFace ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard)
319
 
320
+ [7] [Towards Measuring Fairness in AI: the Casual Conversations Dataset](https://arxiv.org/abs/2104.02821)
321
+
322
 
323
  ## Licence
324