Hugging Face's logo

Spaces:
giskardai
/
giskard-evaluator
Sleeping

App Files Community
200

Report for bhadresh-savani/distilbert-base-uncased-emotion

#144
by ZeroCommand - opened
Discussion
ZeroCommand
 Giskard org

Hi Team,

This is a report from Giskard Bot Scan 🐢.

We have identified 2 potential vulnerabilities in your model based on an automated scan.

This automated analysis evaluated the model on the dataset dair-ai/emotion (subset split, split validation).

ZeroCommand
 Giskard org
👉Performance issues (1)

For records in the dataset where text contains "know", the Precision is 5.25% lower than the global Precision.

Level Data slice Metric Deviation
medium 🟡 text contains "know" Precision = 0.885 -5.25% than global

Taxonomy

avid-effect:performance:P0204
['\n
\n 🔍✨Examples\n\n\n| | text | label | Predicted `label` |\n|----:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------|:--------------------|\n| 17 | i know what it feels like he stressed glaring down at her as she squeezed more soap onto her sponge | anger | sadness (p\xa0=\xa00.98) |\n| 91 | i feel like the people i know are really generous and i have my needs met | joy | love (p\xa0=\xa00.80) |\n| 164 | i have stayed at heritage christian because of the fulfillment that i feel in doing christ s work in action by being the hands the eyes the legs and the voice of supporting the individuals that i have been blessed to know and support | joy | love (p\xa0=\xa00.82) |\n\n
\n\n\n\n\n
\n\n
\n'] Examples are too long to be displayed in this area.
ZeroCommand
 Giskard org
👉Robustness issues (1)

When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 22.2% of the cases. We expected the predictions not to be affected by this transformation.

Level Metric Transformation Deviation
major 🔴 Fail rate = 0.222 Add typos 222/1000 tested samples (22.2%) changed prediction after perturbation

Taxonomy

avid-effect:performance:P0201
['\n
\n 🔍✨Examples\n\n\n| | text | Add typos(text) | Original prediction | Prediction after perturbation |\n|-----:|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------|:--------------------------------|\n| 656 | i feel a little bit more nostalgic when those memories come to mind | i feel a little bit more nosftalic when those memories comwe to mind | love (p\xa0=\xa01.00) | joy (p\xa0=\xa00.99) |\n| 734 | i can talk to her about almost anything i want to and she just listens and she doesnt make me feel like a whiney brat and she helps me sort my thoughts and make decisions while keeping me where she feels im safe | i can talk to her about almost anything i want to and she just lisrens and she doesnt make me feel liek a shiney brat and she helps me sort my thoughts and make decisions while keeping me where she fes im safe | sadness (p\xa0=\xa01.00) | joy (p\xa0=\xa01.00) |\n| 1403 | i feel the need to preface this by saying that i am strongly in favor of keeping violent or otherwise inappropriate videogames out of the hands of minors and i believe that this is an issue that parents and the government need to work on together | i feel the need to preface this by saying that i am ateongly in faor of keeping volent or otherwise inappropriate videogames outo f yhe hands of minor san di believe that this is an issue that parents and the government need to work on tovether | anger (p\xa0=\xa01.00) | sadness (p\xa0=\xa00.88) |\n\n
\n\n\n\n\n
\n\n'] Examples are too long to be displayed in this area.

Sign up or log in to comment