|
---
|
|
license: cc-by-nc-sa-4.0
|
|
language:
|
|
- en
|
|
- de
|
|
- zh
|
|
- fr
|
|
- nl
|
|
- el
|
|
- it
|
|
library_name: transformers
|
|
pipeline_tag: audio-classification
|
|
tags:
|
|
- HuBERT
|
|
- Speech Emotion Recognition
|
|
- SER
|
|
- PyTorch
|
|
---
|
|
|
|
# **ExHuBERT: Enhancing HuBERT Through Block Extension and Fine-Tuning on 37 Emotion Datasets** |
|
Authors: Shahin Amiriparian, Filip Packań, Maurice Gerczuk, Björn W. Schuller |
|
|
|
Fine-tuned [**HuBERT Large**](https://huggingface.co/facebook/hubert-large-ls960-ft) on EmoSet++, comprising 37 datasets, totaling 150,907 samples and spanning a cumulative duration of 119.5 hours. |
|
The model is expecting a 3 second long raw waveform resampled to 16 kHz. The original 6 Ouput classes are combinations of low/high arousal and negative/neutral/positive |
|
valence. |
|
Further details are available in the corresponding [**paper**](https://arxiv.org/) |
|
|
|
**Note**: This model is for research purpose only. |
|
|
|
### EmoSet++ subsets used for fine-tuning the model: |
|
|
|
| | | | | | |
|
| :---: | :---: | :---: | :---: | :---: | |
|
| ABC | AD | BES | CASIA | CVE | |
|
| Crema-D | DES | DEMoS | EA-ACT | EA-BMW | |
|
| EA-WSJ | EMO-DB | EmoFilm | EmotiW-2014 | EMOVO | |
|
| eNTERFACE | ESD | EU-EmoSS | EU-EV | FAU Aibo | |
|
| GEMEP | GVESS | IEMOCAP | MES | MESD | |
|
| MELD | PPMMK | RAVDESS | SAVEE | ShEMO | |
|
| SmartKom | SIMIS | SUSAS | SUBSECO | TESS | |
|
| TurkishEmo | Urdu | | | | |
|
|
|
|
|
|
|
### Usage |
|
|
|
```python |
|
import torch |
|
import torch.nn as nn |
|
from transformers import HubertForSequenceClassification, Wav2Vec2FeatureExtractor |
|
|
|
|
|
|
|
# CONFIG and MODEL SETUP |
|
model_name = '.../HuBERT-EmoSet++' |
|
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("facebook/hubert-base-ls960") |
|
model = HubertForSequenceClassification.from_pretrained(model_name) |
|
model.classifier = nn.Linear(in_features=256,out_features=6) |
|
|
|
sampling_rate=16000 |
|
device = torch.device("cuda" if torch.cuda.is_available() else "cpu") |
|
model = model.to(device) |
|
|
|
|
|
``` |
|
|
|
### Citation Info |
|
|
|
|
|
``` |
|
@inproceedings{Amiriparian24-EEH, |
|
author = {Shahin Amiriparian and Filip Packan and Maurice Gerczuk and Bj\"orn W.\ Schuller}, |
|
title = {{ExHuBERT: Enhancing HuBERT Through Block Extension and Fine-Tuning on 37 Emotion Datasets}}, |
|
booktitle = {{Proc. INTERSPEECH}}, |
|
year = {2024}, |
|
editor = {}, |
|
volume = {}, |
|
series = {}, |
|
address = {Kos Island, Greece}, |
|
month = {September}, |
|
publisher = {ISCA}, |
|
} |
|
``` |
