Finetuning Whisper for translation tasks

[WassayS]

I have an audio dataset of specific domain in the Hindi language, and I want to enhance the whisper translation capabilities of my model. Currently, it can take non-English audio input and translate it into English text.

I understand how to fine-tune the whisper model for transcription tasks like writing same language text as in audio but I'm not sure how to fine-tune it specifically for cross-lingual translation when audio is in another language and we want to improve translation to English performance of whisper model. Could you provide guidance on how to fine-tune the model for this purpose or share any repo?

[AmgadHasan]

Hi.
Did you have any success with this?

[emanueleielo]

I have an audio dataset of specific domain in the Hindi language, and I want to enhance the whisper translation capabilities of my model. Currently, it can take non-English audio input and translate it into English text.

I understand how to fine-tune the whisper model for transcription tasks like writing same language text as in audio but I'm not sure how to fine-tune it specifically for cross-lingual translation when audio is in another language and we want to improve translation to English performance of whisper model. Could you provide guidance on how to fine-tune the model for this purpose or share any repo?

I read somewhere that to fine tune on this task you can follow this guide fine tuning whisper and just change the dataset and set translation as task instead of transcribe.

I need to do the same but I still didn't tried. My worry is the calculation of WER, how can the wer be calculated in the task of translation? There are a lot of possibilities that the text predicted will have the same meaning as the real text but with differents and then the WER will mislead.

Did you manage it?

[AmgadHasan]

I read somewhere that to fine tune on this task you can follow this guide fine tuning whisper and just change the dataset and set translation as task instead of transcribe.

I need to do the same but I still didn't tried. My worry is the calculation of WER, how can the wer be calculated in the task of translation? There are a lot of possibilities that the text predicted will have the same meaning as the real text but with differents and then the WER will mislead.

Did you manage it?

@emanueleielo
Yes, I managed to successfully fine-tune whisper for translation.

For evaluation, WER isn't a good metric for translation. You want to use one of the translation metrics like BLEU Score, METEOR, COMET or similar.

Hope that helps!

[rishikksh20]

Hi @AmgadHasan
Did you get better results with Fine tuning for non-English to English translation compared to normal non finetune whisper model?

[AmgadHasan]

Hi @rishikksh20
Yes. I have seen significant improvement in the model after fine-tuning. The translation accuracy is much better and there is less hallucination.

[dgoryeo]

@AmgadHasan , this sounds very promising. In your experience, how many additional hours of (audio) data set did you need to achieve the improvement?

[EmreOzkose]

Hi, @AmgadHasan , can you share how you prepare custom data?

For example, let's say language pair is en->hi.

print(custom_en_hi_dataloader["train"][0])

would be

{'audio': {'path': 'path_to_en.wav', 
           'array': ...,
           'sampling_rate': 16000},
 'sentence': 'खीर की मिठास पर गरमाई बिहार की सियासत, कुशवाहा ने दी सफाई'}

right?