Suggestion to deal with omission of periods

[busdriverbuddha]

There is a frequent hallucination in Whisper in which segments of the transcript are stripped of a period or full stop. Example (not a real transcription, just to illustrate the issue:

Meghan Elizabeth Trainor is an American singer-songwriter and television personality She rose to prominence after signing with Epic Records in 2014 and releasing her debut single All About That Bass, which reached number one on the U.S. Billboard Hot 100 chart and sold 11 million copies worldwide Trainor has released five studio albums with the label and has received various accolades, including the 2016 Grammy Award for Best New Artist.

I have found that adding about 5 seconds of whitenoise to the beginning of the affected excerpt and retranscribing it usually corrects the punctuation.

Perhaps this could be incorporated to the code. Or, if there were a way to separate the affected region (e.g. with information from the VAD), a separate function could be written to check for this hallucination, export the WAV for the affected region and retranscribe.

[shashikg]

Hi @busdriverbuddha

Possibly these are some of the failure modes of whisper's LLM based decoder.

I have found that adding about 5 seconds of whitenoise to the beginning of the affected excerpt and retranscribing it usually corrects the punctuation.

Interesting, do you have any detailed evaluation on it? Like how much punctuation accuracy improves after adding this? Also any effects on the WER? One issue I see in this approach is that it will unnecessarily increase the inference time.

Can you try this and check if it helps?

files = ['audio.wav']
lang_codes = ['en']
tasks = ['transcribe']
initial_prompts = ['This is a documentary about Meghan Elizabeth.']

out = model.transcribe_with_vad(files,
                                lang_codes=lang_codes,
                                tasks=tasks,
                                initial_prompts=initial_prompts,
                                batch_size=32)

PS: This thing is also in my roadmaps on how to use prompting with whisper model to align the transcription format.

[shashikg]

If you can provide me one sample file, I can try looking into it if VAD margins can be somehow used to improve these issues.

[busdriverbuddha]

I can supply an MP3 file in which this issue happens predictably. How can I share it with you privately?

[shashikg]

You can email me: shashikg.iitk@gmail.com

[busdriverbuddha]

Done. Thank you!

…

On Thu, Feb 1, 2024 at 12:10 PM Shashi Kant ***@***.***> wrote: You can email me: ***@***.*** — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACBPVXW22IDKZQTKQQ2NF7LYROV5PAVCNFSM6AAAAABCTPCSJCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRRGU2TCOJXGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[shashikg]

Hi got your email. I will get back to you by coming weekend.

[busdriverbuddha]

All right, thank you!

…

On Tue, Feb 6, 2024 at 5:15 AM Shashi Kant ***@***.***> wrote: Hi got your email. I will get back to you by coming weekend. — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACBPVXSXCX4G3MDZJTEX3KLYSHRBHAVCNFSM6AAAAABCTPCSJCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRYHE4DOMJWGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>