Improve decoding

[ggerganov]

ref #278 #133 #172 #255 #270

The goal of this PR is to reach OpenAI decoding parity and potentially go beyond

There are several ideas for improving the decoding strategy that will be explored.
There is some chance that these ideas will improve segment and token timestamp precision, but no guarantees.

---

Implemented decoding strategies

• Average log probability threshold support
  Decoded sequences can be discarded based on the average logprob of the tokens. When the avg logprob is below the threshold, it means that the model wasn't very confident in the transcription and we need to apply some fallback strategy to generate a better sequence
• Entropy-based threshold support
  This is similar to OpenAI's compression ration threshold logic used to determine if a sequence is too repetitive. However, in whisper.cpp instead of using zlib compression, we use a basic entropy metric H = -sum(p*log(p)) of the last 32 tokens in the sequence to determine if the decoding has degraded in endless repetition. Low entropy means more repetition. This approach has to be further tested - probably the entropy threshold might need some adjustments
• Temperature support
  By default, the decoding starts with T = 0, deterministically sampling the best token each time based on the computed logits. Upon failure, we increase the temperature and we start sampling the tokens from a discrete probability distribution obtained by scaling the logits with 1/T
• The Greedy decoding strategy
  Uses --best-of number of independent decoders for T > 0. Each decoder keeps a separate decoding sequence. At temperature T > 0.5 we clear any previous context. The rationale is that sometimes the context can confuse the decoder and drive it into a failure case
• The BeamSearch decoding strategy
  At T = 0 we start with --beam-size independent decoders. Each one generates the top --beam-size sequences from it's current state. From all generated candidate sequences, we pick the top --beam-size based on the logprob sum of their tokens and reassign them to the decoders. Upon failure, we increase the temperature and fallback to the Greedy strategy. The BeamSearch decoder is --beam-size times more computationally heavy than the Greedy decoder
  I think it is worth exploring a strategy which initially uses 1 beam at T = 0 and only activates --beam-size decoders upon failure. This would significantly speed-up the processing and I hope it will keep the transcription quality high. Will probably add a flag for that

---

Development notes

• best_of is used only by the Greedy decoder at temperature > 0

• beam_size is used by the BeamSearch decoder

• Both best_of and beam_size require to maintain a separate KV memory for each decoder stream. Need changes both in whisper.h interface + whisper_context / whisper_model to support that. Introduce whisper_decoder

• compression_ratio heuristic might be out-of-scope - I cannot implement zlib.compress from scratch. Maybe use something simpler, like n-gram entropy?

  ChatGPT brainstorming

  
  

• Clear past prompt only for temperature >= 0.5:

  

• patience controls the max number of sequences to obtain from the beam search

• BeamSearch algorithm is explained here: https://arxiv.org/pdf/2204.05424.pdf

• For each decoded sequence, maintain avg_logprob of the tokens in order to implement logprob_threshold fallback:
  https://github.com/openai/whisper/blob/0b1ba3d46ebf7fe6f953acfd8cad62a4f851b49f/whisper/transcribe.py#L119-L120

• Sequence ranking with and without length_penalty:
  https://github.com/openai/whisper/blob/0b1ba3d46ebf7fe6f953acfd8cad62a4f851b49f/whisper/decoding.py#L169-L192

[ggerganov]

This should be pretty close to OpenAI's decoding implementation.
There might be a few bugs left, but I think it's pretty much ready to merge.

The few failure cases that I have are now correctly transcribed using either --best-of 5 or --beam-size 5
By default, I am thinking about leaving beam-search off and have a temperature + --best-of 5 fallback.

[stevevaius2015]

Thank you for your immense work and this wonderful project

[RndyP]

Getting a potential uninitialized variable tid on line 3124 of Whisper.cpp reported by VC++

[RndyP]

First, thanks for all the hard work on this!

I am playing around with 1.1.0 as I write this.

Still have the issue that was closed in #172. The problem is worse now in that the "echos" may eat up CPU like crazy. My test case is to repeat the number "six" multiple times. (Sorry about the 666 humor) I send whisper an audio block of about 4 seconds with the word "six" repeated at least 3 times, and Whisper will now, instead or returning with a large number of sixes, will crunch for up to 1 minute and return various odd strings.

[ggerganov]

@RndyP
Thanks for the reports - these are very useful.
The "crazy CPU" problem is due to the new entropy-based threshold for discarding repetitive sequences. I will fine-tune the parameters to try to avoid the case that you observe and also add parameters to enable/disable and control this functionality for the main example.

[dennislysenko]

Thanks for the fixes 👍