From d497503b5c86b6e4cf21eb9f1f4eaa4e60e1b1f5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dominik=20Mach=C3=A1=C4=8Dek?= Date: Wed, 10 Apr 2024 18:13:07 +0200 Subject: [PATCH] COntributions at README.md + nicer formatting + #77 --- README.md | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index ae63e19..707e58d 100644 --- a/README.md +++ b/README.md @@ -3,16 +3,12 @@ Whisper realtime streaming for long speech-to-text transcription and translation **Turning Whisper into Real-Time Transcription System** -Demonstration paper, by Dominik Macháček, Raj Dabre, Ondřej Bojar, 2023 +Demonstration paper, by [Dominik Macháček](https://ufal.mff.cuni.cz/dominik-machacek), [Raj Dabre](https://prajdabre.github.io/), [Ondřej Bojar](https://ufal.mff.cuni.cz/ondrej-bojar), 2023 -Abstract: Whisper is one of the recent state-of-the-art multilingual speech recognition and translation models, however, it is not designed for real time transcription. In this paper, we build on top of Whisper and create Whisper-Streaming, an implementation of real-time speech transcription and translation of Whisper-like models. Whisper-Streaming uses local agreement policy with self-adaptive latency to enable streaming transcription. We show that Whisper-Streaming achieves high quality and 3.3 seconds latency on unsegmented long-form speech transcription test set, and we demonstrate its robustness and practical usability as a component in live transcription service at a multilingual conference. +Abstract: Whisper is one of the recent state-of-the-art multilingual speech recognition and translation models, however, it is not designed for real-time transcription. In this paper, we build on top of Whisper and create Whisper-Streaming, an implementation of real-time speech transcription and translation of Whisper-like models. Whisper-Streaming uses local agreement policy with self-adaptive latency to enable streaming transcription. We show that Whisper-Streaming achieves high quality and 3.3 seconds latency on unsegmented long-form speech transcription test set, and we demonstrate its robustness and practical usability as a component in live transcription service at a multilingual conference. -Paper PDF: -https://aclanthology.org/2023.ijcnlp-demo.3.pdf - - -Demo video: https://player.vimeo.com/video/840442741 +[Paper PDF](https://aclanthology.org/2023.ijcnlp-demo.3.pdf), [Demo video](https://player.vimeo.com/video/840442741) [Slides](http://ufallab.ms.mff.cuni.cz/~machacek/pre-prints/AACL23-2.11.2023-Turning-Whisper-oral.pdf) -- 15 minutes oral presentation at IJCNLP-AACL 2023 @@ -228,12 +224,20 @@ In more detail: we use the init prompt, we handle the inaccurate timestamps, we re-process confirmed sentence prefixes and skip them, making sure they don't overlap, and we limit the processing buffer window. -Contributions are welcome. - ### Performance evaluation [See the paper.](http://www.afnlp.org/conferences/ijcnlp2023/proceedings/main-demo/cdrom/pdf/2023.ijcnlp-demo.3.pdf) +### Contributions + +Contributions are welcome. We acknowledge especially: + +- [The GitHub contributors](https://github.com/ufal/whisper_streaming/graphs/contributors) for their pull requests with new features and bugfixes. +- [The translation of this repo into Chinese.](https://github.com/Gloridust/whisper_streaming_CN) +- [Ondřej Plátek](https://opla.cz/) for the paper pre-review. +- [Peter Polák](https://ufal.mff.cuni.cz/peter-polak) for the original idea. +- The UEDIN team of the [ELITR project](https://elitr.eu) for the original line_packet.py. + ## Contact