admin-valentin/WhisperLiveKit
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
681 commits 1 branch 0 tags 34 MiB
Commit graph

263 commits

Author SHA1 Message Date
Quentin Fuxa
0a6e5ae9c1 ffmpeg install instruction error indicates --pcm-input alternative 2025-09-17 16:04:17 +02:00
Quentin Fuxa
ee448a37e9 when pcm-input is set, the frontend uses AudioWorklet 2025-09-17 14:55:57 +02:00
Quentin Fuxa
9c051052b0 Merge branch 'main' into ScriptProcessorNode-to-AudioWorklet 2025-09-17 11:28:36 +02:00
Quentin Fuxa
4d7c487614 replace deprecated ScriptProcessorNode with AudioWorklet 2025-09-17 10:53:53 +02:00
Quentin Fuxa
65025cc448 nllb backend can be transformers, and model size can be 1.3B 2025-09-17 10:20:31 +02:00
Quentin Fuxa
bbba1d9bb7 add nllb-backend and translation perf test in dev_notes 2025-09-16 20:45:01 +02:00
Quentin Fuxa
99dc96c644 fixes #224 2025-09-16 18:34:35 +02:00
GeorgeCaoJ
2a27d2030a feat: support web audio 16kHz PCM input and remove ffmpeg dependency 2025-09-15 23:22:25 +08:00
Quentin Fuxa
cd160caaa1 asyncio.to_thread for transcription and translation 2025-09-15 15:23:22 +02:00
Quentin Fuxa
5aa312e437 simulstreaming warmup is done in whisperlivekit.simul_whisper.backend.load_model, not in warmup_online 2025-09-13 20:19:19 +01:00
notV3NOM
ebaf36a8be
Fix warmup file behavior 2025-09-13 20:44:24 +05:30
Quentin Fuxa
a4e9f3cab7 support for raw PCM input option by @YeonjunNotFR 2025-09-11 21:32:11 +02:00
Quentin Fuxa
b06866877a add --disable-punctuation-split option 2025-09-11 21:03:00 +02:00
Quentin Fuxa
967cdfebc8 fix Translation imports 2025-09-11 21:03:00 +02:00
Quentin Fuxa
3c11c60126 fix by @treeaaa 2025-09-11 21:03:00 +02:00
Quentin Fuxa
2963e8a757 translate when at least 3 new tokens 2025-09-09 21:45:00 +02:00
Quentin Fuxa
cb2d4ea88a audio processor lines use now Lines objects instead of dict 2025-09-09 21:45:00 +02:00
Quentin Fuxa
add7ea07ee translator takes all the tokens from the queue 2025-09-09 19:55:39 +02:00
Quentin Fuxa
3358877054 Fix StorageView conversion for CPU/GPU compatibility 2025-09-09 15:44:16 +02:00
Quentin Fuxa
1f7798c7c1 condition on encoder_feature_ctranslate type 2025-09-09 12:16:52 +02:00
Alexander Lindberg
c7b3bb5e58 Fix regression with faster-whisper encoder_feature 2025-09-09 11:18:55 +03:00
Quentin Fuxa
f661f21675 translation asyncio task 2025-09-08 18:34:31 +02:00
Quentin Fuxa
b6164aa59b translation device determined with torch.device 2025-09-08 11:34:40 +02:00
Quentin Fuxa
4209d7f7c0 Place all tensors on the same device in sortformer diarization 2025-09-08 10:20:57 +02:00
Quentin Fuxa
334b338ab0 use platform to determine system and recommand mlx whisper 2025-09-07 15:49:11 +02:00
Quentin Fuxa
72f33be6f2 translation: use of get_nllb_code 2025-09-07 15:25:14 +02:00
notV3NOM
a178ed5c22
fix simulstreaming preload model count argument in cli 2025-09-06 18:18:09 +05:30
Quentin Fuxa
fad9ee4d21
Merge pull request #198 from notV3NOM/main 
Fix scrolling UX with sticky header controls
 2025-09-05 20:46:36 +02:00
Quentin Fuxa
d1a9913c47 nllb v0 2025-09-05 18:02:42 +02:00
notV3NOM
e4ca2623cb
Fix scrolling UX with sticky header controls 2025-09-05 21:25:13 +05:30
Quentin Fuxa
9c1bf37960 fixes #197 2025-09-05 16:34:13 +02:00
notV3NOM
abd8f2c269
Fix exponentially growing simulstreaming silence timer 2025-09-04 21:49:07 +05:30
Quentin Fuxa
f3ad4e39e4 torch.Tensor to torch.as_tensor 2025-09-04 16:39:11 +02:00
Quentin Fuxa
e0a5cbf0e7 v0.1.0 chrome extension 2025-09-04 16:36:28 +02:00
Quentin Fuxa
953697cd86 torch.Tensor to torch.as_tensor 2025-09-04 15:25:39 +02:00
Quentin Fuxa
3bd2122eb4 0.2.8 : only the decoder of whisper is loaded in memory when a different encoder is used 2025-09-02 21:12:25 +02:00
Quentin Fuxa
d5008ed828 mlx/fasterWhisper encoders are loaded once and shared in simulstreaming 2025-09-01 12:33:19 +02:00
Quentin Fuxa
d467716e26 add microphone picker 2025-08-31 10:12:52 +02:00
Quentin Fuxa
199e21b3ef faster-whisper as an optional encoder alternative for simulstreaming 2025-08-30 23:50:16 +02:00
Quentin Fuxa
1d926f2e67 mlx-whisper used as simulstreaming encoder: improve speed for macos systems 2025-08-30 22:19:11 +02:00
Quentin Fuxa
4a71a391b8 get_web_interface_html to get_inline_ui_html for embedded web interface HTML 2025-08-30 13:44:06 +02:00
Quentin Fuxa
1ba171a58d add embedded web interface HTML (single-file version with inline CSS/JS/SVG) 
### Added
- `get_inline_ui_html()`: generates a self-contained version of the web interface, with CSS, JS, and SVG assets inlined directly into the HTML. useful for environments where serving static files is inconvenient or when a single-call UI delivery is preferred.

(cherry picked from commit aa44a92a6775bcc42b3518e78acd16eda275935a)
 2025-08-29 22:00:59 +02:00
Quentin Fuxa
4a5d5e1f3b raise Exception when language == auto and task == translation 2025-08-29 17:44:46 +02:00
Quentin Fuxa
9895bc83bf auto detection of language for warmup if not indicated 2025-08-27 20:37:48 +02:00
Quentin Fuxa
ab98c31f16 trim will happen before audio processor 2025-08-27 18:17:11 +02:00
Quentin Fuxa
4ed62e181d when silences are detected, speaker correction is no more applied 2024-08-24 19:24:00 +02:00
Quentin Fuxa
9a8d3cbd90 improve diarization + silence handling 2024-08-24 19:20:00 +02:00
Quentin Fuxa
b101ce06bd several users share the same sortformer model instance 2024-08-24 19:18:00 +02:00
Quentin Fuxa
c83fd179a8 improves phase shift correction between transcription and diarization 2024-08-24 19:15:00 +02:00
Quentin Fuxa
5258305745 default diarization backend in now sortformer 2025-08-24 18:32:01 +02:00
Quentin Fuxa
ce781831ee punctuation is checked in audio-processor's result formatter 2025-08-24 18:32:01 +02:00
Quentin Fuxa
58297daf6d sortformer diar implementation v0.3 2025-08-24 18:32:01 +02:00
Quentin Fuxa
3393a08f7e sortformer diar implementation v0.2 2025-08-24 18:32:01 +02:00
Quentin Fuxa
26cc1072dd new dockerfile for cpu only. update dockerfile from cuda 12.8 to 12.9 2025-08-22 11:04:35 +02:00
Quentin Fuxa
909ac9dd41 speaker -1 are no more sent in websocket - no buffer when their is a silence 2025-08-21 14:09:02 +02:00
Quentin Fuxa
d94a07d417 default model is now base. default backend simulstreaming 2025-08-21 11:55:36 +02:00
Quentin Fuxa
b32dd8bfc4 Align backend and frontend time handling 2025-08-21 10:33:15 +02:00
Quentin Fuxa
9feb0e597b remove VACOnlineASRProcessor backend possibility 2025-08-20 20:57:43 +02:00
Quentin Fuxa
9dab84a573 update front 2025-08-20 20:15:38 +02:00
Quentin Fuxa
d089c7fce0 .html to .html + .css + .js 2025-08-20 20:00:31 +02:00
Quentin Fuxa
253a080df5 diart diarization handles pauses/silences thanks to offset 2025-08-19 21:12:55 +02:00
Quentin Fuxa
0c6e4b2aee sortformer diar implementation v0.1 2025-08-19 19:48:51 +02:00
Quentin Fuxa
e14bbde77d sortformer diar implementation v0 2025-08-19 17:02:55 +02:00
Quentin Fuxa
7496163467 rename diart backend 2025-08-19 15:02:27 +02:00
Quentin Fuxa
696a94d1ce 1rst sortformer backend implementation 2025-08-19 15:02:17 +02:00
Quentin Fuxa
2699b0974c Fix simulstreaming imports 2025-08-19 14:43:54 +02:00
Quentin Fuxa
d0e9e37ef6 simulstreaming: cumulative_time_offset to keep timestamps correct when audio > 30s 2025-08-17 09:33:47 +02:00
Quentin Fuxa
820f92d8cb audio_max_len to 30 -> 20, ffmpeg timeout 5 -> 20 2025-08-17 09:32:08 +02:00
Quentin Fuxa
e2184d5e06 better handle silences when VAC + correct offset issue with whisperstreaming backend 2025-08-17 01:27:07 +02:00
Quentin Fuxa
7fe0353260 vac model is loaded in TranscriptionEngine, and by default 2025-08-17 00:34:25 +02:00
Quentin Fuxa
0f2eba507e use with_offset to add no audio offset to tokens 2025-08-17 00:33:24 +02:00
Quentin Fuxa
55e08474f3 recycle backend in simulstreaming thanks to new remove hooks function 2025-08-16 23:06:16 +02:00
Quentin Fuxa
28bdc52e1d VAC before doing transcription and diarization. V0 2025-08-16 23:04:21 +02:00
Quentin Fuxa
1652db9a2d Use distinct backend models for simulstreaming and add --preloaded_model_count to preload them 2025-08-15 23:03:55 +02:00
Quentin Fuxa
1c42b867cf Merge branch 'main' of https://github.com/QuentinFuxa/whisper_streaming_web 2025-08-13 10:04:04 +02:00
Quentin Fuxa
d4771e563e Increase END_SILENCE_DURATION to reduce false positives 2025-08-13 10:04:00 +02:00
David Gumberg
3b96fb8776 frontend: Scroll down when appending transcription 2025-08-12 17:31:32 -07:00
David Gumberg
7f93c4b978 frontend: Don't let screen sleep when transcribing. 2025-08-12 17:30:57 -07:00
Quentin Fuxa
15c3df1cba warmup base whisper when using simulstreaming 2025-08-12 18:52:52 +02:00
Quentin Fuxa
728e1f1290 simulstreaming warmup is done for each instance of online, not for the backend 2025-08-12 18:35:04 +02:00
Quentin Fuxa
87b9ed6ecd nonspeech_prob from 1 to 0.5 2025-08-12 18:34:37 +02:00
Quentin Fuxa
38b4ebe8ba Handle 3 types of silences: Indicated by whisper, between tokens, and at the end of the input. Display them in the frontend 2025-08-11 17:56:57 +02:00
Quentin Fuxa
d098af3185 each SimulStreamingOnlineProcessor now contains PaddedAlignAttWhisper instance. SimulStreamingASR only contains loaded whisper model 2025-08-11 08:24:14 +02:00
Quentin Fuxa
4e56130a40 frontend supports dark theme 2025-08-11 08:22:23 +02:00
Quentin Fuxa
2bbdc70187 lags are now updated every 0.1s 2025-08-09 23:11:05 +02:00
Quentin Fuxa
b678a55f63 remove duplicate file 2025-08-09 23:10:34 +02:00
Quentin Fuxa
5491964e81 clean SimulStreamingOnlineProcessor initialization + audio processing 2025-08-09 20:16:27 +02:00
Quentin Fuxa
b05297a96d clean simulwhisper backend and online 2025-08-09 18:02:15 +02:00
Quentin Fuxa
197293e25e refactor(simulstreaming): extract backend + online module into separate files from whisper streaming 2025-08-08 18:07:51 +02:00
Quentin Fuxa
ba41c4ab56 Remove download_simulstreaming_backend 2025-08-08 18:06:40 +02:00
Quentin Fuxa
4cfed6e98e in MultiHeadAttention and ResidualAttentionBlock include cache_id for compatibility with simulstreaming code 2025-08-02 13:16:58 +02:00
Quentin Fuxa
687e3dd5e2 update simulstreaming model.py to match the latest version of whisper sources 2025-08-02 13:16:10 +02:00
Quentin Fuxa
8e056cbdf2 Upgrade SimulStreaming Whisper core from version 20230918 to 20250625 2025-08-02 13:06:36 +02:00
Quentin Fuxa
00424d7ca3 latest version of simulstreaming 2025-07-31 16:44:23 +02:00
Quentin Fuxa
4b738d6f63 fix duplicate line 2025-07-31 16:29:35 +02:00
Quentin Fuxa
8a5e2adb1e simulstreaming: fixes token handling during warm-up phase 2025-07-31 16:25:34 +02:00
Quentin Fuxa
46efbdf1d9 solves https://github.com/QuentinFuxa/WhisperLiveKit/issues/151 2025-07-31 11:42:06 +02:00
Quentin Fuxa
56114d3071 Remove end_attributed_speaker in diarization_online. handled in audio processor 2025-07-16 12:09:43 +02:00
Quentin Fuxa
5b9977c9af Enhanced use_punctuation_split for diarization. further improvements still needed 2025-07-16 12:06:17 +02:00
Quentin Fuxa
12a544164f Merge branch 'main' of https://github.com/QuentinFuxa/whisper_streaming_web 2025-07-16 12:05:01 +02:00
Quentin Fuxa
3ad3683ca7 Refactor speaker assignment in DiartDiarization for clarity and punctuation awareness 2025-07-15 14:38:53 +02:00
Quentin Fuxa
1599bd87a0 work on punctuation_split 2025-07-15 12:04:54 +02:00
Quentin Fuxa
90623400a4 Remove automatic downloading of SimulStreaming dependencies on import failure 2025-07-15 12:04:17 +02:00
choomegan
64e44fb24f fix: logic of adding of pcm_array to diarization_queue 2025-07-15 15:33:41 +08:00
Quentin Fuxa
9ff513093b simulstreaming uses empty space as separator 2025-07-04 17:03:01 +02:00
Quentin Fuxa
aad2c55d8c download_simulstreaming_backend.py now downloads files in the correct lib dir 2025-07-03 11:07:28 +02:00
Quentin Fuxa
2f177c4a3b add __init__.py file to simul_whisper assets directory 2025-07-03 10:41:12 +02:00
Quentin Fuxa
b362eccb23 new command to get simulstreaming backend 2025-07-03 10:24:02 +02:00
Quentin Fuxa
5daaf77258 add download script for SimulStreaming backend 2025-07-03 10:14:45 +02:00
Quentin Fuxa
36cc4412c3 update LICENSE with SimulStreaming dual licensing terms; include in .gitignore additional stuff 2025-07-03 09:21:38 +02:00
Quentin Fuxa
e1d4bf7e94 modify import paths in simul whisper backend so that it works in lib mode 2025-07-01 20:34:47 +02:00
Quentin Fuxa
62bf28949e compatible with the latest version of simulstreaming 2025-07-01 20:10:45 +02:00
Quentin Fuxa
1e3fab9550 copy non python files from simulstreaming when installing package 2025-07-01 19:14:23 +02:00
Quentin Fuxa
f25de6d8a4 ffmpeg-python is not used anymore - ffmpeg is directly called through create_subprocess_exec 2025-07-01 18:53:35 +02:00
Quentin Fuxa
8a175e79d8 Merge branch 'main' of https://github.com/QuentinFuxa/whisper_streaming_web 2025-07-01 18:52:26 +02:00
Quentin Fuxa
dc37b44486 add _read_stderr to empty the stderr 2025-07-01 17:05:58 +02:00
Quentin Fuxa
2d1df92aa7
Merge pull request #145 from SlavikCA/port-fix 
fix port for WS link; use correct HF build arg
 2025-07-01 14:16:58 +02:00
Quentin Fuxa
2c1a603e38 ffmpeg is managed in a thread in FFmpegManager to prevent the all from crashing when an error occurs 2025-07-01 11:19:10 +02:00
Quentin Fuxa
774cee036b increase timeout from 2 to 20s for ffmpeg stdin flush and writing 2025-06-30 18:28:50 +02:00
Quentin Fuxa
d22916988e add SIMULSTREAMING_ERROR_AND_INSTALLATION_INSTRUCTIONS for instructions when simulstreaming files are not there 2025-06-30 17:42:45 +02:00
slavik.fursov
5b8ad94dde fix port for WS link; use correct HF build arg 2025-06-30 08:15:51 -07:00
Quentin Fuxa
f668570292 Trim buffer when no new ASR tokens are issued 2025-06-30 11:55:07 +02:00
Quentin Fuxa
7c0768e8f3 bump version to 0.2.1; enhance error message for simulstreaming missing dependencies 2025-06-27 14:06:35 +02:00
Quentin Fuxa
b42d8b2692 add dual license warning indication when using simulstreaming backend 2025-06-27 10:00:19 +02:00
Quentin Fuxa
8e30e8010a correct timing (lag) calculations in SimulStreamingASR and SimulStreamingOnlineProcessor 2025-06-26 00:13:44 +02:00
Quentin Fuxa
bfec335a5f restore a functionnal buffer_diarization 2025-06-25 23:38:23 +02:00
Quentin Fuxa
6867041254 1rst version of SimulStreaming backend. many improvements needed 2025-06-25 17:59:46 +02:00
Quentin Fuxa
8532a91c7a add segmentation and embedding model options to configuration 2025-06-19 16:29:25 +02:00
Quentin Fuxa
b01b81bad0 improve diarization with lag diarization substraction 2025-06-19 16:18:49 +02:00
Quentin Fuxa
0f79d442ee improve diarization speed + Use punctuation to better align speakers and diarization 2025-06-19 13:03:29 +02:00
Quentin Fuxa
993a83546a core refactoring 2025-06-16 16:13:57 +02:00
Quentin Fuxa
f7644268c1 Message when launching transcription and no audio is detected 2025-05-28 13:25:49 +02:00
Quentin Fuxa
34e8fe260e lag information in real time even when no audio is detected 2025-05-28 12:25:47 +02:00
Quentin Fuxa
6797b88176 Error handling for missing FFmpeg in start_ffmpeg_decoder 2025-05-28 11:43:30 +02:00
Quentin Fuxa
46770efd6c correct error when using VAC 2025-05-28 11:43:18 +02:00
Quentin Fuxa
fea3c3553c logging in ASR proc. includes internal buffer duration and transcription lag 2025-05-07 11:45:00 +02:00
Quentin Fuxa
d6d65a663b errors handling when end of transcription 2025-05-07 10:56:04 +02:00
Quentin Fuxa
083d5b2f44 uses sentinel object when end of transcription, to properly terminate tasks 2025-05-07 10:55:44 +02:00
Quentin Fuxa
8e4674b093 End of transcription : Properly sends signal back to the endpoint 2025-05-07 10:55:12 +02:00
Quentin Fuxa
b56fcffde1 Solves stdin flushes blocking IOhttps://github.com/QuentinFuxa/WhisperLiveKit/issues/110 
https://github.com/QuentinFuxa/WhisperLiveKit/issues/106
https://github.com/QuentinFuxa/WhisperLiveKit/issues/90
https://github.com/QuentinFuxa/WhisperLiveKit/issues/87
https://github.com/QuentinFuxa/WhisperLiveKit/issues/81
https://github.com/QuentinFuxa/WhisperLiveKit/issues/2
 2025-04-12 15:25:46 +02:00
Quentin Fuxa
2def194893 add ssl certificate and key file arguments to parser 2025-04-11 12:20:22 +02:00
Quentin Fuxa
29978da301 adds ssl possibility in basic server 2025-04-11 12:20:08 +02:00
Quentin Fuxa
b708890788 protocol default to ws 2025-04-11 12:14:14 +02:00
Quentin Fuxa
3ac4c514cf remove temp_kit method to get args. uvicorn reload to False for better perfs 2025-04-11 12:02:52 +02:00
Chris Margach
d53b7a323a update sample html to use wss in case of https 2025-04-10 13:46:52 +09:00
Chris Margach
02de5993e6 allow passing of cert and key locations to uvicorn via package 2025-04-10 13:42:30 +09:00
Quentin Fuxa
d94560ef37 Merge branch 'main' of https://github.com/QuentinFuxa/whisper_streaming_web 2025-04-09 11:38:57 +02:00
Quentin Fuxa
0b43035701 enhance chunking to handle audio buffer time limits 2025-04-09 11:34:59 +02:00
Quentin Fuxa
704170ccf3 Logs for https://github.com/QuentinFuxa/WhisperLiveKit/issues/110 https://github.com/QuentinFuxa/WhisperLiveKit/issues/106 
https://github.com/QuentinFuxa/WhisperLiveKit/issues/90
https://github.com/QuentinFuxa/WhisperLiveKit/issues/87
https://github.com/QuentinFuxa/WhisperLiveKit/issues/81
https://github.com/QuentinFuxa/WhisperLiveKit/issues/2
 2025-04-09 11:34:27 +02:00
Chris Margach
51e6a6f6f9 update import after moving target file 2025-04-09 13:33:39 +09:00