Quentin Fuxa
714fb3b14a
custom faster-whisper/mlx whisper encoder available
2025-10-23 20:33:17 +02:00
Quentin Fuxa
0af379c465
DOC: information about file format
2025-10-23 20:32:05 +02:00
Quentin Fuxa
1f684cdd97
fixes #251
2025-10-06 19:53:27 +02:00
Quentin Fuxa
9b1e061b32
forwarded_allow_ips in core
2025-10-04 23:04:00 +02:00
Quentin Fuxa
b4abc158b9
Merge pull request #249 from Damrod/add-ip-forwarding-support
...
fix wss for reverse proxying
2025-10-06 10:20:05 +02:00
Alvaro Ollero
3736458503
Uvicorn exposes a configuration option to enable reverse proxying from a trusted ip. This PR exposes it downstreams to end clients
2025-10-04 22:21:06 +02:00
Quentin Fuxa
374618e050
token speakers are only reattributed for token coming after last_validated_token
2025-10-04 09:52:00 +02:00
Quentin Fuxa
543972ef38
fixes #248
2025-10-04 09:52:00 +02:00
Quentin Fuxa
a7db39d999
solves incorrect spacing in buffer diarization
2025-10-02 23:04:00 +02:00
Quentin Fuxa
a153e11fe0
update when self.diarization_before_transcription
2025-09-28 11:04:00 +02:00
Quentin Fuxa
ca6f9246cc
force language = en for .en models
2025-09-28 11:04:00 +02:00
Quentin Fuxa
d080d675a8
cutom alignment heads parameter for custom models
2025-09-27 11:04:00 +02:00
Quentin Fuxa
40bff38933
Merge pull request #239 from msghik/feature/fine-tuned-model-support
...
feat: Allow loading fine-tuned models in simulstreaming
2025-09-29 10:08:26 +02:00
Quentin Fuxa
2fe3ca0188
connect source to output destination when used as chrome extension to keep audio playing
2025-09-27 13:59:44 +02:00
Quentin Fuxa
545ea15c9a
ensure buffer size to be a multiple of the element size
2025-09-27 13:58:32 +02:00
Quentin Fuxa
8cbaeecc75
cutom alignment heads parameter for custom models
2025-09-27 11:04:00 +02:00
google-labs-jules[bot]
70e854b346
feat: Allow loading fine-tuned models in simulstreaming
...
This change modifies the `simulstreaming` backend to support loading fine-tuned Whisper models via the `--model_dir` argument.
The `SimulStreamingASR` class has been updated to:
- Use the `model_dir` path directly to load the model, which is the correct procedure for fine-tuned `.pt` files.
- Automatically disable the `faster-whisper` and `mlx-whisper` fast encoders when `model_dir` is used, as they are not compatible with standard fine-tuned models.
The call site in `core.py` already passed the `model_dir` argument, so no changes were needed there. This change makes the `simulstreaming` backend more flexible and allows users to leverage their own custom models.
2025-09-27 07:29:30 +00:00
Quentin Fuxa
d55490cd27
typo and simpler conditions
2025-09-26 20:38:26 +02:00
Quentin Fuxa
b22478c0b4
correct silences handling when language not auto
2025-09-25 23:20:00 +02:00
Quentin Fuxa
94c34efd90
chrome extension ws default to localhost
2025-09-25 23:04:00 +02:00
Quentin Fuxa
9fc6654a4a
common frontend for web/ and chrome extension
2025-09-25 23:14:25 +02:00
Quentin Fuxa
4dd5d8bf8a
translation compatible with auto and detected language
2025-09-22 11:20:00 +02:00
Quentin Fuxa
6caf3e0485
correct silence handling in translation
2025-09-27 11:58:00 +02:00
Quentin Fuxa
93f002cafb
language detection after few seconds working
2025-09-20 11:08:00 +02:00
Quentin Fuxa
c5e30c2c07
svg loaded once in javascript, no more need for StaticFiles
2025-09-20 11:06:00 +02:00
Quentin Fuxa
1c2afb8bd2
svg loaded once in javascript, no more need for StaticFiles
2025-09-20 11:06:00 +02:00
Quentin Fuxa
674b20d3af
in buffer while language not detected »
2025-09-21 11:05:00 +02:00
Quentin Fuxa
a5503308c5
O(n) to O(1) for simulstreaming timestamp determination
2025-09-21 11:04:00 +02:00
Quentin Fuxa
e61afdefa3
punctuation is now checked in timed_object
2025-09-22 22:40:39 +02:00
Quentin Fuxa
426d70a790
simulstreaming infer does not return a dictionary anymore
2025-09-21 11:03:00 +02:00
Quentin Fuxa
b03a212fbf
fixes #227 , auto language dectection v0.1 - simulstreaming only - when diarization and auto
2025-09-19 19:15:28 +02:00
Quentin Fuxa
1833e7c921
0.2.10
2025-09-16 23:45:00 +02:00
Quentin Fuxa
0a6e5ae9c1
ffmpeg install instruction error indicates --pcm-input alternative
2025-09-17 16:04:17 +02:00
Quentin Fuxa
ee448a37e9
when pcm-input is set, the frontend uses AudioWorklet
2025-09-17 14:55:57 +02:00
Quentin Fuxa
9c051052b0
Merge branch 'main' into ScriptProcessorNode-to-AudioWorklet
2025-09-17 11:28:36 +02:00
Quentin Fuxa
4d7c487614
replace deprecated ScriptProcessorNode with AudioWorklet
2025-09-17 10:53:53 +02:00
Quentin Fuxa
65025cc448
nllb backend can be transformers, and model size can be 1.3B
2025-09-17 10:20:31 +02:00
Quentin Fuxa
bbba1d9bb7
add nllb-backend and translation perf test in dev_notes
2025-09-16 20:45:01 +02:00
Quentin Fuxa
99dc96c644
fixes #224
2025-09-16 18:34:35 +02:00
GeorgeCaoJ
2a27d2030a
feat: support web audio 16kHz PCM input and remove ffmpeg dependency
2025-09-15 23:22:25 +08:00
Quentin Fuxa
cd160caaa1
asyncio.to_thread for transcription and translation
2025-09-15 15:23:22 +02:00
Quentin Fuxa
5aa312e437
simulstreaming warmup is done in whisperlivekit.simul_whisper.backend.load_model, not in warmup_online
2025-09-13 20:19:19 +01:00
notV3NOM
ebaf36a8be
Fix warmup file behavior
2025-09-13 20:44:24 +05:30
Quentin Fuxa
a4e9f3cab7
support for raw PCM input option by @YeonjunNotFR
2025-09-11 21:32:11 +02:00
Quentin Fuxa
b06866877a
add --disable-punctuation-split option
2025-09-11 21:03:00 +02:00
Quentin Fuxa
967cdfebc8
fix Translation imports
2025-09-11 21:03:00 +02:00
Quentin Fuxa
3c11c60126
fix by @treeaaa
2025-09-11 21:03:00 +02:00
Quentin Fuxa
2963e8a757
translate when at least 3 new tokens
2025-09-09 21:45:00 +02:00
Quentin Fuxa
cb2d4ea88a
audio processor lines use now Lines objects instead of dict
2025-09-09 21:45:00 +02:00
Quentin Fuxa
add7ea07ee
translator takes all the tokens from the queue
2025-09-09 19:55:39 +02:00
Quentin Fuxa
3358877054
Fix StorageView conversion for CPU/GPU compatibility
2025-09-09 15:44:16 +02:00
Quentin Fuxa
1f7798c7c1
condition on encoder_feature_ctranslate type
2025-09-09 12:16:52 +02:00
Alexander Lindberg
c7b3bb5e58
Fix regression with faster-whisper encoder_feature
2025-09-09 11:18:55 +03:00
Quentin Fuxa
f661f21675
translation asyncio task
2025-09-08 18:34:31 +02:00
Quentin Fuxa
b6164aa59b
translation device determined with torch.device
2025-09-08 11:34:40 +02:00
Quentin Fuxa
4209d7f7c0
Place all tensors on the same device in sortformer diarization
2025-09-08 10:20:57 +02:00
Quentin Fuxa
334b338ab0
use platform to determine system and recommand mlx whisper
2025-09-07 15:49:11 +02:00
Quentin Fuxa
72f33be6f2
translation: use of get_nllb_code
2025-09-07 15:25:14 +02:00
notV3NOM
a178ed5c22
fix simulstreaming preload model count argument in cli
2025-09-06 18:18:09 +05:30
Quentin Fuxa
fad9ee4d21
Merge pull request #198 from notV3NOM/main
...
Fix scrolling UX with sticky header controls
2025-09-05 20:46:36 +02:00
Quentin Fuxa
d1a9913c47
nllb v0
2025-09-05 18:02:42 +02:00
notV3NOM
e4ca2623cb
Fix scrolling UX with sticky header controls
2025-09-05 21:25:13 +05:30
Quentin Fuxa
9c1bf37960
fixes #197
2025-09-05 16:34:13 +02:00
notV3NOM
abd8f2c269
Fix exponentially growing simulstreaming silence timer
2025-09-04 21:49:07 +05:30
Quentin Fuxa
f3ad4e39e4
torch.Tensor to torch.as_tensor
2025-09-04 16:39:11 +02:00
Quentin Fuxa
e0a5cbf0e7
v0.1.0 chrome extension
2025-09-04 16:36:28 +02:00
Quentin Fuxa
953697cd86
torch.Tensor to torch.as_tensor
2025-09-04 15:25:39 +02:00
Quentin Fuxa
3bd2122eb4
0.2.8 : only the decoder of whisper is loaded in memory when a different encoder is used
2025-09-02 21:12:25 +02:00
Quentin Fuxa
d5008ed828
mlx/fasterWhisper encoders are loaded once and shared in simulstreaming
2025-09-01 12:33:19 +02:00
Quentin Fuxa
d467716e26
add microphone picker
2025-08-31 10:12:52 +02:00
Quentin Fuxa
199e21b3ef
faster-whisper as an optional encoder alternative for simulstreaming
2025-08-30 23:50:16 +02:00
Quentin Fuxa
1d926f2e67
mlx-whisper used as simulstreaming encoder: improve speed for macos systems
2025-08-30 22:19:11 +02:00
Quentin Fuxa
4a71a391b8
get_web_interface_html to get_inline_ui_html for embedded web interface HTML
2025-08-30 13:44:06 +02:00
Quentin Fuxa
1ba171a58d
add embedded web interface HTML (single-file version with inline CSS/JS/SVG)
...
### Added
- `get_inline_ui_html()`: generates a self-contained version of the web interface, with CSS, JS, and SVG assets inlined directly into the HTML. useful for environments where serving static files is inconvenient or when a single-call UI delivery is preferred.
(cherry picked from commit aa44a92a6775bcc42b3518e78acd16eda275935a)
2025-08-29 22:00:59 +02:00
Quentin Fuxa
4a5d5e1f3b
raise Exception when language == auto and task == translation
2025-08-29 17:44:46 +02:00
Quentin Fuxa
9895bc83bf
auto detection of language for warmup if not indicated
2025-08-27 20:37:48 +02:00
Quentin Fuxa
ab98c31f16
trim will happen before audio processor
2025-08-27 18:17:11 +02:00
Quentin Fuxa
4ed62e181d
when silences are detected, speaker correction is no more applied
2024-08-24 19:24:00 +02:00
Quentin Fuxa
9a8d3cbd90
improve diarization + silence handling
2024-08-24 19:20:00 +02:00
Quentin Fuxa
b101ce06bd
several users share the same sortformer model instance
2024-08-24 19:18:00 +02:00
Quentin Fuxa
c83fd179a8
improves phase shift correction between transcription and diarization
2024-08-24 19:15:00 +02:00
Quentin Fuxa
5258305745
default diarization backend in now sortformer
2025-08-24 18:32:01 +02:00
Quentin Fuxa
ce781831ee
punctuation is checked in audio-processor's result formatter
2025-08-24 18:32:01 +02:00
Quentin Fuxa
58297daf6d
sortformer diar implementation v0.3
2025-08-24 18:32:01 +02:00
Quentin Fuxa
3393a08f7e
sortformer diar implementation v0.2
2025-08-24 18:32:01 +02:00
Quentin Fuxa
26cc1072dd
new dockerfile for cpu only. update dockerfile from cuda 12.8 to 12.9
2025-08-22 11:04:35 +02:00
Quentin Fuxa
909ac9dd41
speaker -1 are no more sent in websocket - no buffer when their is a silence
2025-08-21 14:09:02 +02:00
Quentin Fuxa
d94a07d417
default model is now base. default backend simulstreaming
2025-08-21 11:55:36 +02:00
Quentin Fuxa
b32dd8bfc4
Align backend and frontend time handling
2025-08-21 10:33:15 +02:00
Quentin Fuxa
9feb0e597b
remove VACOnlineASRProcessor backend possibility
2025-08-20 20:57:43 +02:00
Quentin Fuxa
9dab84a573
update front
2025-08-20 20:15:38 +02:00
Quentin Fuxa
d089c7fce0
.html to .html + .css + .js
2025-08-20 20:00:31 +02:00
Quentin Fuxa
253a080df5
diart diarization handles pauses/silences thanks to offset
2025-08-19 21:12:55 +02:00
Quentin Fuxa
0c6e4b2aee
sortformer diar implementation v0.1
2025-08-19 19:48:51 +02:00
Quentin Fuxa
e14bbde77d
sortformer diar implementation v0
2025-08-19 17:02:55 +02:00
Quentin Fuxa
7496163467
rename diart backend
2025-08-19 15:02:27 +02:00
Quentin Fuxa
696a94d1ce
1rst sortformer backend implementation
2025-08-19 15:02:17 +02:00
Quentin Fuxa
2699b0974c
Fix simulstreaming imports
2025-08-19 14:43:54 +02:00
Quentin Fuxa
d0e9e37ef6
simulstreaming: cumulative_time_offset to keep timestamps correct when audio > 30s
2025-08-17 09:33:47 +02:00
Quentin Fuxa
820f92d8cb
audio_max_len to 30 -> 20, ffmpeg timeout 5 -> 20
2025-08-17 09:32:08 +02:00
Quentin Fuxa
e2184d5e06
better handle silences when VAC + correct offset issue with whisperstreaming backend
2025-08-17 01:27:07 +02:00
Quentin Fuxa
7fe0353260
vac model is loaded in TranscriptionEngine, and by default
2025-08-17 00:34:25 +02:00
Quentin Fuxa
0f2eba507e
use with_offset to add no audio offset to tokens
2025-08-17 00:33:24 +02:00
Quentin Fuxa
55e08474f3
recycle backend in simulstreaming thanks to new remove hooks function
2025-08-16 23:06:16 +02:00
Quentin Fuxa
28bdc52e1d
VAC before doing transcription and diarization. V0
2025-08-16 23:04:21 +02:00
Quentin Fuxa
1652db9a2d
Use distinct backend models for simulstreaming and add --preloaded_model_count to preload them
2025-08-15 23:03:55 +02:00
Quentin Fuxa
1c42b867cf
Merge branch 'main' of https://github.com/QuentinFuxa/whisper_streaming_web
2025-08-13 10:04:04 +02:00
Quentin Fuxa
d4771e563e
Increase END_SILENCE_DURATION to reduce false positives
2025-08-13 10:04:00 +02:00
David Gumberg
3b96fb8776
frontend: Scroll down when appending transcription
2025-08-12 17:31:32 -07:00
David Gumberg
7f93c4b978
frontend: Don't let screen sleep when transcribing.
2025-08-12 17:30:57 -07:00
Quentin Fuxa
15c3df1cba
warmup base whisper when using simulstreaming
2025-08-12 18:52:52 +02:00
Quentin Fuxa
728e1f1290
simulstreaming warmup is done for each instance of online, not for the backend
2025-08-12 18:35:04 +02:00
Quentin Fuxa
87b9ed6ecd
nonspeech_prob from 1 to 0.5
2025-08-12 18:34:37 +02:00
Quentin Fuxa
38b4ebe8ba
Handle 3 types of silences: Indicated by whisper, between tokens, and at the end of the input. Display them in the frontend
2025-08-11 17:56:57 +02:00
Quentin Fuxa
d098af3185
each SimulStreamingOnlineProcessor now contains PaddedAlignAttWhisper instance. SimulStreamingASR only contains loaded whisper model
2025-08-11 08:24:14 +02:00
Quentin Fuxa
4e56130a40
frontend supports dark theme
2025-08-11 08:22:23 +02:00
Quentin Fuxa
2bbdc70187
lags are now updated every 0.1s
2025-08-09 23:11:05 +02:00
Quentin Fuxa
b678a55f63
remove duplicate file
2025-08-09 23:10:34 +02:00
Quentin Fuxa
5491964e81
clean SimulStreamingOnlineProcessor initialization + audio processing
2025-08-09 20:16:27 +02:00
Quentin Fuxa
b05297a96d
clean simulwhisper backend and online
2025-08-09 18:02:15 +02:00
Quentin Fuxa
197293e25e
refactor(simulstreaming): extract backend + online module into separate files from whisper streaming
2025-08-08 18:07:51 +02:00
Quentin Fuxa
ba41c4ab56
Remove download_simulstreaming_backend
2025-08-08 18:06:40 +02:00
Quentin Fuxa
4cfed6e98e
in MultiHeadAttention and ResidualAttentionBlock include cache_id for compatibility with simulstreaming code
2025-08-02 13:16:58 +02:00
Quentin Fuxa
687e3dd5e2
update simulstreaming model.py to match the latest version of whisper sources
2025-08-02 13:16:10 +02:00
Quentin Fuxa
8e056cbdf2
Upgrade SimulStreaming Whisper core from version 20230918 to 20250625
2025-08-02 13:06:36 +02:00
Quentin Fuxa
00424d7ca3
latest version of simulstreaming
2025-07-31 16:44:23 +02:00
Quentin Fuxa
4b738d6f63
fix duplicate line
2025-07-31 16:29:35 +02:00
Quentin Fuxa
8a5e2adb1e
simulstreaming: fixes token handling during warm-up phase
2025-07-31 16:25:34 +02:00
Quentin Fuxa
46efbdf1d9
solves https://github.com/QuentinFuxa/WhisperLiveKit/issues/151
2025-07-31 11:42:06 +02:00
Quentin Fuxa
56114d3071
Remove end_attributed_speaker in diarization_online. handled in audio processor
2025-07-16 12:09:43 +02:00
Quentin Fuxa
5b9977c9af
Enhanced use_punctuation_split for diarization. further improvements still needed
2025-07-16 12:06:17 +02:00
Quentin Fuxa
12a544164f
Merge branch 'main' of https://github.com/QuentinFuxa/whisper_streaming_web
2025-07-16 12:05:01 +02:00
Quentin Fuxa
3ad3683ca7
Refactor speaker assignment in DiartDiarization for clarity and punctuation awareness
2025-07-15 14:38:53 +02:00
Quentin Fuxa
1599bd87a0
work on punctuation_split
2025-07-15 12:04:54 +02:00
Quentin Fuxa
90623400a4
Remove automatic downloading of SimulStreaming dependencies on import failure
2025-07-15 12:04:17 +02:00
choomegan
64e44fb24f
fix: logic of adding of pcm_array to diarization_queue
2025-07-15 15:33:41 +08:00
Quentin Fuxa
9ff513093b
simulstreaming uses empty space as separator
2025-07-04 17:03:01 +02:00
Quentin Fuxa
aad2c55d8c
download_simulstreaming_backend.py now downloads files in the correct lib dir
2025-07-03 11:07:28 +02:00
Quentin Fuxa
2f177c4a3b
add __init__.py file to simul_whisper assets directory
2025-07-03 10:41:12 +02:00
Quentin Fuxa
b362eccb23
new command to get simulstreaming backend
2025-07-03 10:24:02 +02:00
Quentin Fuxa
5daaf77258
add download script for SimulStreaming backend
2025-07-03 10:14:45 +02:00
Quentin Fuxa
36cc4412c3
update LICENSE with SimulStreaming dual licensing terms; include in .gitignore additional stuff
2025-07-03 09:21:38 +02:00
Quentin Fuxa
e1d4bf7e94
modify import paths in simul whisper backend so that it works in lib mode
2025-07-01 20:34:47 +02:00
Quentin Fuxa
62bf28949e
compatible with the latest version of simulstreaming
2025-07-01 20:10:45 +02:00
Quentin Fuxa
1e3fab9550
copy non python files from simulstreaming when installing package
2025-07-01 19:14:23 +02:00
Quentin Fuxa
f25de6d8a4
ffmpeg-python is not used anymore - ffmpeg is directly called through create_subprocess_exec
2025-07-01 18:53:35 +02:00
Quentin Fuxa
8a175e79d8
Merge branch 'main' of https://github.com/QuentinFuxa/whisper_streaming_web
2025-07-01 18:52:26 +02:00
Quentin Fuxa
dc37b44486
add _read_stderr to empty the stderr
2025-07-01 17:05:58 +02:00
Quentin Fuxa
2d1df92aa7
Merge pull request #145 from SlavikCA/port-fix
...
fix port for WS link; use correct HF build arg
2025-07-01 14:16:58 +02:00
Quentin Fuxa
2c1a603e38
ffmpeg is managed in a thread in FFmpegManager to prevent the all from crashing when an error occurs
2025-07-01 11:19:10 +02:00
