152

OpenSubs: I built a subtitle generator that runs Whisper in the browser, so the video never gets uploaded (opensubs.app)

submitted 1 day ago by opensubs@lemmy.world to c/opensource@lemmy.ml

41 comments fedilink hide all child comments

I work on OpenSubs, a free, open source (AGPL-3.0) subtitle tool that runs entirely in the browser tab.

You drop in a video file, and Whisper transcribes it on your own machine, using transformers.js with WebGPU where available and WebAssembly otherwise. The model (40–250 MB) downloads once and is cached. There is no upload endpoint in the product, so the video has nowhere to go.

After that you can:

fix lines by typing over them (click a timestamp to jump to that moment)
translate into 20 languages with Chrome's built-in on-device translator
pick one of 12 caption styles, including word-by-word highlighting
export SRT / VTT / ASS, or burn the subtitles into an MP4 (libass compiled to WebAssembly, encoded with WebCodecs)

A few things I learned building it:

Whisper hallucinates on silence and music ("Thanks for watching!", or the Japanese equivalent). A Silero VAD pass runs before Whisper, and a cleanup step drops the known stock phrases.
Singing doesn't count as speech for the VAD, so a music video gets a "no speech found" warning. You can still force it.

Honest limits: it only takes video files, not audio-only files. Cue timings can't be edited yet. Builds are release candidates. Everything that runs locally is free with no account; the only paid part is optional cloud translation on our backend (US$5 for 1000 credits), and you can bring your own Claude / OpenAI / DeepL key instead.

Site: https://opensubs.app/ Code: https://github.com/open-subs/opensubs

Feedback welcome, especially on languages where the transcription goes wrong.

top 41 comments

sorted by: hot top new old

[-] antianarchist@sopuli.xyz 9 points 8 hours ago

Nice, another blindly generated app by Claude Code. 37 commits and already announcing it tells it all.

[-] mysterious_cake@feddit.nl 3 points 12 hours ago* (last edited 6 hours ago)

Did whisper improve since last year?

Last time I tried to generate subtitles for a German TV show, the subtitles were more inacurrate than what I would have written down (German is my third language and I struggle to understand anything that isn’t voiceovered) and timing was so bad I didn’t even bother to generate subtitles for the second episode.

[-] geneva_convenience@lemmy.ml 10 points 10 hours ago* (last edited 10 hours ago)

No but other models have. parakeet-unified is currently the best small model.

You can tell OP vibe coded the whole thing from their outdated model choice.

[-] victorz@lemmy.world 6 points 10 hours ago

You can tell OP vibe coded the whole thing from their outdated model choice.

I hate this landscape we're in now. I'm always suspicious of whether something was created by a thieving computer program or a real human. Just like I'm always suspicious of crazy videos these days being AI or not. Neither can I watch an ad anymore without suspicion. It's so much extra mental overhead.

I'm tired, boss.

[-] geneva_convenience@lemmy.ml 6 points 10 hours ago* (last edited 10 hours ago)

You can also just click the repository and see that Claude is a contributor which usually gives it away.

My point is more so that OP didn't even put in the effort to ask claude to look up the current best TTS model to use but instead one-shotted the thing

[-] victorz@lemmy.world 2 points 8 hours ago

Valid point.

My point was more than you didn't even have to check who is a contributor to make sure they were all human. I don't like this future. ☹️

[-] myszka@lemmy.ml 3 points 11 hours ago

I've used Whisper on my private peertube instance recently and the results were pretty decent

[-] zurohki@aussie.zone 27 points 21 hours ago

Whisper hallucinates on silence and music (“Thanks for watching!”, or the Japanese equivalent).

I've had Whisper ask me to donate to anime fansubbing groups. I wonder what it was trained on?

[-] Buckshot@programming.dev 5 points 12 hours ago

Sometimes if I hit the voice typing on my phone, which I'm pretty sure is whisper, and then don't say anything, it just comes out with nonsense.

Just did this and also got...

Thanks for watching!

[-] Kalothar@lemmy.ca 1 points 13 hours ago

What’s the max file size?

I’ve been looking for the fan edit of marvels Infinity saga.

It’s about 50 hours long divided into many many files ranging from 3-10 gb each. Would it be able to handle this? Or what’s the best way to split them up

[-] danielquinn@lemmy.ca 9 points 22 hours ago

It sounds like a pretty cool project, thanks for sharing! As a browser-based project designed to run offline, you might want to consider shipping it as an electron app.

[-] gary_host_laptop@lemmy.ml 2 points 15 hours ago

tauri is better and more efficient for this.

[-] magnue@lemmy.world 1 points 15 hours ago

I'm a lazy reader so don't know if this is implemented but it would be cool if it could be done live with the current audio out of the system. I know this was done for streamers etc using OBS/localvocal but it's a pain to get working in my experience - having to fiddle around with OBS filter settings.

[-] JustEnoughDucks@slrpnk.net 3 points 22 hours ago

Why not just use the whisper plugin for Bazarr?

[-] gary_host_laptop@lemmy.ml 3 points 15 hours ago

i get ya, but also a lot of people don't have a dedicated self hosted setup. recently my hdd died and all my films were there, so now i'm downloading and deleting, which just feels like a drag to start bazarr for one film. this is handy for some people.

[+] Sammirr@aussie.zone -7 points 1 day ago

Open source, but made with a copyright infringing LLM.

[-] gary_host_laptop@lemmy.ml 5 points 15 hours ago

i love how anti ai people have become the wardens of copyright somehow.

[-] Sammirr@aussie.zone 2 points 7 hours ago

Statement of fact. Claude was engineered with copyright material.

[-] gary_host_laptop@lemmy.ml 1 points 4 hours ago

whats wrong is not copyright theft. whats wrong is labour that was stolen.

[-] Sammirr@aussie.zone 1 points 1 hour ago* (last edited 1 hour ago)

Yes and no. As an author of open source projects, they were made available for people to fork, improve and conrribute under permissive licenses. Then come LLMs that uses code like mine to generate code or slop. My choice about license was made before LLMs. Now they use derivatives of my work without attribution as required by the license.

Even if I change the license, it won't be respected, and nothing else will change. To me, that's copyright infringement on me, ignoring the hours I put in.

On a separate note, LLMs only produce more of the same, ignoring the environmental and economic impacts. Often, more of the same is not what we need. It leads to the same mistakes being made. I've never used LLM to code, never will.

[-] AlmightyDoorman@kbin.earth 17 points 1 day ago

Whisper is not an llm, just because it is trained on big datasets does not make it an LLM. Copyright infringement still happened during the training.

[-] lime@feddit.nu 16 points 23 hours ago

no, but claude is, and it's the main contributor to the repo.

[-] thingsiplay@lemmy.ml 4 points 23 hours ago

Which software are we talking about? Unfortunately no one links to it, so I will guess. Whisper: website | source is an Ai model, specifically powered by OpenAi.

[-] TheTechnician27@lemmy.world -1 points 23 hours ago* (last edited 23 hours ago)

Whisper is not an llm, just because it is trained on big datasets does not make it an LLM.

The training dataset consists of 680,000 hours of labeled audio-transcript pairs sourced from the internet using semi-supervised learning.

Brother, I don't think you know what a large language model is if you think it being trained on a human lifespan's worth of audio transcripts isn't enough.

Its job is NLP. It is an LLM, if not as general-use as something like ChatGPT.

[-] VonReposti@feddit.dk 9 points 19 hours ago

LLM is a subset of NLP. All LLMs are NLP but not all NLP are LLMs. To be more precise, LLM is a method of achieving NLP while NLP is the goal. Whisper is just a specialised speech-to-text engine. It doesn't have the text reasoning engine that is the core of LLMs. Yes, it has a dataset to inform it's encoder and decoder, but that doesn't make it an LLM. If you strip out the encoder, then you're getting closer.

[-] TheTechnician27@lemmy.world 0 points 12 hours ago* (last edited 12 hours ago)

If you strip out the encoder, then you're getting closer.

That's flat-out not true. BERTs, for example, are LLMs and are encoder-only, just like GPTs are decoder-only. Whisper is trained on a large corpus of text and does NLP tasks. That's all it is to be an LLM.

A transformer model being encoder-only, encoder-decoder, or decoder-only has fuck-all to do with whether it's an LLM. In fact, it being a transformer model at all isn't required for it to be an LLM.

[-] VonReposti@feddit.dk 1 points 8 hours ago

BERT is a special case. The academic consensus is rightly considering it an LLM since it operates on text but the industry doesn't consider it an LLM since LLM in its colloquial meaning has drifted to mean a model generating text (which BERT doesn't). Whisper is not an LLM since it's working on audio, so it is an Automatic Speech Recognition model (ASR).

Text in = LLM in academic circles (BERT and GPT)

Text in, text out = LLM in all circles (GPT only)

Audio in, text out ≠ LLM (Whisper)

Now, we can argue whether we want to accept the colloquial meaning of LLM, but the fact is that Whisper is not an LLM. And neither ASR nor BERT causes even a fraction of the damage GPT is doing, but that's another discussion.

[+] andycallisto@social.noleron.com -7 points 1 day ago

@opensubs

In the browser

Never gets uploaded

What are you talking about?

[-] virku@lemmy.world 17 points 1 day ago

The browser runs a lot of stuff that doesn't get passed on to the servers you connect to. In this case it can run a local whisper model locally and pass your video stream through it, rather than pass it through some server online.

[-] andycallisto@social.noleron.com -3 points 1 day ago

@virku Then what's the point? You can just download the model and run it locally without using this service...

[-] magnue@lemmy.world 2 points 15 hours ago

This is how loads of things run these days. Browser handles the UI because it's way easier to get working on all kinds of different user machines. I don't like it personally but that's why.

It's how pretty much any server dashboard runs.

[-] Ferk@lemmy.ml 4 points 19 hours ago* (last edited 19 hours ago)

It's not a "service" but a program that runs locally.

Downloading the model and running it locally is essentially what it does. If you do it by yourself then you'd have to write your own scripts. At which point, you might as well share those scripts with the community putting them in Github, and maybe including them in a web extension to integrate with the browser and/or in a static website that locally runs it via WASM... precisely what this project is doing...

[-] virku@lemmy.world 6 points 21 hours ago

Ease of use..

[-] beyoublahaj@lemmy.world 1 points 1 day ago* (last edited 1 day ago)

If i understood this right...

Instead of uploading a file you just download the model. The sound then gets transcribed as usual without the service accessing the file.

[-] andycallisto@social.noleron.com -2 points 1 day ago

@beyoublahaj So audio gets uploaded?

[-] beyoublahaj@lemmy.world 1 points 1 day ago

oh nah I reread the post and edited my comment. I misunderstood too ^^`

[-] andycallisto@social.noleron.com 1 points 1 day ago

@beyoublahaj So what's the point of this service if you can download the model yourself and use it without this website?..

[-] magnue@lemmy.world 2 points 15 hours ago

As people have tried to explain - it's not a website. The UI for the software simply runs via the browser.

[-] Ferk@lemmy.ml 1 points 20 hours ago* (last edited 20 hours ago)

You don't have to use the website.. the project is not just that website, it's also packaged as a browser extension, a desktop app and a CLI program.

That website is just a convenient way to see it working (while still processing it locally) without installing anything.

If you download the raw model and write up scripts to create subtitles out of video by yourself then you are essentially re-building the same app.. the result would be similar, just duplicating effort.

[-] thingsiplay@lemmy.ml 1 points 23 hours ago

I assume you can use it on any platform that runs a browser, without installing software on the system. I use (other type of) software that runs in the browser, but locally. This type of browser software is handy.