This is an automated archive made by the Lemmit Bot.
The original was posted on /r/stablediffusion by /u/fpgaminer on 2024-09-21 18:37:01+00:00.
This is an update and follow-up to my previous post (). To recap, JoyCaption is being built from the ground up as a free, open, and uncensored captioning VLM model for the community to use in training Diffusion models.
- Free and Open: It will be released for free, open weights, no restrictions, and just like bigASP, will come with training scripts and lots of juicy details on how it gets built.
- Uncensored: Equal coverage of SFW and NSFW concepts. No "cylindrical shaped object with a white substance coming out on it" here.
- Diversity: All are welcome here. Do you like digital art? Photoreal? Anime? Furry? JoyCaption is for everyone. Pains are being taken to ensure broad coverage of image styles, content, ethnicity, gender, orientation, etc.
- Minimal filtering: JoyCaption is trained on large swathes of images so that it can understand almost all aspects of our world. almost. Illegal content will never be tolerated in JoyCaption's training.
The Demo
WARNING ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ ⚠️ This is a preview release, a demo, alpha, highly unstable, not ready for production use, not indicative of the final product, may irradiate your cat, etc.
JoyCaption is still under development, but I like to release early and often to garner feedback, suggestions, and involvement from the community. So, here you go!
What's New
Wow, it's almost been two months since the Pre-Alpha! The comments and feedback from the community have been invaluable, and I've spent the time since then working to improve JoyCaption and bring it closer to my vision for version one.
- First and foremost, based on feedback, I expanded the dataset in various directions to hopefully improve: anime/video game character recognition, classic art, movie names, artist names, watermark detection, male nsfw understanding, and more.
- Second, and perhaps most importantly, you can now control the length of captions JoyCaption generates! You'll find in the demo above that you can ask for a number of words (20 to 260 words), a rough length (very short to very long), or "Any" which gives JoyCaption free reign.
- Third, you can now control whether JoyCaption writes in the same style as the Pre-Alpha release, which is very formal and clincal, or a new "informal" style, which will use such vulgar and non-Victorian words as "dong" and "chick".
- Fourth, there are new "Caption Types" to choose from. "Descriptive" is just like the pre-alpha, purely natural language captions. "Training Prompt" will write random mixtures of natural language, sentence fragments, and booru tags, to try and mimic how users typically write Stable Diffusion prompts. It's highly experimental and unstable; use with caution. "rng-tags" writes only booru tags. It doesn't work very well; I don't recommend it. (NOTE: "Caption Tone" only affects "Descriptive" captions.)
The Details
It has been a grueling month. I spent the majority of the time manually writing 2,000 Training Prompt captions from scratch to try and get that mode working. Unfortunately, I failed miserably. JoyCaption Pre-Alpha was turning out to be quite difficult to fine-tune for the new modes, so I decided to start back at the beginning and massively rework its base training data to hopefully make it more flexible and general. "rng-tags" mode was added to help it learn booru tags better. Half of the existing captions were re-worded into "informal" style to help the model learn new vocabulary. 200k brand new captions were added with varying lengths to help it learn how to write more tersely. And I added a LORA on the LLM module to help it adapt.
The upshot of all that work is the new Caption Length and Caption Tone controls, which I hope will make JoyCaption more useful. The downside is that none of that really helped Training Prompt mode function better. The issue is that, in that mode, it will often go haywire and spiral into a repeating loop. So while it kinda works, it's too unstable to be useful in practice. 2k captions is also quite small and so Training Prompt mode has picked up on some idiosyncrasies in the training data.
That said, I'm quite happy with the new length conditioning controls on Descriptive captions. They help a lot with reducing the verbosity of the captions. And for training Stable Diffusion models, you can randomly sample from the different caption lengths to help ensure that the model doesn't overfit to a particular caption length.
Caveats
As stated, Training Prompt mode is still not working very well, so use with caution. rng-tags mode is mostly just there to help expand the model's understanding, I wouldn't recommend actually using it.
Informal style is ... interesting. For training Stable Diffusion models, I think it'll be helpful because it greatly expands the vocabulary used in the captions. But I'm not terribly happy with the particular style it writes in. It very much sounds like a boomer trying to be hip. Also, the informal style was made by having a strong LLM rephrase half of the existing captions in the dataset; they were not built directly from the images they are associated with. That means that the informal style captions tend to be slightly less accurate than the formal style captions.
And the usual caveats from before. I think the dataset expansion did improve some things slightly like movie, art, and character recognition. OCR is still meh, especially on difficult to read stuff like artist signatures. And artist recognition is ... quite bad at the moment. I'm going to have to pour more classical art into the model to improve that. It should be better at calling out male NSFW details (erect/flaccid, circumcised/uncircumcised), but accuracy needs more improvement there.
Feedback
Please let me know what you think of the new features, if the model is performing better for you, or if it's performing worse. Feedback, like before, is always welcome and crucial to me improving JoyCaption for everyone to use.