33

A distributed, content-addressed filesystem across the internet, not just in a home LAN, than can be mounted (fuse or whatever else), doesn't require storing data twice, has a reasonable API and acceptable documentation.

Does something like that exist?

you are viewing a single comment's thread
view the rest of the comments
[-] leanleft@lemmy.ml 1 points 1 month ago* (last edited 1 month ago)

256 KB is the chosen default. so this is the best choice. however.. i wonder..
if you share all 3900 finetune versions of llama 3.1 8b or all 2900 finetune versions of qwen 2.5 7b instruct

there is likely alot of matching chunks if you break the chunk size super small.

so it may not be the most optimal chunk size for sharing.. but you could (potentially have waay more peers (because peeps are all hosting diff models with some shared matches between them)

** a more difficult question is wether to create ,efficient, highly compressed archive of specific popular sets of related models("most-popular, starter packs"). this requires communication,coordination,effort to get it EXACTLY right.
IMO this is a trade for using centralized trackers and random uploaders.

this post was submitted on 03 May 2025
33 points (92.3% liked)

Programming

28322 readers
637 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev



founded 3 years ago
MODERATORS