i would put the small distributed files behind an API and set a cache on the client?
if you don't want to do that due to wanting local offline then maybe separate the databases as plugins and download/subscribe based on if users enable them, similar to how CoMaps works.
There are a couple of features in the BitTorrent specs for updating torrents, like embedding a signed RSS URL in the torrent file itself, or using mutable DHT entries, but I’ve never tried them out and I couldn’t tell you which clients support those features, and you’d have to manage signing keys since those features are completely decentralized and use public and private keys to ensure authenticity. Definitely worth looking into, but I apologize I don’t have anything concrete to link for you, I’m just aware of the features in the spec.
I think most clients also support pulling torrents from plain RSS feeds containing links to torrent files. Some sites like showrss do that.
Let’s assume I have a software application with millions of files.
rsync is generally good at things like this.
I would just be creating records within the main database.
So is it one file? Or millions?
if there's not so many devices just rsync or something similar from one central server
if there's a large number of devices i'd try torrent
Bless your heart op.
Thank you 🙏️
Please take this as support: You need to look a little bit more into theories. You are reinventing things that don't need reinventing (categorizing data), but are not thinking about the real pain points of building search that return relevant results (full text search performance, scalability, semantics, extracting user intention from queries).
Here's an article that I think is very good on the topic
That's because I don't want that.
I remember, as a kid, wandering the local library. Looking at all the books. Seeing the call numbers and author names. Exploring the shelves, as they transitioned from topic to topic to topic.
My idea is to treat the Internet as a giant library, with domains being books on a shelf, and allowing my users to start at a random location, and then peruse the different listings as they transition from slight variations of the main topic.
With all due respect, you don't have the vision, man. To start looking at cultural anthropology, then have a link to a biological anthropology blog 3 lines down? That's the goal. It's not about a narrowed search. It's about cataloging something, so people can go looking for it. To enjoy the hunt.
rsync:// server
All receivers (I call client) must get all the files from the source (I call server) ?
Then, does the clients are known ? (Example : you can access, or you can left a tutorial and they can install a new software)
If yes for all files :
Git could work, but you have to host the server, cause no free online website do allow big content.
If no for all files : Then a solution could be to share with https://zeronet.io/
This post is not done writing, i'll complete later.
Maybe syncthing?
maybe caldav/nextcloud
webdav*
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Follow the wormhole through a path of communities !webdev@programming.dev