The new iroh-blobs design restores the initial "one unique identifier for each file" ethos of IPFS (abandonned by IPFS), to reduce swarm fragmentation. It takes the standard Blake3 hashing algorithm to generate file CIDs (equivalent to Bittorrent magnet infohash).

The protocol uses the Mainline Bittorrent DHT (not a custom DHT). There is no notion of block CIDs (no potential deduplication, but also less scalability issues for announcing each block CID on the DHT).

So that's a lot of design simplifications that will improve implementation time.

If you are interested, have a try at Iroh-share, a CLI and TUI Rust implementation.

publication croisée depuis : https://lemmy.dbzer0.com/post/67080379

Hello, since it's complicated to index DHTs, I figured it'd be more efficient to build an index of fingerprints from real data once.

So I've been collecting releases hashes for this index. It can be used for various purposes:

• check the integrity of your own files (bit rot is a real thing)
• identify BTv2 torrent files that contain specific files (a database of torrent files is required)
• locate alive IPFS swarms to join more easily (no need to read all your data multiple times to recompute various CIDs yourself)

The collection contains around 1K releases and weights 40MB. I've prioritized scene Bluray rips of movies (1080p / 2160p). No infohash will be included, as these are not reproducible enough.

I'm using a basic script to add a new release (filename must match the official release name). I'm using others to discover scene releases in a filesystem; retrieve official release names from files using the srrdb api (crc32 search); collect torrents from Prowlarr and H&R them (although I'd prefer crowd-source directly from the community!).

The index is stored on git to allow collaboration. It is hosted using Radicale to avoid centralization and reduce hosting pressures.

If you are interested, join and add your own hashes to the collection in Radicle patches! (see instructions in the README)

Let me know what you think, suggest improvements or discuss similar projects you know about!

Hello, since it's complicated to index DHTs, I figured it'd be more efficient to build an index of fingerprints from real data once.

So I've been collecting releases hashes for this index. It can be used for various purposes:

• check the integrity of your own files (bit rot is a real thing)
• identify BTv2 torrent files that contain specific files (a database of torrent files is required)
• locate alive IPFS swarms to join more easily (no need to read all your data multiple times to recompute various CIDs yourself)

The collection contains around 1K releases and weights 40MB. I've prioritized scene Bluray rips of movies (1080p / 2160p). No infohash will be included, as these are not reproducible enough.

I'm using a basic script to add a new release (filename must match the official release name). I'm using others to discover scene releases in a filesystem; retrieve official release names from files using the srrdb api (crc32 search); collect torrents from Prowlarr and H&R them (although I'd prefer crowd-source directly from the community!).

The index is stored on git to allow collaboration. It is hosted using Radicale to avoid centralization and reduce hosting pressures.

If you are interested, join and add your own hashes to the collection in Radicle patches! (see instructions in the README)

Let me know what you think, suggest improvements or discuss similar projects you know about!