Say you have files in /home/bridgenjoyer/Data, /mnt/Backup, and /mnt/data

So

for d in /home/bridgeenjoyer/Data /mnt/backup /mnt/data
 do
    find $d -type f -print0
  done | xargs -0 md5sum | sort

• generates a list of all plain files by path name
• runs md5sum on them, which outputs a checksum as "checksum filename"
• sorts them by checksum

So, for identical files, you get identical repeated checksum antries.

You can filter out repeated entries (e.g. using awk, a shell script, or python) and extract their name and path.

There are also various tools which deduplicate identical files by hard linking them, if they are on the same file system. These will get the same inode number shown with ls -l. You can find these tools in the Arch wiki.

Then you need a strategy how to organize these identical files. You cold put all files into an archive folder, tidy them up, and only make and keep backups which you don't modify.

If you only care about saving drive space, you can also just use a file system that has deduplication built in, like btrfs, and just run a dedup pass.

This doesn't deduplicate file references, but does save space on disk.

Easy question, hard answer.

If you want the easy answer: one new folder, look through everything you want to keep, and copy it to the new folder. Then delete the rest.

For a quick deuplication, fdupes will find exact duplicates. Read the man page, because there's a cache option you probably want to use.

If there are inexact duplicates, like photos or videos, use tools like czkawka.

rsync has a --dry-run / -n flag that would tell you which files would be synced. Combined with --archive /-a it runs recursively.

That said, it's assumes file paths are matching between source and destination, so if things are scattered, it might not work well.

If you're really scattered, I'd consider using a simple python script to walk directories and create a json file with results (maybe the sha256 for key and filename, size, absolute path, and created/modified stamps as fields). Then when you hit a duplicate key, you can dump the results and prompt a delete.

Or something. I'm just spit balling here.

I'm in the long process of sorting out tons of duplicate files too, and I'm using krokiet, which is the new GUI version of czkawka. It's not perfect, but it is the best tool I have found so far.

There’s a couple programs I use.

Meld, for comparing folders exactly: http://meldmerge.org/

Czkawka for finding duplicate and similar files all scattered around: https://czkawka.net/

I'm only familiar with CLI tools but at least for smaller directories diff works well. Say you have a backup directory and you want to see how it differs from a source directory:

diff -rq /source/directory /backup/directory

Another trick is to use find to look for a unique file to see where, if at all, it's backed up:

find /backup/directory -name unique-file-name.jpg

Basically you want something that will hash the file contents and compare hashes, this gets around filename differences.

Define where your "final" location will be and compare each drive to that (the 1st one will be the starting point, so pick the best , largest drive

fdupes / rmlint will get you most of the way there.

Then you'll want to compare similar files, so maybe something like meld or beyondcompare might help.

This will take a loooong time, but will be worth it

I'm sure someone else has a better method, but I'd just use whatever file manager you have to consolidate everything into one folder and let the file system flag any dupes.

Have ten movie folders on a bunch of different drives? Pick the biggest one, attempt to move the next biggest one into it, ignore all dupes, repeat.

Again, though, there's probably a better way