Tools Duplicates? get rid of them.

[Arthur]

Duplicate Detection Tools​

Short post with links to my preferred duplicate files cleaners.

Spoiler: 1 - VIDUPE [doesn't work for pics]

Please, Log in or Register to view spoilers

Spoiler: 2 - DUPEGURU [doesn't work for videos]

Please, Log in or Register to view spoilers

Spoiler: 3 - VIDEO DUPLICATE FINDER [works for pics too] (unstable)

Please, Log in or Register to view spoilers

Also remember to do proper "manual" (pun somewhat intended ) curation before. This means actually checking what you downloaded. In my case I hunt down headless pics, cunt closeups, bad mirror shots, LQ , and other uninteresting stuff.
As a good practice: DON'T SIMPLY DOWNLOAD TO A FOLDER AND LEAVE THE STUFF THERE. You will never curate that stuff once it gained volume.


There you have them. If you have more recommendations try to follow the OP format if possible; as the minimum should have the download link and a little description.
​

 

[⨳ βoogiε ɳigɧʦ ※]

Is there any better than

Please, Log in or Register to see links and images

?
No longer updated but works just fine

The thing that's great about vidupe is that
1 it's quite fast
2 you can tweak the comparison threshold
3 employs two methods: pHash (perceptual hashing) and SSIM (structural similarity)

you should give it a try,highly recommend it!

post any alternatives below

 

[Master Roshi]

DupeGuru is pretty good, still in development, and is available for Mac and Linux as well:

Please, Log in or Register to see links and images

 

[fang]

If you are using a linux system and don't want to install applications:

You can use the following command to calculate a hash of all the files in the current directory and all sub folders, compare them and print out only the duplicates:

Please, Log in or Register to view codes content!

Notice: A hash is like an ID for a file. If 2 files have the same hash then the files are EXACTLY the same. If 2 files are the same but in a different resolution then this will NOT mark them as duplicates.

 

[Bamrikotito]

These are the main tools I use.

Please, Log in or Register to see links and images

: this extend the command that fang said to also search for similar images instead of finding 1:1 dupes, I use it for photos and 1:1 dupes. Can also do videos but I found it underperforming.

Please, Log in or Register to see links and images

: Tool I use for videos, can also do images but I find czkawka faster.

As always false positives and non-detection using similarity algorithms can happen. I've heard of dupeguru and have it installed but I still haven't used it yet, these two meets my needs.

 

[⨳ βoogiε ɳigɧʦ ※]

The thing that's great about vidupe is that
1 it's quite fast
2 you can tweak the comparison threshold
3 employs two methods: pHash (perceptual hashing) and SSIM (structural similarity)

you should give it a try,highly recommend it!

 

[Bamrikotito]

Please, Log in or Register to view quotes

Oh yea, It is just because you said it is no longer developed, the one I linked uses ffmpeg (used by youtube-dl or yt-dlp) and can also change settings.

 

[X34]

Antidupl.net is what I use for pictures

Please, Log in or Register to see links and images

Has a good interface for showing the pics and has an auto feature which works to some degree but will hand things back to you when it needs your input. You can change things like match percentage etc, best to try it out


I also use vidupe for vids and like it. I would like if it had some sort of auto feature even if it only did some

 

[pvtrich]

https://simpcity.su/threads/video-duplicate-removal-tools.10757/

 

[iNeed2Sleep]

CCleaner

 

[Sourcerer]

I use SimilarImages (

Please, Log in or Register to see links and images

). Tried using others, but this one seems to work best. It works on different file formats, though I mostly use it for videos. Last update was in 2013, though it works fine on Windows 10. It caches the file paths, so running it the second time will run much faster, and will only check for new or modified files. It also remembers which matches you skipped (e.g. false positives) and it doesn't show them again.

 

[pushitplayit]

Please, Log in or Register to view quotes

2nd these. I use these in combo as well. I would like to find a comprehensive review of all these dupe finders to see which is best so I would only need to use 1 software, but my google'fu fails me.

 

[Pornwaffe]

Please, Log in or Register to view quotes

Do you know of any similar functionality in windows?

 

[fang]

Please, Log in or Register to view quotes

Sadly not. On windows i used

Please, Log in or Register to see links and images

, which someone else recommended already and i love him/her for it. (Bamrikotito)

Simplified installation process:

1. Click on the following link to download the tool:

Please, Log in or Register to see links and images

2. Unpack it with winrar / 7zip

3. Doubleclick the czkawka_gui.exe

Then simply Add a folder at the top, click on "Duplicate Files" on the left and lastly on the "Search" button in the lower left. Easier and better than windows powershell commands.

 

[BobaBagus]

i use

Please, Log in or Register to see links and images

for this exact reason.
Stash is a self-hosted webapp that have duplicate checker function.

Please, Log in or Register to see links and images

 

[fuckerboy007]

I use

Please, Log in or Register to see links and images

for this

 

[itbftw]

Please, Log in or Register to view quotes

Powershell...
$c=@{};gci -Filter * -Path "C:\path" -File|%{$h=(Get-FileHash -LiteralPath $_.fullName -Algorithm MD5).hash;if($c[$h]){$_.fullname}else{$c[$h]=$_.fullname}}

 

[Pornwaffe]

Please, Log in or Register to see links and images