Drop images
A folder or zip; thousands are fine. Hashing runs in the background with a progress bar.
Datasets
Duplicate Image Finder
Duplicates between training and validation make a model look better than it is. This tool hashes every image twice, an exact hash of the bytes and a perceptual hash of the picture, and groups images that match within a chosen distance. Review the groups, keep one from each, and download the cleaned set or a list to delete.
Loading the tool…
How it works
Set the threshold
Distance 0 finds exact and resized copies; 6-10 finds re-encoded and lightly cropped versions; higher catches near-identical frames from video.
Review and export
Each group shows its images side by side. Keep the largest by default, or choose. Export a zip without the duplicates, or a text list of files to remove.
Formats, limits and the numbers that matter
Hashes
Exact: SHA-256 of the file. Perceptual: 64-bit difference hash on a 9x8 greyscale downscale; distance is Hamming distance.
Cross-split check
Drop train and val folders together and the tool highlights pairs that cross the split, which is the leak that matters most.
Limits
Distance is compared pairwise within hash buckets, so 20,000 images stay fast; beyond that expect a minute or two.
Common questions
Why not just compare file names?
Because copies are renamed, re-saved and resized; the perceptual hash sees through all three.
Can it find duplicate text rows?
The JSONL Inspector reports exact duplicate rows; near-duplicate text is not covered here.
Are labels handled?
When removing an image, its YOLO label file with the same stem is removed too.