Are duplicate files safe to delete? What a duplicate finder cannot know

The first time we pointed our own duplicate finder at a whole C: drive, it offered up 3,145 files, 25.1 GB, all ticked and ready for the Recycle Bin. Among them was msalruntime_x86.dll, sitting in the Visual Studio Installer’s own folder. The copy it had decided to keep was in a temp directory that Windows would clear on its own schedule.

That is not a bug in the hashing. The two files really were byte-for-byte identical. It is a bug in the idea that byte-for-byte identical means one of them is spare.

three reasons the same bytes exist twice on purpose

Programs ship their own copies

A Windows program loads its libraries from its own directory first. That is deliberate: shipping the exact DLL it was tested against is what stops it breaking when something else upgrades a shared one. So the same library legitimately exists inside five different program folders, and each of those five programs needs the copy in its folder. A scanner sees one file and four wins.

Windows keeps one file under several names

The component store, WinSxS, is largely hard links - a second name for the same data on disk, not a second copy. Deleting one name frees nothing at all, because the data stays as long as any name points at it. You get zero space back and a missing path.

Dependency trees copy on purpose

node_modules, Python virtual environments, game asset packs, Unity libraries: these duplicate files per project so each project stays self-contained and reproducible. Deduplicating across them is how you get a project that builds on Tuesday and not on Wednesday, with no change in the repository to explain it.

what should never be in the scan

The working list, in the order it removes the most danger. A duplicate finder that does not do this is asking you to audit thousands of paths by eye:

• Windows, Program Files, Program Files (x86), ProgramData and AppData - by known-folder path and by name, because on a real profile AppData is not even marked hidden.
• Any folder registered by an installed program, plus game libraries - Steam, Epic and the rest keep enormous identical asset files per title.
• Code projects. A folder containing .git is the reliable signal; your home directory is not.
• Hidden and system files, dot-folders, and files whose extension makes them program material - executables, libraries, virtual machine disks.
• Anything with a link count above one. That is a hard link, and removing it saves nothing.

What is left is the part that was always the point: your documents, photos, music, downloads and desktop. That is where real duplicates live - the same invoice saved twice, a photo import run twice, the installer you downloaded in March and again in August.

the OneDrive trap

Files in OneDrive, iCloud Drive or Dropbox may be placeholders: a name and a size on disk with the contents still in the cloud. To hash one, a scanner has to read it - and reading it downloads it. Point a duplicate finder at a synced folder and you can pull your entire cloud drive onto a laptop with 40 GB free, which is the opposite of the errand you set out on.

A well-behaved scanner checks the placeholder attributes on the open file handle and skips anything not already local. If yours does not document this, test it on a small synced folder and watch your free space before trusting it with the big one.

the size check nobody does

There is one more way this goes wrong, and it is the reason that 25.1 GB figure is worse than it looks. The Recycle Bin is not a folder that grows - it has a fixed size per drive, and on that machine it was 25.8 GB. Sending 25.1 GB into it would have evicted nearly everything already in there, and a batch any larger than the cap is destroyed outright rather than recycled. A bulk cleanup is exactly the situation where “it’s fine, it goes to the Recycle Bin” stops being true.

doing it safely

1. Scan one folder, not the drive. Pictures, then Downloads, then Documents. If a tool only offers “scan C:”, that tells you how much thought went into it.
2. Check the keep rule. “Keep the oldest” and “keep the newest” both routinely keep the wrong one. Better: keep the copy that is not named like a copy and not sitting in Downloads or on the Desktop - the one something is most likely to be pointing at.
3. Verify at the moment of deletion, not at scan time. Minutes or hours pass between a scan and a click. A copy should be re-hashed immediately before it is moved, along with the copy that is staying.
4. Never let the last copy go. Obvious, and a sort order plus a “select all” button has broken it in more than one tool.
5. Delete in batches your Recycle Bin can hold, or move to a to-delete folder and wait a week.

the part no scanner can know

We still got one wrong after all of that. A project without a .git folder looks like ordinary documents, and two identical audio tracks inside an audiobook folder were offered as duplicates - they were two chapters that genuinely had the same silence in them, or the same file legitimately placed twice by whatever produced the audiobook. From the outside, a file cannot say why it exists.

Which is why the last line of defence is not smarter rules. It is that everything goes to the Recycle Bin, the batch is small enough to fit, and you can put it back.

Who wrote this. We build owntools, which includes a disk analyser. Every rule above was added after our own tool offered something it should not have - the Visual Studio DLL came out of a real run on a real machine, not a hypothetical.