Same name does not mean same file, and same content hides under different names. Hash every candidate, group the identical ones and review the list before anything is deleted.
Group files by **Get-FileHash**, keep the oldest path per hash, and export the candidates with **Export-Csv** before deleting. Verify the groups with a dry run first.
Hashing a whole drive takes hours. Identical files always have identical size, so group by length and skip every size that occurs once.
$root = 'D:\Archive'
Get-ChildItem $root -Recurse -File -ErrorAction SilentlyContinue | Group-Object Length | Where-Object Count -gt 1 | Sort-Object Count -Descending | Select-Object -First 10 Name,Count**Get-FileHash** reads every byte, so run it on the shortlist. MD5 is fine for duplicate detection and much faster than SHA256 on large sets.
$files = Get-ChildItem $root -Recurse -File -ErrorAction SilentlyContinue | Group-Object Length | Where-Object Count -gt 1 | ForEach-Object { $_.Group }
$hashes = $files | Get-FileHash -Algorithm MD5
$hashes.CountA hash group with more than one path is a real duplicate set. The oldest file is usually the original, and the newest is usually the copy that was downloaded twice.
$groups = $hashes | Group-Object Hash | Where-Object Count -gt 1
$groups | ForEach-Object { [pscustomobject]@{ Hash = $_.Name; Count = $_.Count; GB = [math]::Round((($_.Group | ForEach-Object { (Get-Item $_.Path).Length } | Measure-Object -Sum).Sum) / 1GB, 2) } } | Sort-Object GB -Descending | Select-Object -First 10Nothing is deleted in this step. The CSV lists every duplicate with its path, size and timestamp, which is the list you read before running any removal.
$plan = $groups | ForEach-Object { $keep = $_.Group | Sort-Object { (Get-Item $_.Path).LastWriteTime } | Select-Object -First 1; $_.Group | Where-Object Path -ne $keep.Path | ForEach-Object { [pscustomobject]@{ Keep = $keep.Path; Delete = $_.Path; Hash = $_.Hash } } }
$plan | Export-Csv D:\duplicates.csv -NoTypeInformation
$plan.Count
Import-Csv D:\duplicates.csv | Select-Object -First 5**-WhatIf** shows what would happen; run it once before the real call. Deleting to the recycle bin instead of permanently is the safer default for a first pass.
Import-Csv D:\duplicates.csv | ForEach-Object { Remove-Item $_.Delete -WhatIf }
# then, after the review
Import-Csv D:\duplicates.csv | ForEach-Object { Remove-Item $_.Delete -Force -ErrorAction SilentlyContinue }
(Import-Csv D:\duplicates.csv).CountThe sum before and after is the honest number. Keep the CSV until you are sure nothing important went with it.
$before = (Get-PSDrive D).Free
Remove-Item D:\duplicates.csv -ErrorAction SilentlyContinue
$after = (Get-PSDrive D).Free
'{0:N1} GB freed' -f (($after - $before) / 1GB)
Get-ChildItem $root -Recurse -File | Measure-Object Length -SumIt hashes far too many files. Limit the size filter, exclude folders such as node_modules and Windows, and run it per folder instead of per drive.
That is a hash collision, practically impossible with MD5 on real data. If it happens, compare with Compare-Object on the bytes.
System files and other users' profiles are unreadable. Filter to your own folders, or run the script elevated and accept that some paths still fail.
A sync client restored them. Pause the sync tool before the cleanup and let it reconcile afterwards.
For finding accidental duplicates, yes. It is not a security check, and it does not prove the files are malicious or benign.
No. That is the point: copies with different names and dates are caught, and files with identical names but different content are not.
Yes, as a scheduled task that writes a CSV for review. Never let it delete unattended; a wrong filter is expensive.
Deletion is permanent with -Force. Restore the files from the recycle bin or your backup, and keep the exported CSV so you know exactly which paths were removed.
Free · no account needed · tracks your progress