Measure the problem first
A folder that takes seconds to enumerate has a count and a size. Both numbers decide whether the cause is the view, the disk or the indexer.
$p = 'C:\Users\me\Downloads'
(Get-ChildItem $p -Force | Measure-Object).Count
(Get-ChildItem $p -Force | Measure-Object -Property Length -Sum).Sum
Measure-Command { Get-ChildItem $p -Force | Out-Null } | Select-Object TotalMilliseconds