Hi, I work daily with this problem, the way I have resolved it is as follows,

Your Data in column A, insert new column at B, Sort A a-z acending. in B2 insert formula Exact B2 Exact as B1 copy down whole completed column, then filter "True" (All true's are duplicates, the first duplicate is ignored. These true's can be deleted. == No duplicates

When you've Done this a couple of times , takes seconds !!