The remove duplicates will only work if the entire row is duplicated.Different techniques are documented at this place