I suspect your data are loaded from an application, such as SAP, or oracle. In other words, you data are not originally created in excel, rather they are exported in to excel from external application. If my assumption is right, even 20k rows will crash excel.
That is my experience any way. You need to convert these data in VALUES ONLY FIRST before you work on them.