LinkBack URL
About LinkBacks
Hello all,
I have a large textfile (up to 20 000 000 rows), how can I read the last line fastest?
KalleK
Hello all,
I have a large textfile (up to 20 000 000 rows), how can I read the last line fastest?
KalleK
The suggestion in your other thread (ADO) is probably the easiest, but if the file contains fixed length records (All lines exactly the same size) then there is another possibility... Is it?
Hello cytop, no it is not fixed length. It is csv file with number and date and time. How do I use ADO to read?
The following should return the details from the last record in a CSV/TXT file. Minor changes are needed if another delimiter is used and it assumes the first line contains headings (although that won't really matter as you only want the last line.
It requires a reference to Microsoft ActiveX Data Objects (Tested with 6.1)
Please Login or Register to view this content.
Thanks! I will test and compare the times. Is is posibble with ADO to start to read from a line? Say line 40000 of a file with 2 000 000 lines
You can use the MOVE method, or one of it's variants.
In all cases I am ignoring the fact you might try to move before the first or after the last record - this will raise an error so you need to include error handling.Please Login or Register to view this content.
IF you only want to pick particular records - for example where a 'NAME' column contains 'Smith', you can specify that in the initial selection:
You can then move between those records matching 'SMITH' but it's beginning to get a little complicated now.Please Login or Register to view this content.
Thanks cytop! I get it to work and see if it is faster then my curret solution to find the end of the file. Thanks again for the help!
There is another method where you open the file in Binary mode, skip to the last position and then skip back 1 character at at time until you hit a Carriage Return or Line Feed character.
For example
This is the basis for what I mentioned earlier about fixed record lengths. If each line was exactly the same length it would be possible to calculate the offset position of any physical line directly. As is, it can really only be used to return the last line. Or perhaps the last 2nd/3rd and so on, lines but it begins to get messy.Please Login or Register to view this content.
The main advantage of this method is that positioning at the end of the file is instantaneous and then it only has to loop backwards to find the terminator for the previous line.
There may be an issue where the last line does not terminate with a Carriage Return or Line Feed. This is something you are going to have to check yourself and make any necessary changes to the code that sets the initial position of the pointer at the start of the loop.
Last edited by cytop; 12-06-2016 at 10:16 AM.
There are currently 1 users browsing this thread. (0 members and 1 guests)
Posting Permissions
Register To Reply
Bookmarks