AleksandarFilipov
How to seek a huge text file at the speed of light (well elixir :)
Hi forum, it’s been a while,
I would like some input from you brainics on the following;
I have huge textfile (GB) where each line starts timestamp.
timestamp0 data
timestamp1 data
...
So given a timestamp as input I need to locate the closest timestamp in the file and then parse line by line. Naturally this seek operation should be quick!
Well we know that this has been done a billion times before, however it hasn’t been done by me 
I should be mentioned that the huge textfile, is produced during a recording session which I control - opening for a possibility to produce a secondary meta data file during the recording. (well to be fair there is actually already a meta data file containing some json (however no indexing)).
Surely there must exist a “by the book” solution here.
Thanks!
Most Liked
dimitarvp
…or you can import it into SQLite and query that. 
hst337
Is the file ordered by timestamp? If it is, you can perform binary search using :file.pread
fmn
Well, perhaps I should put more emphasis Something in “Something like:”
I am sure if you will take a closer look, you will find more issues - it’s just an idea.
That being said, I don’t think what you wrote is correct. If timestamp is at the end of file, none of the processes will exit because of the condition you described.
EDIT: To clarify - in what I described, search is going forward. If timestamp you are looking for is lower than first timestamp you encounter during the search, it means this particular process can’t find it. But other can.
hst337
If timestamp you are looking for is lower than first timestamp you encounter during the search, it means this particular process can’t find it
And if the timestamp you’re looking is higher than the first timestamp of the next segment, current process can exit straight away.
So, N-1 processes will exit straight away in the algorithm you’re describing.
I think what you actually want to implement is some kind of binary (or N-ary) search. The idea is close to divide and conquer, but they’re not the same
fmn
Ahh, that’s what you mean. Cool idea actually, what we are missing here, is processes having common state or talking to each another.
Yes, in that sense it could be very interesting optimization, I agree.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Develop your skills with books, videos, and courses.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"