"Lazy loading" performance vs. randomness

[stevensona]

This is a response to your blog post and modification to RanCat regarding a a supposed 100% performance improvement.

It's not acceptable to only consider the first 1000 elements of a file when selecting one at random when, more than likely, the dictionary will be presorted and have much more than 1000 elements.

I suggest you take the following approach:

1. Get the size of the file in bytes size = os.stat(filename).st_size
2. Open the file f= open(filename)
3. Jump to a random spot in said file f.seek(random.randint(0, size)
4. Move backward(or forward) until you hit a newline
5. Read the entire line word = f.readLine()
6. Close file f.close()

I wanted to see what you thought before putting together a PR.

[mattjegan]

I think this definitely could be viable, I hadn't thought of it before. One thing to keep in mind when implementing this is that we need it to work when RanCat._unique = True

I'm looking forward to your PR 😄

[mattjegan]

Hey @stevensona , are you still happy to implement this?

[stevensona]

@mattjegan I don't think I will have time to do this soon, unfortunately.