-
Notifications
You must be signed in to change notification settings - Fork 46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
big collections redux #14
Comments
hey @javaknight do you have any more info on the crash? and how many records are in the collection? |
If the collection is truly large, it might be better to do something a little different:
Of course ideally you'd share the timeout between multiple users subbing to the same publication. Sounds like a whole package of it's own :) |
(This package pulls down an caches the |
@tmeasday: I think you mean |
Ahh, thanks @chhib - I updated the code so people aren't confused. |
@tmeasday Why is it necessary to cache the |
@colllin if you are talking about livedata's That's my understanding of it anyway. Possibly someone could figure out a way to use the low-level oplog driver and deal with these issues, not sure. |
@tmeasday Yes, that's what I was talking about. I didn't realize |
To be clear it's the oplog that is imperfect (I think there are a bunch of issues around the exact timing of doing your initial query vs where you start observing the oplog from).
|
@javaknight, I had this same problem because my collection at 100,000+ rows - I am implementing a "scrollbox" that loads a sliding window over a collection to emulate the browser loading the entire collection. I implemented the solution @tmeasday posted above at https://github.com/jchristman/meteor-collection-scroller/blob/master/lib/collections.js if you wanna check it out (also at http://scroller.meteor.com). Atmosphere Link |
@tmeasday regarding your setInterval example, instead couldn't you just make the observer based on a cursor that finds a limit of one row, sorted by newest to oldest, and then increment the count only when needed rather than by interval. And of course call
|
@faceyspacey - Seems like a good idea for collections where you do have a date field to work with. I'm not sure that the |
is there really no way for meteor's observers to skip calling all the |
|
then i guess overwriting |
so I guess |
@faceyspacey if you are going to think about wacky solutions like this, I'd suggest just denormalizing the count somewhere. |
well then just resetting the count on removes would be the solution. using a counts collection with the count from one publication denormalized into one row there. |
@tmeasday using setInterval can also be improved by keeping track of the previous count and only sending data to the client if the current (thisCount) is different from the previous count. |
Knowing the current limitation of oplog in combination with the exisitng observer api, I think the best solution for scale is to compute and store counts (in a mongodb collection or in the relevant doc) on insert and remove. Ex. On adding or removing comments from a post, update comments counter storage (possible on post, ie. post.commentsCount). This might look silly but works and scale very well. |
I think we could make this better if we do the following steps:
|
Just one point I'm a bit unclear on. If I have a large collection but only want to count a small number of these (like unread notifications for a particular online user not all notifications for all users). Then I am only caching the documents in the cursor right not the entire collection so this package would work very well for counting small numbers of things. However I suppose if I had 500 online users each with only 10 unread notifications I'd still be caching 5000 documents right? |
I need this to watch a big collection, but I don't want to actually publish the big collection to the client. I just need the count of the collection to be published, and if that count changes, then I need the change to be reactive and sent to the client.
Currently when I fire this up on a large collection, it just crashes my server after showing me the initial count.
The text was updated successfully, but these errors were encountered: