Make it possible to undeclare variables

[bullno1]

In some cases, unneeded variables could be removed to save space.

[cmeiklejohn]

I thought about this a bunch in detail and this definitely appears to be a lot less trivial than I originally thought.

One problem here is guarantees on ordering:

• We'd have to keep a tombstone around marking the object as deleted using a logical clock that dominates the logical clock that's associated with the last known write, like something like Riak does. Then, we would need custom logic in all of the anti-entropy backends that says that a normal CRDT merge is done, in addition to, special data store logic that states that an object tombstone overwrites the merge. This tombstone couldn't be an object like bottom, because that would always merge to the most recent state.

• This is further complicated with the delta mechanism, because we'd have to find a way to model a tombstone as a delta -- which, it can't be a value itself in the lattice, because again, we'd end up with the same merging problem.

• One alternative, would be to extend lattices with a top value that was used to represent deletions (or, variable undeclares) -- this seems the most reasonable solution since we can trivially extend all lattices in the system with a top value in the runtime, and ensure that merges always merge to top when either side of the merge is a top. However, this complicates matters further because it would prevent values from being redeclared ever again.

• Now, a more interesting thing might be to use the partial replication mechanism in Lasp: this ensures that values that are not within a "replication group" aren't stored on that node -- right now, this just filters objects out from the anti-entropy protocol, but ideally, we could also have this periodically prune the backend of the objects that are no longer needed. This might be the more reasonable idea here.

I'm looping @russelldb in to see what he thinks, since this is a problem that's at least been thought of in the context of Riak many times.

[cmeiklejohn]

@russelldb is also correct to point out that a single logical clock for the entire node can also be used to handle a removal as well.

The complications here come from the fact that internally in Lasp's key-value store, that any CRDT supporting an interface (Erlang behavior) is supported, so we can't assume a uniform data representation. So, either we need to extend the types in a generic way, or we need to support something specific in the backend (ie. Russell's solution or the partial replication scheme I proposed.)

[cmeiklejohn]

Obviously, another issue here to worry about is concurrent updating with a undeclare -- which would effectively restore the value.

In fact, the declare operation is superfluous anyway, because all it does is create a local register with the bottom value for the lattice, which is implicitly done through the update operation.

[tsloughter]

This is what I was planning to look at in case it was the cause of our memory leak. I think the memory spikes too quickly to this be the main issue we are seeing, but it could still be adding to it over time I figure.

[cmeiklejohn]

Can you extrapolate on why you think this might be the memory leak you are experiencing?

[tsloughter]

@cmeiklejohn based on the comment from @bullno1 it sounded like no longer used variables would continue to take up space, and our staging environment costs of continually creating new devices/channels that get registered, used briefly, and never used again, 24/7.

Just a thought, even if it is leaving the space taken it may be so little it doesn't matter and we have other concerns, I'm still trying to find where the issue is.

[tsloughter]

I'd been trying to find what was eating up all the memory on our node for a while now and finally discovered why it was so hard to find :). mem column for ets:i() is in words, not bytes, so now I see that the 1gig I couldn't account for is the lasp_ets_storage_backend.

[cmeiklejohn]

Yes, that's right.

So, you're creating new keys often and abandoning other keys? Is that the root cause of the issue? If so, we probably need to come up with a solution for this sooner rather than later.

Can you confirm this is the actual issue?

[tsloughter]

Just realized I never responded here and only in gitter.

I think the abandoning is not currently the issue. I'd expect that growth to be much slower if not for the growing waiting_threads element in dv record. They look to be related to enforce_once

[cmeiklejohn]

Can you dump the ets table (or, even just a single table entry) so we can inspect the stored state and identify where the bloating is coming from?

[cmeiklejohn]

FWIW: the previous fix on mater should have removed the issue with waiting_threads growing forever. That ensured that we prune terminated process at every invocation.

[tsloughter]

Oh, I'll try master.

And from the ets table the other day:

[{threshold,read,<0.1746.0>,awset,
            {strict,{state_awset,{[{<0.1741.0>,
                                    [{<<131,100,0,18,110,117,99,108,101,117,115,64,49,...>>,
                                      1}]}],
                                  {[{<<131,100,0,18,110,117,99,108,101,117,115,64,49,48,...>>,
                                     1}],
                                   []}}}}},

[tsloughter]

I'll be pushing the use of master tomorrow to see how it does.

It looks like there were more commits after the gc branch commit we were working off.