Getting LMDB-backed version to a working state

[dm3]

Hello!

I was looking at the current master and it looks very promising. The only part that is missing is hydrating the references when an entity is pulled from the database. I would like to complete the implementation. What follows is a couple of questions I came up with after an ~hour of reading the code:

1. Purpose of the primary database is storing the entities. Purpose of the secondary database is not very clear. I've looked up the implementation for Graphene-0.3.3 and the secondary database there was driven by the Berkeley DB implementation. It had the secondary indices for all of the properties of the entity. Was there a plan to use the secondary LMDB database for secondary indices too? I understand it would also need to store the references between the entities - or would that be a tertiary DB?.
2. Entity classes are created free of references to external resources. This way an entity created outside of Graphene and the one returned from Graphene::get are identical. Not sure if that's a design goal though. If we wanted to support lazily loaded references, for example, we'd need to embed the reference DB in the entity. However, I don't really like the concept as funny things will happen if you access the entity outside of a read transaction. Was there a certain semantics that you had in mind regarding the reference hydration?

[krisskross]

1. Yes, the secondary database is intended to store the references similar to Berkeley DB. I'm not sure if secondary indices or a tertiary DB is needed at the moment. Maybe implement references first and expand design if needed?
2. I think lazily loaded references is needed to get acceptable performance. I had the same concern around embedding the secondary database in the entity. It's probably best to provide an operation external from the entity to fetch references. Maybe some kind of cursor abstraction? We may have to restrict fetching references within a transaction. A quick and dirty solution might return the ids and let user fetch the entities manually. But we can probably do better than that.

I'm not familiar with the concept "reference hydration" - can you elaborate?

[dm3]

I believe the term "hydration" originated in Hibernate as a name for a process of populating object's fields with the query results. It became commonly used in other contexts as a synonym to deserialization into objects/entities. So didn't mean anything fancy :) Just loading the referenced entities.

Thanks for the answers, I'll think a bit about possible solutions given that we want to:

1. Allow for lazy-loading of entities (probably makes sense to add an eager option and default to fetching the references on first access inside a read transaction. Another problem is nested lazy references, what if you need to eagerly fetch a deep tree of entities? Then the fetch operation itself should be marked as eager/lazy instead of declaring a modifier on the entity.)
2. Keep the entities free of references to resources.

[krisskross]

Sounds good.

[krisskross]

I was just curious to know how things are going? Do you need any help?