Question: Multi-user scenarios, atomicity, locks, etc

[jpray]

This is a really cool project. I'm trying to determine the limit of what it could/should be used for when it comes to situations where multiple users/sessions could be trying to write to the DB at the same time. I see the checkIsModified option, which seems to leave only a small window (between the checking if its been modified and the actual write) where another user/session could have written to the DB undetected. Would you say that one would need to implement there own lockfile solution to close that window completely?

For atomic transactions involving multiple models, I'm thinking a custom solution would also be needed. Do you agree?

Thanks again for the cool library.

[matt-filion]

You have hit one of my soon to do features right on the head. CheckIsModified is mostly for efficiency. Collision detection is going to be using a head request to see if the eTag of the document is the same> I was thinking of logic similar to the following.

First, all loaded and saved documents will need to start returning a version or eTag of some sort. Assuming that part is in place I see the save operation doing the following.

• Load metadata from target object.
• Check eTag to see if it matches, if it does not, fail.
• Check for 'lock' in metadata to see if another process already has it reserved.
• If lock exists, check for a date on the lock to see if its expired*. If its not expired, then fail
• Add lock to metadata (uuid that only this process is aware of)
• Re-load metadata, check to see if eTag still matches (just in case another process updated underneath of lock creation. If it does not match, clear lock and fail.
• Save document
• Delete lock

I would definitly agree with your point that s3-db is not appropriate when you need transactional atomicity. You can do a good enough job for a single object for most scenarios, but rollback and what not would require an intermediary of some sort to handle all the complexities of ACID transcations. Something like step functions could be used to simulate it, but thats well outside of the realm of this API at the moment.

[matt-filion]

One change to the above. After teh lock is written. Will wait for a random amount of time 50-100ms and check the lock again. If its as expected will proceed, otherwise will fault since another process 'won' the lock.

[jpray]

Thanks for the details. That helps a lot. It will be interesting to see what performance is achievable with the new algorithm.

[matt-filion]

I'm estimating 100-300ms. S3 latency is generally quite acceptable but with all the touch points above I doubt it will be irrelevant. My estimate is for lambda hitting S3 btw, this would be much worse local to s3 of course.