feat: Store timestamp and prepare GSI #7
Conversation
|
|
||
| "Persist timestamp" should { | ||
|
|
||
| "be the same for events stored in same transaction" in { |
There was a problem hiding this comment.
This is what we have in r2dbc, but I think we could actually make it increasing by (at least) one micros per event. For r2dbc I think the only reason we use same timestamp is that the timestamp can (optionally) be created by the database.
There was a problem hiding this comment.
changed this in 60aafab
| previousNow.updateAndGet { prev => | ||
| if (prev.isAfter(n)) prev.plus(1, ChronoUnit.MICROS) | ||
| else n | ||
| } |
There was a problem hiding this comment.
I'm actually not sure if this is needed or if Instant.now() is safe enough. I wouldn't trust System.currentTimeMillis, which is used by the Instant system clock, but it's only used for the offset baseline and it's always 1024 seconds back in time. Question is if getNanoTimeAdjustment is monotonically increasing?
There was a problem hiding this comment.
Independent of that, it's now needed.
The timestamp is now increasing by at least 1 micros, which is needed for the sorting of the by slice GSI. seqNr ordering and timestamp ordering should be the same.
The GSI is eventually consistent and may not preserve visibility order, but we handle that in other way. Having it in strict timestamp order will at least make the query return the items in the right order when the GSI has been populated.
There was a problem hiding this comment.
👍🏼 Yes, sort keys will need to be unique.
There was a problem hiding this comment.
not for gsi, actually, for ordinary tables the primary key has to be unique. I'm sure I read that was not the case for gsi. We will not have unique timestamps anyway, because different Akka nodes can create the same timestamp.
There was a problem hiding this comment.
Right, makes sense that index primary keys don't need to be unique. Could probably also have a composite sort key of timestamp and seq nr, if that's cleaner than needing the increasing timestamps.
* needed for sorting in the by slice GSI
| val slice = persistenceExt.sliceForPersistenceId(persistenceId) | ||
|
|
||
| def putItemAttributes(item: SerializedJournalItem) = { | ||
| import JournalAttributes._ | ||
| val attributes = new JHashMap[String, AttributeValue] | ||
| attributes.put(Pid, AttributeValue.fromS(persistenceId)) | ||
| attributes.put(SeqNr, AttributeValue.fromN(item.seqNr.toString)) | ||
| attributes.put(Slice, AttributeValue.fromN(slice.toString)) | ||
| attributes.put(EntityType, AttributeValue.fromS(item.entityType)) | ||
| attributes.put(EntityTypeSlice, AttributeValue.fromS(s"$entityType-$slice")) |
There was a problem hiding this comment.
Simpler to have a single attribute for GSI partition key. Otherwise we could combine multiple attributes into the partition key if it's useful to have the slice attribute separate.
There was a problem hiding this comment.
I thought it could only be one attribute for the partition key and one attribute for the sort key? Otherwise it would be better to use a partition key composed of entity type and slice attributes, for the gsi.
There was a problem hiding this comment.
I thought composite keys was a thing. I can take another look.
There was a problem hiding this comment.
Yeah, looks like only single attributes are possible and there's no composite key support.
No description provided.