Add objects processor #215
Conversation
rtso
force-pushed
the
rtso/object-processor
branch
from
e65816e to
508cd45
Compare
rtso
force-pushed
the
rtso/object-processor
branch
4 times, most recently
from
b6afe39 to
00f7a4f
Compare
| /// Tracks all object related metadata in a hashmap for quick access (keyed on address of the object) | ||
| pub type ObjectAggregatedDataMapping = HashMap<CurrentObjectPK, ObjectAggregatedData>; | ||
|
|
||
| /// This contains metadata for the object | ||
| #[derive(Serialize, Deserialize, Debug, Clone)] | ||
| pub struct ObjectAggregatedData { | ||
| pub fungible_asset_store: Option<FungibleAssetStore>, | ||
| pub token: Option<TokenV2>, | ||
| } | ||
|
|
There was a problem hiding this comment.
There was a problem hiding this comment.
I think it's better to create a new struct specifically for objects. FungibleAssetAggregatedData expects non-null values which don't make sense with how objects are being parsed in objects processor. We also potentially want to add more things to ObjectAggregatedData in the future.
There was a problem hiding this comment.
FungibleAssetAggregatedData is designed to do what you're saying. Basically I realized that we need a lighter version than the token one so created that one. The only non null value there is the object itself which you definitely need for objects.
If you want to push for this as the minimum object metadata holder then I'd suggest rewriting the existing code to use what you're doing here. Not great to duplicate the same mechanism multiple times. We're already duplicating some of this across token and fungible assets.
There was a problem hiding this comment.
What could be also cleaner if you had guaranteed object core and persist that mapping across transactions. And then when you see object core again you don't even need to parse it. You can just grab it from the mapping.
There was a problem hiding this comment.
What if one of the struct gets deleted from the object. Can that happen?
rtso
force-pushed
the
rtso/object-processor
branch
from
a00fd21 to
6c5a9c6
Compare
| conn, | ||
| diesel::insert_into(schema::objects::table) | ||
| .values(&items_to_insert[start_ind..end_ind]) | ||
| .on_conflict((transaction_version, write_set_change_index)) |
There was a problem hiding this comment.
consider do_nothing for a transaction + event_indexer, this insertion is unlikely to change.
There was a problem hiding this comment.
for backfill i need do_update but after backfill we should change it back to do_nothing
There was a problem hiding this comment.
Yes but you don't need to change all the fields. Just the ones that you're backfilling.
There was a problem hiding this comment.
Oh actually you also needt o change inserted_at time otherwise it doesn't update.
rust/processor/migrations/2023-12-07-233224_add_objects_model/up.sql
Outdated
Show resolved
Hide resolved
rtso
force-pushed
the
rtso/object-processor
branch
from
6c5a9c6 to
85f9946
Compare
rtso
force-pushed
the
rtso/object-processor
branch
from
85f9946 to
72460e9
Compare
bowenyang007
force-pushed
the
rtso/object-processor
branch
from
a6b854a to
8e32c3c
Compare
| .set(( | ||
| is_token.eq(excluded(is_token)), | ||
| is_fungible_asset.eq(excluded(is_fungible_asset)), | ||
| )), |
There was a problem hiding this comment.
you do need to set all fields here.
* Add objects processor * Use ObjectAggregatedDataMapping in fungible asset and token v2 processors * Respond to comments * Update is_address_fungible_token check since data mapping type changed * rebase * add proper on conflict set --------- Co-authored-by: bowenyang007 <[email protected]>
* Add payload type to default processor * lint * Add objects processor (#215) * Add objects processor * Use ObjectAggregatedDataMapping in fungible asset and token v2 processors * Respond to comments * Update is_address_fungible_token check since data mapping type changed * rebase * add proper on conflict set --------- Co-authored-by: bowenyang007 <[email protected]> --------- Co-authored-by: Renee Tso <[email protected]>
Description
Backfill
Given this splits out the objects processor we need to start the new processor from when objects were created. No need to backfill default_processor though.
Testing
object is token
Testnet version 528464455
object is fungible asset
Testnet version 802980551
object is both token and fungible asset
Testnet version 617679498