feat(task db): implement a task DB #208
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…d in execute are 'NULL'
…tatus, persistent views, remove debug print
…r same task in case of failures-retry
petarvujovic98
temporarily deployed
to
test-environment
GitHub Actions
Inactive
petarvujovic98
changed the title
petarvujovic98
temporarily deployed
to
test-environment
GitHub Actions
Inactive
petarvujovic98
had a problem deploying
to
test-environment
GitHub Actions
Failure
petarvujovic98
had a problem deploying
to
test-environment
GitHub Actions
Failure
petarvujovic98
had a problem deploying
to
test-environment
GitHub Actions
Failure
petarvujovic98
had a problem deploying
to
test-environment
GitHub Actions
Failure
petarvujovic98
temporarily deployed
to
test-environment
GitHub Actions
Inactive
petarvujovic98
temporarily deployed
to
test-environment
GitHub Actions
Inactive
petarvujovic98
temporarily deployed
to
test-environment
GitHub Actions
Inactive
…tion (#296) * impl a mem db for easy integration Signed-off-by: smtmfft <[email protected]> * fix clippy and unit test Signed-off-by: smtmfft <[email protected]> * fix fmt Signed-off-by: smtmfft <[email protected]> --------- Signed-off-by: smtmfft <[email protected]>
petarvujovic98
temporarily deployed
to
test-environment
GitHub Actions
Inactive
petarvujovic98
had a problem deploying
to
test-environment
GitHub Actions
Failure
petarvujovic98
temporarily deployed
to
test-environment
GitHub Actions
Inactive
* enable sqlite db by feature Signed-off-by: smtmfft <[email protected]> * debug lifetime Signed-off-by: smtmfft <[email protected]> * resolve lifetime issue and make all tests pass Signed-off-by: smtmfft <[email protected]> * refactor(task_db): simplify structure for sqlite and use cached statements * feat(task_db): abstract task db implementation into wrapper * fix(task_db): add await to test call * fix(task_db): fix import declaration * fix(task_db): add async and mutable variables * fix(host): fix task manager usage * fix(task_db): fix test for async * Update Cargo.toml use in-mem as default. --------- Signed-off-by: smtmfft <[email protected]> Co-authored-by: smtmfft <[email protected]> Co-authored-by: smtmfft <[email protected]>
petarvujovic98
temporarily deployed
to
test-environment
GitHub Actions
Inactive
petarvujovic98
temporarily deployed
to
test-environment
GitHub Actions
Inactive
petarvujovic98
approved these changes
Jul 2, 2024
petarvujovic98
temporarily deployed
to
test-environment
GitHub Actions
Inactive
Champii
pushed a commit
to Champii/raiko
that referenced
this pull request
Jul 2, 2024
* task-manager: dump fight vs sqlite * task-manager: SQL tables+views+triggers success - but arguments passed in execute are 'NULL' * task-manager: passing enqueue_task tests * task-manager: cleanup - ease copy pasting to SQL script, Registered status, persistent views, remove debug print * task-manager: add DB size query * task-manager: id_proof is unneeded + prettify queries * task-manager: change DB schema - allow multiple provers and status for same task in case of failures-retry * task-manager: allow task updates * task-manager: retrieve cached proofs from DB * task-manager: add status check * task-manager: add progress reports * chore(task_manager): Run cargo fmt * feat: address small lints * feat(task-manager): use result type with thiserror * chore(task-db): fix typos * refactor(task-manager): clean up tests * fix(docker): unignore task manager * [WIP](task_manager): write initial task handler stubs * chore(task_manager): run cargo fmt * [WIP](task_manager): write status and proof get handlers * refactor(host): use merge instead of nest * chore(format): format workflow files * chore(deps): use consistent dependency style * chore(host): rename tx to task_channel * [WIP](task_manager): add initial submit logic * chore(clippy): remove unused parameter * chore(clippy): remove unused imports * refactor(core): add copy trait to proof types * feat(task_manager): simplify db and adapt tests * fix(clippy): fix dereference issue * [WIP]: handle proof request by worker and update task status * [WIP]: add block fetching and initial blockhash getting for submit * [WIP]: handle task creation, status and proof retrieval * fix(host): fix route sub-path * feat(raiko): abstract task manager and impl a mem db for easy integration (taikoxyz#296) * impl a mem db for easy integration Signed-off-by: smtmfft <[email protected]> * fix clippy and unit test Signed-off-by: smtmfft <[email protected]> * fix fmt Signed-off-by: smtmfft <[email protected]> --------- Signed-off-by: smtmfft <[email protected]> * fix: throw error instead of panicing on runtime checks * fix(core,task_manager): add custom ensure and require fns * feat(task_db): sqlite and in memory abstraction (taikoxyz#301) * enable sqlite db by feature Signed-off-by: smtmfft <[email protected]> * debug lifetime Signed-off-by: smtmfft <[email protected]> * resolve lifetime issue and make all tests pass Signed-off-by: smtmfft <[email protected]> * refactor(task_db): simplify structure for sqlite and use cached statements * feat(task_db): abstract task db implementation into wrapper * fix(task_db): add await to test call * fix(task_db): fix import declaration * fix(task_db): add async and mutable variables * fix(host): fix task manager usage * fix(task_db): fix test for async * Update Cargo.toml use in-mem as default. --------- Signed-off-by: smtmfft <[email protected]> Co-authored-by: smtmfft <[email protected]> Co-authored-by: smtmfft <[email protected]> * feat(task_manager): return empty list on key not found * feat(host,task_manager): add tracing and handle workers * feat(host): fix response structure * chore(clippy): remove unused imports * fix(ci): remove git merge added lines * fix(task_manager): add blob proof type field --------- Signed-off-by: smtmfft <[email protected]> Co-authored-by: Petar Vujović <[email protected]> Co-authored-by: smtmfft <[email protected]> Co-authored-by: smtmfft <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
When dealing with provers we need to be able to persist state of proving requests in case of network failure or hardware restart or cancellations.
In particular for Bonsai, without storing state, we would be unable to know how many tasks are in the queue, risking waiting in vain if there were none, or overloading the queue by mistake.
Usage
The DB introduced in this PR should be used either in front of Raiko, from taiko-geth or taiko-reth. Or after raiko. It can be either integrated in an existing product or as a microservice, for example as a daemon or a server.
Note
This PR currently has no doc as I need to leave it 95% finished due to personal reasons.
For @petarvujovic98, as discussed please review the error model so that it fits Raiko with anyhow, thiserror, ....
I might not be able to update it for several days so feel free to take over.
API
For now this provides
This will need a way to clear the GuestInput/Payload with "older than Duration" parameter, default 18 days (EIP-4844 blobs expiry) as those are several megabytes.