diff --git a/docs/knowledgebase/learn-substrate/off-chain-features.md b/docs/knowledgebase/learn-substrate/off-chain-features.md new file mode 100644 index 0000000000..d6c78dd7da --- /dev/null +++ b/docs/knowledgebase/learn-substrate/off-chain-features.md @@ -0,0 +1,102 @@ +--- +title: Off-Chain Features +--- + +## Overview + +There is often a need to query and/or process off-chain data before it can be included in the +on-chain state. The conventional way of doing this is through oracles. Oracles are external services +that typically listen to blockchain events and trigger tasks accordingly. When these tasks complete, +their results are submitted back to the blockchain using transactions. While this approach works, it +still has several flaws with respect to security, scalability, and infrastructure efficiency. + +To make the off-chain data integration secure and more efficient, Substrate provides off-chain +features: + +- **Off-Chain Worker (OCW)** subsystem allows execution of long-running and possibly non- + deterministic tasks (e.g. web requests, encryption/decryption and signing of data, random number + generation, CPU-intensive computations, enumeration/aggregation of on-chain data, etc.) that could + otherwise require longer than the block execution time. + +- **Off-Chain Storage** offers storage that is local to a Substrate node that can be accessed both + by off-chain workers (both read and write access) and on-chain logic (write access via off-chain + indexing but not read access). This is great for different worker threads to communicate to each + others and for storing user- / node-specific data that does not require consensus over the whole + network. + +- **Off-Chain Indexing** allows the runtime, if opted-in, to write directly to the off-chain storage + independently from OCWs. This serves as a local/temporary storage for on-chain logic and + complement to its on-chain state. + +Off-chain features run in their own Wasm execution environment outside of the Substrate runtime. +This separation of concerns makes sure that block production is not impacted by long-running +off-chain tasks. However, as the off-chain features are declared in the same code as the runtime, +they can easily access on-chain state for their computations. + +![Off-chain Workers](assets/off-chain-workers-v2.png) + +## Off-Chain Workers + +Off-chain workers have access to extended APIs for communicating with the external world: + +- Ability to + [submit transactions](https://substrate.dev/rustdocs/v2.0.0/sp_runtime/offchain/trait.TransactionPool.html) + (either signed or unsigned) to the chain to publish computation results. +- A fully-featured HTTP client allowing the worker to access and fetch data from external services. +- Access to the local keystore to sign and verify statements or transactions. +- An additional, local + [key-value database](https://substrate.dev/rustdocs/v2.0.0/sp_runtime/offchain/trait.OffchainStorage.html) + shared between all off-chain workers. +- A secure, local entropy source for random number generation. +- Access to the node's precise + [local time](https://substrate.dev/rustdocs/v2.0.0/sp_runtime/offchain/struct.Timestamp.html). +- The ability to sleep and resume work. + +OCWs can be initiated from within a special function in your runtime implementation, +[`fn offchain_worker(block: T::BlockNumber)`](https://substrate.dev/rustdocs/v2.0.0/frame_support/traits/trait.OffchainWorker.html). +communicate results back to the chain, off-chain workers can submit signed or unsigned transactions +to be included in subsequent blocks. + +Note that the results from off-chain workers are not subject to regular transaction verification. A +verification mechanism (e.g. voting, averaging, checking sender signatures, or simply "trusting") +should be implemented to determine what information gets into the chain. + +For more information on how to use off-chain workers in your next runtime development project, +please refer to our [Development Guide](../runtime/off-chain-workers). + +## Off-Chain Storage + +As its name indicated, the storage is not stored on-chain. It can be accessed by off-chain worker +threads (both read and write access) and on-chain logic (write only, refer to off-chain indexing +below). This storage is not populated among the blockchain network and does not need to have +consensus computation over it. + +As an off-chain worker thread is being spawned off during each block import, there could be more +than one off-chain worker thread running at any given time. So, similar to any multi-threaded +programming environment, there are also utilities to +[mutex lock]() the storage when accessing +them for data consistency. + +Off-chain storage serves as a bridge for various off-chain worker threads to communicate to each +others and between off-chain and on-chain logics. It can also be read using remote procedure calls +(RPC) so it fits the use case of storing indefinitely growing data without over-consuming the +on-chain storage. + +## Off-Chain Indexing + +Storage in the context of blockchain is mostly about on-chain state. But it is expensive (as it is +populated to each node in the network) and not recommended for historical or user-generated data +which grow indefinitely over time. + +We have off-chain storage for this purpose. In addition of being accessible by OCWs, Substrate also +includes a feature called "off-chain indexing" allowing the runtime to write directly to the +off-chain storage independently from OCWs. Nodes have to opt-in for persistency of this data via +`--enable-offchain-indexing` flag when starting up the Substrate node. + +Unlike OCWs, which are not executed during initial blockchain synchronization, off-chain indexing is +populating the storage every time a block is processed, so the data is always consistent and will be +exactly the same for every node with indexing enabled. + +## Learn More + +- [Off-Chain Workers Development Guide](../runtime/off-chain-workers) diff --git a/docs/knowledgebase/learn-substrate/off-chain-indexing.md b/docs/knowledgebase/learn-substrate/off-chain-indexing.md deleted file mode 100644 index 9c157cbe7d..0000000000 --- a/docs/knowledgebase/learn-substrate/off-chain-indexing.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -title: Off-Chain Indexing ---- - -## Overview - -When we think about storage in the context of blockchains, we usually think about the On-Chain state. -Using on-chain storage is expensive and not recommended for historical or user-generated data -which can exponentialy accumulate over time. -Luckily every Substrate node comes with Off-chain Database as well. It is usually associated -with [Off-chain Workers](./off-chain-workers) and it is valid, since the main use case for the -Off-chain DB is exactly this, but Substrate includes a feature called "Off-chain Indexing", -which allows the Runtime to write directly to the Off-chain DB independently from OCWs. Nodes -have to opt-in for persistency of this data via `--enable-offchain-indexing` CLI flag. - -Unlike Off-chain workers, which are not executed during initial blockchain synchronization, -Indexing is populating the DB every time a block is processed, so the data is always consistent -and will be exactly the same for every node with indexing enabled. - -Off-chain DB can be read using RPC calls so this makes it an interesting use case for storing -historical data for the UI without polluting the on-chain state. - -To use indexing data from Runtime: -```rust -sp_io::offchain_index::set(&b"my_key", &"my_value".encode()); -``` - -Data in Off-chain DB can be either read by Off-chain workers (local storage API) or via RPC, using -`offchain_localStorageGet` method. - - diff --git a/docs/knowledgebase/learn-substrate/off-chain-workers.md b/docs/knowledgebase/learn-substrate/off-chain-workers.md deleted file mode 100644 index 60381f941b..0000000000 --- a/docs/knowledgebase/learn-substrate/off-chain-workers.md +++ /dev/null @@ -1,48 +0,0 @@ ---- -title: Off-Chain Workers ---- - -## Overview - -There is often a need to query and/or process off-chain data before it can be included in the -on-chain state. The conventional way of doing this is through oracles. Oracles are external services -that typically listen to blockchain events and trigger tasks accordingly. When these tasks complete, -their results are submitted back to the blockchain using transactions. While this approach works, it -still has several flaws with respect to security, scalability, and infrastructure efficiency. - -To make the off-chain data integration secure and more efficient, Substrate provides off-chain -workers. The off-chain worker subsystem allows execution of long-running and possibly non- -deterministic tasks (e.g. web requests, encryption/decryption and signing of data, random number -generation, CPU-intensive computations, enumeration/aggregation of on-chain data, etc.) that could -otherwise require longer than the block execution time. - -Off-chain workers have their own Wasm execution environment outside of the Substrate runtime. This -separation of concerns is to make sure that the block production is not impacted by the long-running -tasks. However, as the off-chain workers are declared in the same code as the runtime, they can -easily access on-chain state for their computations. - -![Off-chain Workers](assets/off-chain-workers-v2.png) - -## APIs - -Offchain workers have access to extended APIs for communicating with the external world: - -- Ability to submit transactions (either signed or unsigned) to the chain to publish computation - results. -- A fully-featured HTTP client allowing the worker to access and fetch data from external services. -- Access to the local keystore to sign and verify statements or transactions. -- An additional, local key-value database shared between all off-chain workers. -- A secure, local entropy source for random number generation. -- Access to the node's precise local time and the ability to sleep and resume work. - -Off-chain workers can be initiated from within a special function in your runtime implementation, -`fn offchain_worker(block: T::BlockNumber)`. The function is executed after each block import. To -communicate results back to the chain, off-chain workers can submit signed or unsigned transactions -to be included in subsequent blocks. - -Note that the results from off-chain workers are not subject to regular transaction verification. A -verification mechanism (e.g. voting, averaging, checking sender signatures, or simply "trusting") -should be implemented to determine what information gets into the chain. - -For more information on how to use off-chain workers in your next runtime development project, -please refer to our [Development Guide](../runtime/off-chain-workers). diff --git a/docs/knowledgebase/runtime/off-chain-workers.md b/docs/knowledgebase/runtime/off-chain-workers.md index 9ee07a2fa8..1e66e109b4 100644 --- a/docs/knowledgebase/runtime/off-chain-workers.md +++ b/docs/knowledgebase/runtime/off-chain-workers.md @@ -4,7 +4,7 @@ title: Off-Chain Workers This article covers the technical aspects of using off-chain workers in a Substrate runtime. For a conceptual overview of off-chain workers see the -[Conceptual Guide](../learn-substrate/off-chain-workers). +[Conceptual Guide](../learn-substrate/off-chain-features#off-chain-workers). ## Using Off-Chain Workers in the Runtime @@ -401,6 +401,7 @@ using an external library to parse the JSON result in a `no_std` environment. ### Learn More +- [Off-Chain Features Conceptual Guide](../learn-substrate/off-chain-features#off-chain-workers) - [Signed Transactions](../learn-substrate/extrinsics#signed-transactions) - [Unsigned Transactions](../learn-substrate/extrinsics#unsigned-transactions) diff --git a/website/sidebars.json b/website/sidebars.json index 65c6ff8621..e157858cb3 100644 --- a/website/sidebars.json +++ b/website/sidebars.json @@ -12,7 +12,7 @@ "knowledgebase/learn-substrate/account-abstractions", "knowledgebase/learn-substrate/session-keys", "knowledgebase/learn-substrate/weight", - "knowledgebase/learn-substrate/off-chain-workers" + "knowledgebase/learn-substrate/off-chain-features" ], "Runtime": [ "knowledgebase/runtime/index",