Page Not Found
We could not find what you were looking for.
Please contact the owner of the site that linked you to the original URL and let them know their link is broken.
\\n Fig. Event Driven Architecture with Message Queues - RabbitMQ\\n
\\n\\n#### Advantages\\n\\n- Improved Scalability\\n- Enhanced Responsiveness\\n- Enhanced Fault Tolerance\\n- Simplified Complex Workflows\\n- Real-Time Data Processing\\n\\n#### Challenges\\n\\n- Event Ordering\\n- Data Consistency\\n- Monitoring and Debugging\\n- Event Deduplication\\n\\nYou can mitigate or reduce these challenges by following best practices like Event Sourcing, Idempotent Processing, CQRS (Command Query Responsibility Segregation), and Event Versioning.\\n\\n### 2. The [Saga Pattern](https://microservices.io/patterns/data/saga.html)\\n\\nThis design pattern aims to achieve consistency across different services in a distributed system by breaking complex transactions spanning multiple components into a series of local transactions. Each of these transactions triggers an event or message that starts the next transaction in the sequence. If any local transaction fails to complete, a series of compensating actions roll back the effects of preceding transactions. While the orchestration of local transactions can vary, the pattern aims to achieve consistency in a microservices-based system. Events are designed to be stored in durable storage systems or logs, providing a trail to reconstruct the system to a state after a failure. While the saga pattern is an effective way to ensure consistency, it can be challenging to implement timer/timeout-based workflows and to design and implement the compensating actions for local transactions.\\n\\n**Note**: In the Saga pattern, a compensating transaction must be idempotent and retryable. These principles ensure that transactions can be managed without manual intervention.\\n\\n\\n\\n\\n Fig. The Saga Pattern for Order delivery system\\n
\\n\\n#### Advantages\\n\\n- Ensures data consistency in a distributed system without tight coupling.\\n- Provides Roll back if one of the operations in the sequence fails.\\n\\n#### Drawbacks\\n\\n- Might be challenging to implement initially.\\n- Hard to debug.\\n- Compensating transactions don\u2019t always work.\\n\\n### 3. [Stateful Orchestrators](https://docs.oracle.com/en/applications/jd-edwards/cross-product/9.2/eotos/creating-a-stateful-orchestration-release-9-2-8-3.html#u30249073)\\n\\nStateful orchestrators provide a solution for long-running workflows by maintaining the state of each step in a workflow. Each step in a workflow represents a task, and these tasks are represented as states inside workflows. Workflows are defined as state machines or directed acyclic graphs (DAGs). In this approach, an orchestrator handles task execution order, transitioning, handling retries, and maintaining state. In the event of a failure, the system can recover from the persisted state. Stateful orchestrators offer significant value in fault tolerance, consistency, and observability. It\u2019s one of the solutions proven effective in modern distributed computing. Some well-known services that provide this solution include:\\n\\n- [Apache Airflow](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/overview.html)\\n- [Azure Logic Apps](https://azure.microsoft.com/en-us/products/logic-apps)\\n\\n#### Advantages\\n\\n- **High Resiliency**: Stateful orchestrators provide high resiliency in case of outages, ensuring that workflows can continue from where they left off.\\n- **Data Persistence**: They allow you to keep, review, or reference data from previous events, which is useful for long-running processes.\\n- **Extended Runtime**: Stateful workflows can continue running for much longer than stateless workflows, making them suitable for complex and long-running tasks.\\n\\n#### Challenges\\n\\n- **Additional Complexity**: They introduce additional complexity, requiring you to manage issues such as load balancing, CPU and memory usage, and networking.\\n- **Cost**: With stateful workflows, you pay for the VMs that are running in the cluster, whereas with stateless workflows, you pay only for the actual compute resources consumed.\\n\\n### 4. Durable Execution\\n\\nDurable execution refers to the ability of a system to preserve the state of an application and persist execution despite failures or interruptions. Durable execution ensures that for every task, its inputs, outputs, call stack, and local variables are persisted. These constraints, or rather features, allow a system to automatically retry or continue running in the face of infrastructure or system failures, ultimately ensuring completion.\\n\\nDurable execution isn\u2019t a completely distinct solution from the ones listed above but rather incorporates some of their strengths while presenting a more comprehensive approach to achieving consistency, fault tolerance, data integrity, resilience for long-running processes, and observability.\\n\\nFig. Durable workflow engine
\\n\\n#### Advantages\\n\\n- **Reduced Manual Intervention**: Minimizes the need for human intervention by handling retries and failures programmatically.\\n- **Improved Observability**: Provides a clear audit trail and visibility into the state of workflows, which aids in debugging and monitoring.\\n- **Scalability**: Scales efficiently across distributed systems while maintaining workflow integrity.\\n\\n#### Challenges\\n\\n- **Resource Intensive**: Persistent state storage and management can consume significant resources, especially in large-scale systems.\\n- **Latency**: The need to persist state and handle retries can introduce latency in the execution flow.\\n\\nAs durable execution grows to be a fundamental driver of distributed computing, some of the solutions which use this architecture are\\n\\n- [Temporal](https://temporal.io/)\\n- [Uber Cadence](https://cadenceworkflow.io/)\\n\\nAmong these, [Temporal](https://temporal.io/) has grown in influence, used by companies like SnapChat, HashiCorp, Stripe, DoorDash, and DataDog. Its success is driven by its practical application in real-world scenarios and the expertise of its founders.\\n\\nAt Metatype, we recognize the value of durable execution and are committed to making it accessible. Our [Temporal Runtime](/docs/reference/runtimes/temporal) integrates seamlessly into our declarative API development platform, enabling users to harness the power of Temporal directly within Metatype. For those interested in exploring further, our documentation provides a detailed guide on getting started with [Temporal Runtime](/docs/reference/runtimes/temporal).\\n\\nBelow is an example of how you can build a simple API to interact with an order delivery temporal workflow within Metatype.\\n\\n:::note\\nIf you are new to Metatype or haven\u2019t set it up yet in your development environment. You can follow this [guideline](/docs/tutorials/quick-start).\\n:::\\n\\nFor this example, the order delivery system will have few components/services such as Payment, Inventory and Delivery.\\n\\nYour temporal workflow definition should look similar to the one below.\\n\\n\\n\\n {\\n console.log(`Processing payment for order ${orderId}`);\\n // Simulate payment processing logic\\n await sleep(2);\\n return \\"Payment processed\\";\\n}\\n\\nexport async function checkInventory(orderId: string): Promise {\\n console.log(`Checking inventory for order ${orderId}`);\\n // Simulate inventory check logic\\n await sleep(2);\\n return \\"Inventory available\\";\\n}\\n\\nexport async function deliverOrder(orderId: string): Promise {\\n console.log(`Delivering order ${orderId}`);\\n // Simulate delivery logic\\n await sleep(5);\\n return \\"Order delivered\\";\\n}\\n```\\n\\n
\\n\\nActivities definition inside `src/activities.ts`:`
\\n\\n```typescript\\nasync function sleep(time: number) {\\n return new Promise((resolve) => {\\n setTimeout(resolve, time);\\n });\\n}\\n\\nexport async function processPayment(orderId: string): Promise\\n\\n;\\n checkInventory(orderId: string): Promise;\\n deliverOrder(orderId: string): Promise;\\n }>({\\n startToCloseTimeout: \\"10 seconds\\",\\n });\\n\\nexport async function OrderWorkflow(orderId: string): Promise {\\n const paymentResult = await processPayment(orderId);\\n const inventoryResult = await checkInventory(orderId);\\n const deliveryResult = await deliverOrder(orderId);\\n return `Order ${orderId} completed with results: ${paymentResult}, ${inventoryResult}, ${deliveryResult}`;\\n}\\n```\\n
\\n\\nWorkflow definition inside `src/workflows.ts`:
\\n```typescript import {proxyActivities} from \\"@temporalio/workflow\\";\\n\\nexport const { processPayment, checkInventory, deliverOrder } =\\n proxyActivities<{\\n processPayment(orderId: string): Promise\\n
\\n\\nAfter you have setup the above components, now you need a client to start of any `OrderWorkflow`. Here is where metatype comes in, through the simple APIs [Temporal Runtime](/docs/reference/runtimes/temporal) exposes, you can communicate with your temporal cluster.\\nDown below is the workflow communication bridge for this system expressed within a [typegraph](/docs/reference/typegraph) which includes endpoints to start a new workflow and describe an existing one.\\n\\n```typescript\\nimport { Policy, t, typegraph } from \\"@typegraph/sdk/index.ts\\";\\nimport { TemporalRuntime } from \\"@typegraph/sdk/providers/temporal.ts\\";\\n\\ntypegraph(\\n {\\n name: \\"order_delivery\\",\\n },\\n (g: any) => {\\n const pub = Policy.public();\\n\\n const temporal = new TemporalRuntime({\\n name: \\"order_delivery\\",\\n hostSecret: \\"HOST\\",\\n namespaceSecret: \\"NAMESPACE\\",\\n });\\n\\n const workflow_id = \\"order-delivery-1\\";\\n\\n const order_id = t.string();\\n\\n g.expose(\\n {\\n start: temporal.startWorkflow(\\"OrderWorkflow\\", order_id),\\n describe: workflow_id\\n ? temporal.describeWorkflow().reduce({ workflow_id })\\n : temporal.describeWorkflow(),\\n },\\n pub,\\n );\\n },\\n);\\n```\\n\\n Worker definintion inside `src/worker.ts`:
\\n\\n```typescript\\nimport { NativeConnection, Worker } from \\"@temporalio/worker\\";\\nimport * as activities from \\"./activities\\";\\nimport { TASK_QUEUE_NAME } from \\"./shared\\";\\n\\nasync function run() {\\n const connection = await NativeConnection.connect({\\n address: \\"localhost:7233\\",\\n });\\n\\n const worker = await Worker.create({\\n connection,\\n namespace: \\"default\\",\\n taskQueue: TASK_QUEUE_NAME,\\n workflowsPath: require.resolve(\\"./workflows\\"),\\n activities,\\n });\\n\\n await worker.run();\\n}\\n\\nrun().catch((err) => {\\n console.error(err);\\n process.exit(1);\\n});\\n```\\n\\n\\n
\\n\\nActivities definition inside `activities.py`.
\\n\\n```python\\nfrom temporalio import activity\\nimport time\\n\\n@activity.defn\\nasync def process_payment(order_id: str) -> str:\\n print(f\\"Processing payment for order {order_id}\\")\\n # Simulate payment processing logic\\n time.sleep(5)\\n return \\"Payment processed\\"\\n\\n@activity.defn\\nasync def check_inventory(order_id: str) -> str:\\n print(f\\"Checking inventory for order {order_id}\\")\\n # Simulate inventory check logic\\n time.sleep(4)\\n return \\"Inventory available\\"\\n\\n@activity.defn\\nasync def deliver_order(order_id: str) -> str:\\n print(f\\"Delivering order {order_id}\\")\\n time.sleep(8)\\n # Simulate delivery logic\\n return \\"Order delivered\\"\\n```\\n\\n\\n
\\n\\nAfter you have setup the above components, now you need a client to start of any `OrderWorkflow`. Here is where metatype comes in, through the simple APIs [Temporal Runtime](/docs/reference/runtimes/temporal) exposes, you can communicate with your temporal cluster.\\nDown below is the workflow communication bridge for this system expressed within a [typegraph](/docs/reference/typegraph) which includes endpoints to start a new workflow and describe an existing one.\\n\\n```python\\nfrom typegraph import t, typegraph, Policy, Graph\\nfrom typegraph.providers.temporal import TemporalRuntime\\n\\n\\n@typegraph()\\ndef example(g: Graph):\\n public = Policy.public()\\n\\n temporal = TemporalRuntime(\\n \\"example\\", \\"HOST\\", namespace_secret=\\"NAMESPACE\\"\\n )\\n\\n workflow_id = \\"order-delivery-1\\"\\n\\n order_id = t.string()\\n\\n g.expose(\\n public,\\n start=temporal.start_workflow(\\"OrderWorkflow\\", order_id),\\n describe=temporal.describe_workflow().reduce({\\"workflow_id\\": workflow_id})\\n if workflow_id\\n else temporal.describe_workflow(),\\n )\\n```\\n\\n Worker defintion inside `run_worker.py`.
\\n\\n```python\\nimport asyncio\\n\\nfrom temporalio.client import Client\\nfrom temporalio.worker import Worker\\n\\nfrom activities import process_payment, deliver_order, check_inventory\\nfrom shared import ORDER_DELIVERY_QUEUE\\nfrom workflows import OrderWorkflow\\n\\n\\nasync def main() -> None:\\n client: Client = await Client.connect(\\"localhost:7233\\", namespace=\\"default\\")\\n worker: Worker = Worker(\\n client,\\n task_queue=ORDER_DELIVERY_QUEUE,\\n workflows=[OrderWorkflow],\\n activities=[process_payment, check_inventory, deliver_order],\\n )\\n await worker.run()\\n\\n\\nif __name__ == \\"__main__\\":\\n asyncio.run(main())\\n```\\n\\n\\n
\\n\\nYou need to add only the last two lines as the others are auto-generated. Note that secrets are defined under the `example` parent, which is the name of your typegraph. If the name doesn\'t match, you will face secret not found issues when deploying your typegraph.\\n\\nBefore deploying the above typegraph, you need to start the temporal server and the worker. You need to have [temporal](https://learn.temporal.io/getting_started/typescript/dev_environment/#set-up-a-local-temporal-service-for-development-with-temporal-cli) installed on your machine.\\n\\nmetatype.yaml
\\n\\n```yaml\\ntypegate:\\n dev:\\n url: \\"http://localhost:7890\\"\\n username: admin\\n password: password\\n secrets:\\n example:\\n POSTGRES: \\"postgresql://postgres:password@postgres:5432/db\\"\\n HOST: \\"http://localhost:7233\\"\\n NAMESPACE: \\"default\\"\\n```\\n\\n\\n\\n\\n\\n ```typescript npx ts-node src/worker.ts ```\\n \\n\\n```python python run_worker.py ``` \\n\\n \\n
\\n\\nAfter booting the temporal server, run the command down below to get a locally spinning [typegate](/docs/reference/typegate) instance with your typegraph deployed.\\n\\n```bash\\nmeta dev\\n```\\n\\nAfter completing the above steps, you can access the web GraphQL client of the typegate at [`http://localhost:7890/example`](http://localhost:7890/example). Run this query inside the client to start your workflow.\\n\\n```graphql\\nmutation {\\n start(\\n workflow_id: \\"order-delivery-3\\"\\n task_queue: \\"order-delivery-queue\\"\\n args: [\\"order12\\"]\\n )\\n}\\n```\\n\\nAfter a successful run, you will get the following result which includes the `run_id` of the workflow which has just been started.\\n\\nBoot up temporal
\\n\\nStart the temporal server.\\n\\n```bash\\ntemporal server start-dev\\n```\\n\\nStart the worker.\\n\\n![\\"FIRST](\\"/images/wasi_vfs_python_and_rust.svg\\")
\\n\\n### File system access\\n\\nSince the PyO3 project doesn\'t support [statically linking](https://github.com/PyO3/pyo3/issues/416) the Python runtime, we\'ll need to find a way dynamically link `libpython`.\\nThankfully, Wasm does support [dynamic linking](https://github.com/WebAssembly/design/blob/main/DynamicLinking.md) and wasm builds of [`libpython`](https://github.com/vmware-labs/webassembly-language-runtimes/tree/main/python) are available curtsy of the WebAssembly Language Runtimes project. \\nBringing all of this together isn\'t as simple though as PyO3\'s tries to load `libpython` from certain _paths_, a concept that isn\'t exactly clearly defined in Wasm\'s post POSIX webtopia.\\n\\nOur first solution was to use [wasi-vfs](https://github.com/kateinoigakukun/wasi-vfs), a tool which allows you to embed a virtual file system, accessible through preview1 APIs, directly into your wasm binaries.\\nThis way, we could prepare a single wasm artifact that contains both the `libpython` build and the custom glue code.\\n\\nThis approach turned out to be quite hacky though and after encountering several issues, we ultimately decided to go with **preopens**.\\nPreopens are another virtual file-system solution where you map an actual file-system directory to a virtual directory visible to a running Wasm instance.\\nThis means we\'ll need to prepare the `libpython` Wasm file on disk before running the instance but it was an acceptable solution.\\nWe also use preopens to provide some of the user submitted code to our custom python runtime.\\n\\nThe following rust snippet demonstrates the preopens looked like in use:\\n\\n```rust\\nfn init_Python_vm() -> Result