scheduler kata

Grinding the api-queue-worker pattern for fun

Motivation

• practice PostgreSQL
• practice PostgREST
• practice benchmarking
• practice Rust

Requirements

Expose an API that can:

• Create a task of a specific type and execution time, returning the task's ID
• Show a list of tasks, filterable by their state (whatever states you define) and/or their task type
• Show a task based on its ID
• Delete a task based on its ID
• The tasks must be persisted into some external data store (your choice).
• Process each task only once and only at/after their specified execution time.
• Support running multiple instances of your code in parallel.

Additional materials

• Queues in Postgres | Postgres.FM 042 | #PostgreSQL #Postgres podcast on YouTube
• Choose Postgres queue technology by Adriano Caloiaro
• PGMQ A lightweight message queue. Like AWS SQS and RSMQ but on Postgres
• River Atomic, transaction-safe, robust job queueing for Go applications. Backed by PostgreSQL and built to scale

Iterations log

• v0 quick and dirty implementation where main focus was for exploring axum framework and rust multithreading

• v1 going down the PostgreSQL rabbit hole and some sneak-peaking at this codebases:

  • https://github.com/windmill-labs/windmill
  • https://github.com/svix/svix-webhooks

• v2 mostly reusing v1, incorporating aide for OAS, bundling scheduler, queue and worker

• v3 telemetry, benchmarking and performance improvements

• v4 new approach inspired by PGMQ but also other additional materials. Some learnings:

  • long polling vs short polling trade-offs