Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support for SQLite backend #130

Open
wants to merge 49 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
3fc305a
change mod init name
pranavmodx Mar 20, 2024
c3df22a
sqlite backend work so far
pranavmodx Mar 20, 2024
4375dbb
migration works
pranavmodx Mar 22, 2024
9bff228
init db
pranavmodx Mar 22, 2024
a1ba274
enqueuing works
pranavmodx Mar 25, 2024
7cfd550
notification - related work
pranavmodx Mar 25, 2024
ab982c6
job executes but db gets locked so not commited
pranavmodx Mar 26, 2024
ed54b2a
pending jobs works
pranavmodx Mar 27, 2024
3800287
db lock issue fixed, enqueuing works
pranavmodx Mar 27, 2024
5042ed4
comment few logs
pranavmodx Mar 27, 2024
66f6154
fix enqueue locking indefinitely issue
pranavmodx Mar 28, 2024
5e0dcb0
retrying works
pranavmodx Mar 28, 2024
e2d5d04
fix job chaining sync issue
pranavmodx Apr 2, 2024
3be1f0d
code cleanup
pranavmodx Apr 2, 2024
8b2be88
remove queue monitor and related fields
pranavmodx Apr 2, 2024
9fdb1cb
use ctx versions of sql apis
pranavmodx Apr 2, 2024
59201dd
code cleanup
pranavmodx Apr 2, 2024
e56993a
update and fix max_retries entries in db
pranavmodx Apr 4, 2024
a35e870
fix sqlite3 db path
pranavmodx Apr 5, 2024
82a13d9
make queueListenerChan buffer size configurable
pranavmodx Apr 8, 2024
00b8429
fix order of scanned columns in query
pranavmodx Apr 8, 2024
4dea09d
remove deadline from query
pranavmodx Apr 8, 2024
a575976
add in progress job status
pranavmodx Apr 9, 2024
fd7e5cc
add dbgs
pranavmodx Apr 9, 2024
330893c
add dbgs
pranavmodx Apr 9, 2024
ddbc6b1
add dbgs
pranavmodx Apr 9, 2024
4859e6e
close db when calling shutdown
pranavmodx Apr 11, 2024
66a7f1e
implement initFutureJobs
pranavmodx Apr 11, 2024
0ac8178
remove dbgs
pranavmodx Apr 17, 2024
faaa11d
create neoq_dead_jobs table; implement moveToDeadQueue
pranavmodx Apr 17, 2024
aeebbf2
implement StartCron
pranavmodx Apr 17, 2024
31de8d5
add basic test layout
pranavmodx Apr 19, 2024
9e7265d
add tests for multiple queues, cron
pranavmodx Apr 19, 2024
b710137
add tests job errors, future job scheduling
pranavmodx Apr 23, 2024
85e7409
change author back to original
pranavmodx May 5, 2024
f2e60ba
cleanup postgres backend logs
pranavmodx May 5, 2024
a159d2b
code cleanup; carve out updateJobToInProgress
pranavmodx May 5, 2024
b1c10a3
fix futureJobs concurrency issue
pranavmodx May 11, 2024
7857380
extract queueListenerChan for a given queue beforehand
pranavmodx May 11, 2024
bf6680e
add support for deadline, insert deadline and max_retries in enqueueJ…
pranavmodx May 12, 2024
f29e7b3
add tests - TestMultipleProcessors, TestMultipleCronNodes
pranavmodx May 12, 2024
d59a5f4
populate id and deadline to dead queue
pranavmodx May 12, 2024
a4fe135
support json payload; remove payload2
pranavmodx May 12, 2024
e2b4608
add test for duplicate job rejection
pranavmodx May 27, 2024
1ce892a
code cleanup
pranavmodx May 31, 2024
0691d74
extract db path by trimming prefix
pranavmodx Jun 7, 2024
91e3609
remove id in query for moveToDeadQueue
pranavmodx Jun 7, 2024
b818224
edit comment for initializeDB
pranavmodx Jun 8, 2024
e4da1db
replace TrimLeft with TrimPrefix
pranavmodx Jun 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions backends/sqlite/migrations/1_create_db_tables.up.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
CREATE TABLE neoq_jobs (
id integer primary key not null,
fingerprint text not null unique,
queue text,
status text default "new",
payload text,
retries integer default 0,
max_retries integer default 3,
run_after datetime default CURRENT_TIMESTAMP,
deadline datetime,
ran_at datetime,
created_at datetime default CURRENT_TIMESTAMP,
error text
);

CREATE TABLE neoq_dead_jobs (
id integer primary key not null,
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like your editor/autoformatter adds an extreanous indent at the start of these CREATE TABLE statements (same for the neoq_jobs table).

fingerprint text not null,
queue text,
status text default "failed",
payload text,
retries integer,
max_retries integer,
run_after datetime default CURRENT_TIMESTAMP,
deadline datetime,
ran_at datetime,
created_at datetime default CURRENT_TIMESTAMP,
error text
);
Loading