Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: postgres datastore implementation - part 2 #205

Merged
merged 29 commits into from
Dec 21, 2024
Merged

feat: postgres datastore implementation - part 2 #205

merged 29 commits into from
Dec 21, 2024

Conversation

mfiedorowicz
Copy link
Member

@mfiedorowicz mfiedorowicz commented Dec 17, 2024 •
edited
Loading

  • consolidate ingestion logs and change sets repositories into single one (as related to each other)
  • remove redis migrations
  • remove writing ingestion logs to redis
  • refactor postgres schemas
  • implement postgres repository
Screenshot 2024-12-20 at 16 43 38

Integration tests:
Screenshot 2024-12-20 at 16 32 29

mfiedorowicz and others added 17 commits December 4, 2024 12:14
@mfiedorowicz
feat: change postgres PK type
be2497c 
use INTEGER PRIMARY KEY GENERATED ALWAYS AS IDENTITY instead of SERIAL
ref: https://wiki.postgresql.org/wiki/Don%27t_Do_This#Don.27t_use_serial

Signed-off-by: Michal Fiedorowicz <[email protected]>
@mfiedorowicz
feat: reconciler - implement postgres.CreateIngestionLog
0199542 
Signed-off-by: Michal Fiedorowicz <[email protected]>
@mfiedorowicz
feat: change postgres char to varchar
da90760 
ref: https://wiki.postgresql.org/wiki/Don%27t_Do_This#Don.27t_use_char.28n.29
Signed-off-by: Michal Fiedorowicz <[email protected]>
@mfiedorowicz
chore: SQL format
b469c38 
Signed-off-by: Michal Fiedorowicz <[email protected]>
@mfiedorowicz
Merge branch 'develop' into postgres-db-store-pt-2
6e174f6
@mfiedorowicz
feat: update postgres schemas and queries
013e630 
generate code with sqlc and mockery

Signed-off-by: Michal Fiedorowicz <[email protected]>
@mfiedorowicz
chore: regenerate mocks
ad4918c 
Signed-off-by: Michal Fiedorowicz <[email protected]>
@mfiedorowicz
feat: add branch ID to change set
8bcc199 
Signed-off-by: Michal Fiedorowicz <[email protected]>
@mfiedorowicz
feat: implement storage of ingestion logs and change sets in postgres…
413b6c2 
… database

Signed-off-by: Michal Fiedorowicz <[email protected]>
@mfiedorowicz
feat: retrieve ingestion logs from postgres
df235eb 
Signed-off-by: Michal Fiedorowicz <[email protected]>
@mfiedorowicz
chore: adjust unit tests
a72bdc4 
Signed-off-by: Michal Fiedorowicz <[email protected]>
@dependabot @mfiedorowicz
chore(deps): bump golang.org/x/crypto in /diode-server (#200)
35b87cd 
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.28.0 to 0.31.0.
- [Commits](golang/crypto@v0.28.0...v0.31.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@ltucker @mfiedorowicz
chore: docker compose development overrides (#201)
19897cc
@ltucker @mfiedorowicz
feat: add branch support (#202)
57d9c37 
Co-authored-by: Michal Fiedorowicz <[email protected]>
@ltucker @mfiedorowicz
feat: ActionIngestionLog rpc (#203)
9a3b235
@ltucker @mfiedorowicz
chore: enable gofumpt lint (#204)
78d16a0
@mfiedorowicz
chore: gofumpt
e3012cc 
Signed-off-by: Michal Fiedorowicz <[email protected]>
@mfiedorowicz mfiedorowicz self-assigned this Dec 17, 2024
@mfiedorowicz
chore: remove redis migrations
c12f1d1 
Signed-off-by: Michal Fiedorowicz <[email protected]>
@mfiedorowicz
chore: remove writing ingestion log to redis
c2048dc 
Signed-off-by: Michal Fiedorowicz <[email protected]>
@mfiedorowicz
fix: adjust unit tests
5269c76 
Signed-off-by: Michal Fiedorowicz <[email protected]>
@mfiedorowicz
Merge branch 'feat-netbox-assurance' into postgres-db-store-pt-2
7228a54 
# Conflicts:
#	diode-server/reconciler/ingestion_processor.go
#	diode-server/reconciler/logs_retriever.go
@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 17, 2024 •
edited
Loading

Go test coverage

STATUS ELAPSED PACKAGE COVER PASS FAIL SKIP
🟢 PASS 0.00s github.com/netboxlabs/diode/diode-server/dbstore/postgres 0.0% 0 0 0
🟢 PASS 0.01s github.com/netboxlabs/diode/diode-server/gen/dbstore/postgres 0.0% 0 0 0
🟢 PASS 1.23s github.com/netboxlabs/diode/diode-server/ingester 71.2% 9 0 0
🟢 PASS 1.04s github.com/netboxlabs/diode/diode-server/migrator 68.2% 4 0 0
🟢 PASS 0.01s github.com/netboxlabs/diode/diode-server/netbox 0.0% 0 0 0
🟢 PASS 1.60s github.com/netboxlabs/diode/diode-server/netboxdiodeplugin 78.9% 42 0 0
🟢 PASS 1.46s github.com/netboxlabs/diode/diode-server/reconciler 83.5% 49 0 0
🟢 PASS 1.10s github.com/netboxlabs/diode/diode-server/reconciler/applier 88.9% 1 0 0
🟢 PASS 0.01s github.com/netboxlabs/diode/diode-server/reconciler/changeset 0.0% 0 0 0
🟢 PASS 1.31s github.com/netboxlabs/diode/diode-server/reconciler/differ 80.9% 81 0 0
🟢 PASS 1.02s github.com/netboxlabs/diode/diode-server/server 85.7% 14 0 0
🟢 PASS 1.01s github.com/netboxlabs/diode/diode-server/version 100.0% 1 0 0

Total coverage: 75.4%

@mfiedorowicz mfiedorowicz changed the title [WIP] feat: postgres datastore implementation - part 2 feat: postgres datastore implementation - part 2 Dec 20, 2024
@mfiedorowicz mfiedorowicz marked this pull request as ready for review December 20, 2024 16:39
@leoparente
Copy link
Contributor

[Question] did you run go mod tidy after changes?

@mfiedorowicz
chore: go mod tidy
0ad52e5 
Signed-off-by: Michal Fiedorowicz <[email protected]>
@mfiedorowicz
feat: postgres tables with hybrid IDs
4145fed 
id integer for PK
external_id UUID for external exposure

Signed-off-by: Michal Fiedorowicz <[email protected]>
@ltucker
Copy link
Contributor

ltucker commented Dec 20, 2024 •
edited
Loading

I don't see any showstoppers here, I've been using / looking over this branch for the last few days without issue.
Especially in the interest of not making this any larger, I'm fine with this on the feature branch and addressing additional things with followups.

@mfiedorowicz mfiedorowicz merged commit 3f46f8a into feat-netbox-assurance Dec 21, 2024
3 checks passed
@mfiedorowicz mfiedorowicz deleted the postgres-db-store-pt-2 branch December 21, 2024 10:46
mfiedorowicz added a commit that referenced this pull request Dec 23, 2024
@mfiedorowicz @dependabot @ltucker
feat: postgres datastore implementation - part 2 (#205)
99c7000 
Signed-off-by: Michal Fiedorowicz <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Luke Tucker <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@leoparente leoparente leoparente left review comments

@ltucker ltucker ltucker approved these changes

Assignees

@mfiedorowicz mfiedorowicz

Labels
dependencies diode-ingester diode-proto diode-reconciler diode-server go internal
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mfiedorowicz @leoparente @ltucker