BFD-3854: IDR pipeline POC #2548
Open
+625
−7
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aschey-forpeople
requested review from
mjburling,
dondevun and
malessi
as code owners
| from pydantic import BaseModel | ||
|
|
||
|
|
||
| class IdrBeneficiary(BaseModel): |
There was a problem hiding this comment.
pydantic's model classes make it easy to work with typed data coming out of the database. It does introduce some overhead due to runtime type checking though, so we may want to tweak this in the future if the performance hit is large.
| class SnowflakeExtractor(Extractor): | ||
| def __init__(self, batch_size: int): | ||
| super().__init__() | ||
| self.conn = snowflake.connector.connect(user="", password="", account="") |
There was a problem hiding this comment.
This is untested for now because we don't have Snowflake access. This will be used for prod data and the PostgresExtractor above can be used for test data.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
JIRA Ticket:
BFD-3854
What Does This PR Do?
Creates the initial POC for the IDR pipeline. The Snowflake connector is not fully set up yet since we don't have access.
In the meantime, we can mock the IDR schema directly in Postgres for testing purposes. I've currently included a small subset of the data and simplified queries so we can quickly make changes without dealing with the large number of columns that we will eventually need to handle.
The current approach used here is to stream data out of IDR in batches and load them into Postgres using
COPY. It's possible that this approach will change once we are able to do performance testing with real data.There is a simple integration test included to verify the data load is working on a basic level.
What Should Reviewers Watch For?
If you're reviewing this PR, please check for these things in particular:
What Security Implications Does This PR Have?
Please indicate if this PR does any of the following:
Adds any new software dependencies
Modifies any security controls
Adds new transmission or storage of data
Any other changes that could possibly affect security?
I have considered the above security implications as it relates to this PR. (If one or more of the above apply, it cannot be merged without the ISSO or team security engineer's (
@sb-benohe) approval.)Validation
Have you fully verified and tested these changes? Is the acceptance criteria met? Please provide reproducible testing instructions, code snippets, or screenshots as applicable.
Tests included.