Feat message management #378
Conversation
☂️ Python Coverage
Overall Coverage
New FilesNo new covered files... Modified FilesNo covered modified files...
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #378 +/- ##
=======================================
Coverage 94.04% 94.04%
=======================================
Files 259 259
Lines 15841 15932 +91
=======================================
+ Hits 14898 14984 +86
- Misses 943 948 +5 ☔ View full report in Codecov by Sentry. |
|
Hey @danangmassandy I might need your assistance on testing this, as i realised the farm crop parquet files are sent to an S3 bucket. |
…test for Codecov coverage
| @@ -465,13 +473,62 @@ def extract_csv_output(self): | |||
|
|
|||
| return file_path | |||
|
|
|||
| def filter_message_output(self): | |||
There was a problem hiding this comment.
This should be done in process_farm_registry_data, right before writing to final parquet file.
There was a problem hiding this comment.
also need to add a config from Preferences whether to run the message checks or not.
| :rtype: pd.Timestamp or None | ||
| """ | ||
| # Read historical messages | ||
| historical_data = read_grid_crop_data( |
There was a problem hiding this comment.
This should read to parquet file in minio with date filter, similar to
https://github.com/kartoza/tomorrownow_gap/blob/main/django_project/dcas/queries.py#L287
| if pd.isna(message_code): | ||
| continue # Skip empty messages | ||
|
|
||
| last_sent_date = get_last_message_date( |
There was a problem hiding this comment.
can we query once to get all messages for given a farm from min_allowed_date? then do the check against this list. This is to avoid too many query by each code and farm.
WIP
Fix #308