SB-24793 - Assessment archival Job Implementation #452
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…entation and testcase
…entation and testcase
…entation and testcase
…b implementation and testcase" This reverts commit 4ca0b4c.
…entation and testcase
…d enabling the code coverage
…atchId or date to archive the data. if the date has not provided and archiveForLastWeek = true then it will by default it will archive the data for the last week records
…atchId or date to archive the data. if the date has not provided and archiveForLastWeek = true then it will by default it will archive the data for the last week records
…oducts into assessment-archival-latest
…te case for s3 to handle.
maheshkumargangula
requested changes
Oct 22, 2021
data-products/src/main/scala/org/sunbird/analytics/exhaust/util/ExhaustUtil.scala
Outdated
Show resolved
Hide resolved
data-products/src/main/scala/org/sunbird/analytics/job/report/AssessmentArchivalJob.scala
Show resolved
Hide resolved
…hanges - Updated the test cases.
…hanges - added the description
maheshkumargangula
approved these changes
Oct 22, 2021
SanthoshVasabhaktula
requested changes
Oct 25, 2021
| val format = fileFormat.getOrElse("csv.gz") | ||
| val batchId = filteredBlobFields.getOrElse("batchId", "*") | ||
| val year = filteredBlobFields.getOrElse("year", "*") | ||
| val weekNumb = filteredBlobFields.getOrElse("weekNum", "*") |
There was a problem hiding this comment.
Numb? Keep it Num or Number
Comment on lines
19
to
31
| val url = store match { | ||
| case "local" => | ||
| filePath + s"${batchId}/${year}-${weekNumb}-*.${format}" | ||
| // $COVERAGE-OFF$ for azure testing | ||
| case "azure" => | ||
| val key = AppConf.getConfig("azure_storage_key") | ||
| val file = s"${filePath}${batchId}/${year}-${weekNumb}-*.${format}" | ||
| s"wasb://$bucket@$key.blob.core.windows.net/$file" | ||
| case "s3" => | ||
| //TODO - Need to support the S3 As well. | ||
| throw new Exception("s3 is currently not supported.") | ||
| // $COVERAGE-ON$ | ||
| } |
There was a problem hiding this comment.
Move this to a CommonUtil method and add support for S3 and GCP.
There was a problem hiding this comment.
We have added the support for S3. Can we take up GCP support as an enhacement in next release?
Dependent Core PR: https://github.com/project-sunbird/sunbird-analytics-core/pull/116/files
|
Kudos, SonarCloud Quality Gate passed! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Type of change
Please choose appropriate options.
How Has This Been Tested?
Please describe the tests that you ran to verify your changes in the below checkboxes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Test Configuration:
Checklist: