QA script for irradiance data for TA2/3 #756
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lboeman
reviewed
Dec 1, 2021
Comment on lines
+590
to
+597
| if not validated: | ||
| logger.info('validation appears hung. reposting data') | ||
| session.post_observation_values(o.observation_id, values) | ||
| validated = wait_for_validation(session, o, values) | ||
| if not validated: | ||
| logger.info('validation appears hung. reposting data') | ||
| session.post_observation_values(o.observation_id, values) | ||
| validated = wait_for_validation(session, o, values) |
There was a problem hiding this comment.
Do you want to do this twice? Seems like wait_for_validation does some amount of retrying?
There was a problem hiding this comment.
wait_for_validation retries fetching data from API, which helps with a temporary hang or long queue. But that doesn't help if the job hangs indefinitely. So this re-posts the data. Not sure how many times I should try it.
irradiance_qa/irradiance_qa.py
Outdated
| variable = o.variable.lower() | ||
| _data_to_post = data_to_post[site_name][variable] | ||
| # Split into chunks to stay under API upload limit. | ||
| grouped_data = _data_to_post.groupby(lambda x: (x.year, x.month)) |
There was a problem hiding this comment.
You might even split this into weeks? just so that validation jobs are quick.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This script applies extra quality control to the data used in the TA2/3 evaluations. It downloads the data, processes it, and posts it back to the Arbiter. Posting to the "official" observations requires the reference account. Script will create observations in regular users organization.
@cwhanse wrote the original data processing scripts for each site. I just refactored it into this script.
Not sure where to put this, but I want to finally make the script visible in some way or another.
python irradiance_qa.py --helpdocs/source/api.rstfor API changes.docs/source/whatsnewfor all changes. Includes link to the GitHub Issue with:issue:`num`or this Pull Request with:pull:`num`. Includes contributor name and/or GitHub username (link with:ghuser:`user`).