Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rebase PR #1

Open
wants to merge 121 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
121 commits
Select commit Hold shift + click to select a range
d7aeb91
fixed urlbases
sepehrkrz Oct 11, 2023
a0a06e4
fixed urlbases
sepehrkrz Oct 11, 2023
30f0270
fixed urlbases
sepehrkrz Oct 11, 2023
58cd17b
fixed urlbases
sepehrkrz Oct 11, 2023
4e65991
Delete .ipynb_checkpoints directory
sepehrkrz Oct 11, 2023
0aec7c6
Delete nwmurl/.ipynb_checkpoints directory
sepehrkrz Oct 11, 2023
0f572e6
Update setup.py
sepehrkrz Oct 27, 2023
a2d7a3e
Update README.md
sepehrkrz Oct 27, 2023
84b3cf2
Update README.md
sepehrkrz Oct 27, 2023
5801969
Merge pull request #2 from CIROH-UA/main
sepehrkrz Oct 27, 2023
f0ddf63
Update urlgennwm.py
sepehrkrz Oct 27, 2023
b4f0a14
0.0.9
sepehrkrz Oct 29, 2023
21a3fb9
Merge pull request #18 from sepehrkrz/main
sepehrkrz Oct 29, 2023
610a5de
Create python-publish.yml
sepehrkrz Oct 29, 2023
892fded
Merge pull request #3 from CIROH-UA/main
sepehrkrz Oct 29, 2023
5d0054c
V 0.1.0
sepehrkrz Oct 29, 2023
6e3d659
Merge pull request #19 from sepehrkrz/main
sepehrkrz Oct 29, 2023
0faa946
Update python-publish.yml
sepehrkrz Oct 29, 2023
641b839
Merge pull request #4 from CIROH-UA/main
sepehrkrz Oct 29, 2023
401088f
Update python-publish.yml
sepehrkrz Oct 29, 2023
8d33a61
Merge pull request #5 from CIROH-UA/main
sepehrkrz Oct 29, 2023
c9ffc0c
Update python-publish.yml supertools
sepehrkrz Oct 29, 2023
5c3ee16
Merge pull request #6 from CIROH-UA/main
sepehrkrz Oct 29, 2023
d3dc83b
Update python-publish.yml setuptools!!!
sepehrkrz Oct 29, 2023
50c936c
Merge pull request #7 from CIROH-UA/main
sepehrkrz Oct 29, 2023
198e6cb
Update python-publish.yml wheel
sepehrkrz Oct 29, 2023
a98a2bf
Merge pull request #8 from CIROH-UA/main
sepehrkrz Oct 29, 2023
2f4bced
Update python-publish.yml new token
sepehrkrz Oct 29, 2023
556d910
Update python-publish.yml git token
sepehrkrz Oct 29, 2023
731bad8
Update python-publish.yml nwmurlgit token
sepehrkrz Oct 29, 2023
4750485
Merge pull request #9 from CIROH-UA/main
sepehrkrz Oct 29, 2023
7834195
Update python-publish.yml new login method
sepehrkrz Oct 29, 2023
994018f
Merge pull request #10 from CIROH-UA/main
sepehrkrz Oct 29, 2023
108c50a
new python-publish.yml
sepehrkrz Oct 29, 2023
4378e8c
Merge pull request #11 from CIROH-UA/main
sepehrkrz Oct 29, 2023
f70bbb4
Update setup.py
sepehrkrz Oct 29, 2023
0dc2dcc
Merge pull request #12 from CIROH-UA/main
sepehrkrz Oct 29, 2023
abdc9bb
Update python-publish.yml new yaml
sepehrkrz Oct 29, 2023
d72a8e7
Merge pull request #13 from CIROH-UA/main
sepehrkrz Oct 29, 2023
22f18e4
Update python-publish.yml setuptools wheel
sepehrkrz Oct 29, 2023
bb7a0ae
Merge pull request #14 from CIROH-UA/main
sepehrkrz Oct 29, 2023
66d32bb
Update python-publish.yml
sepehrkrz Oct 29, 2023
4d9b7e5
Update python-publish.yml updated token
sepehrkrz Oct 29, 2023
980b815
Merge pull request #15 from CIROH-UA/main
sepehrkrz Oct 29, 2023
5263725
Update python-publish.yml dist*
sepehrkrz Oct 29, 2023
2d0207c
Merge pull request #16 from CIROH-UA/main
sepehrkrz Oct 29, 2023
f5c8c9a
Update python-publish.yml new auth
sepehrkrz Oct 29, 2023
3ca6a50
Merge pull request #17 from CIROH-UA/main
sepehrkrz Oct 29, 2023
3ce9439
Update python-publish.yml new pass
sepehrkrz Oct 29, 2023
1f320db
Merge pull request #18 from CIROH-UA/main
sepehrkrz Oct 29, 2023
2aea138
Update python-publish.yml new method auth
sepehrkrz Oct 29, 2023
71ce12a
Merge pull request #19 from CIROH-UA/main
sepehrkrz Oct 29, 2023
06bf7da
Update python-publish.yml secrets added
sepehrkrz Oct 29, 2023
d15f277
Merge pull request #20 from CIROH-UA/main
sepehrkrz Oct 29, 2023
f5e1ac5
Update python-publish.yml idk
sepehrkrz Oct 29, 2023
f334a3d
Update python-publish.yml old method
sepehrkrz Oct 29, 2023
f5171af
Merge pull request #21 from CIROH-UA/main
sepehrkrz Oct 29, 2023
834ae32
Update python-publish.yml new2 script action
sepehrkrz Oct 29, 2023
96530f9
Merge pull request #22 from CIROH-UA/main
sepehrkrz Oct 29, 2023
471ad4f
Update python-publish.yml actionv2
sepehrkrz Oct 29, 2023
e79772b
Merge pull request #23 from CIROH-UA/main
sepehrkrz Oct 29, 2023
47b493f
Update python-publish.yml action v1 revert
sepehrkrz Oct 29, 2023
4b6407b
Merge pull request #24 from CIROH-UA/main
sepehrkrz Oct 29, 2023
9b30b41
Update python-publish.yml wheel
sepehrkrz Oct 29, 2023
e7d8e11
Merge pull request #25 from CIROH-UA/main
sepehrkrz Oct 29, 2023
fce9033
Update python-publish.yml asda
sepehrkrz Oct 29, 2023
157a85e
Merge pull request #26 from CIROH-UA/main
sepehrkrz Oct 29, 2023
9b70bb2
Update python-publish.yml twine update
sepehrkrz Oct 30, 2023
e89b02d
Merge pull request #27 from CIROH-UA/main
sepehrkrz Oct 30, 2023
4055cb7
Update python-publish.yml print test
sepehrkrz Oct 30, 2023
6c4f57b
Merge pull request #28 from CIROH-UA/main
sepehrkrz Oct 30, 2023
8f66191
Update python-publish.yml list secrets
sepehrkrz Oct 30, 2023
43000c4
Merge pull request #29 from CIROH-UA/main
sepehrkrz Oct 30, 2023
1672a7d
Update python-publish.yml print repo
sepehrkrz Oct 30, 2023
f64a820
Merge pull request #30 from CIROH-UA/main
sepehrkrz Oct 30, 2023
99dcc1d
Update python-publish.yml print sepehr
sepehrkrz Oct 30, 2023
3b6b28b
Merge pull request #31 from CIROH-UA/main
sepehrkrz Oct 30, 2023
55ae1c9
Update python-publish.yml var and sec
sepehrkrz Oct 30, 2023
b565d81
Merge pull request #32 from CIROH-UA/main
sepehrkrz Oct 30, 2023
1976d47
Update python-publish.yml prod
sepehrkrz Oct 30, 2023
72ccb7e
Merge pull request #33 from CIROH-UA/main
sepehrkrz Oct 30, 2023
b742d48
Update python-publish.yml secvars new
sepehrkrz Oct 30, 2023
266529b
Merge pull request #34 from CIROH-UA/main
sepehrkrz Oct 30, 2023
90e51ae
Create secrets.yaml
sepehrkrz Oct 30, 2023
4a55852
Update secrets.yaml
sepehrkrz Oct 30, 2023
00a5b72
Update python-publish.yml
sepehrkrz Oct 30, 2023
71d251f
Update python-publish.yml
sepehrkrz Oct 30, 2023
12a6b4a
Delete .github/workflows directory
sepehrkrz Oct 30, 2023
aa0cb8e
Create python-publish.yml
sepehrkrz Oct 30, 2023
4b9ee1f
Merge pull request #36 from CIROH-UA/main
sepehrkrz Oct 30, 2023
b637d84
Update python-publish.yml
sepehrkrz Oct 30, 2023
9938abf
Merge pull request #37 from CIROH-UA/main
sepehrkrz Oct 30, 2023
a292771
Merge pull request #20 from sepehrkrz/main
sepehrkrz Oct 31, 2023
1221db9
Update python-publish.yml no prod
sepehrkrz Oct 31, 2023
e4cbb02
Merge pull request #38 from CIROH-UA/main
sepehrkrz Oct 31, 2023
ab0b9c8
Merge pull request #21 from sepehrkrz/main
sepehrkrz Oct 31, 2023
b1707c9
Added Retrospective data and save to file option
sepehrkrz Oct 31, 2023
4cbf814
Merge pull request #22 from sepehrkrz/main
sepehrkrz Oct 31, 2023
5594c8b
Update python-publish.yml
sepehrkrz Oct 31, 2023
5bee59b
Support for Retrospective data
sepehrkrz Oct 31, 2023
8276882
Merge pull request #23 from sepehrkrz/main
sepehrkrz Oct 31, 2023
5d85268
Update python-publish.yml
sepehrkrz Oct 31, 2023
db1bf14
Update setup.py
sepehrkrz Oct 31, 2023
1e465bf
Merge pull request #24 from sepehrkrz/main
sepehrkrz Oct 31, 2023
296f8de
Update python-publish.yml
sepehrkrz Oct 31, 2023
341df14
Merge pull request #25 from sepehrkrz/main
sepehrkrz Oct 31, 2023
a2654ea
Update setup.py
sepehrkrz Oct 31, 2023
3781fb7
Merge pull request #26 from sepehrkrz/main
sepehrkrz Oct 31, 2023
9309a63
Update python-publish.yml
sepehrkrz Oct 31, 2023
a1d5af6
Update setup.py
sepehrkrz Oct 31, 2023
f1ccdb2
Merge pull request #27 from sepehrkrz/main
sepehrkrz Oct 31, 2023
3a2fbc6
Update python-publish.yml
sepehrkrz Oct 31, 2023
59348ae
Merge pull request #28 from sepehrkrz/main
sepehrkrz Oct 31, 2023
a4bd8f6
Update __init__.py
sepehrkrz Oct 31, 2023
8a03aa3
Update setup.py
sepehrkrz Oct 31, 2023
7176f16
Merge pull request #29 from sepehrkrz/main
sepehrkrz Oct 31, 2023
430a44b
Delete nwmurl/.ipynb_checkpoints directory
sepehrkrz Oct 31, 2023
ee4da98
Corrected the bug in the url and created specific test cases (#17)
RohanSunkarapalli Oct 31, 2023
cd6fd25
add tests
Oct 31, 2023
df0c911
increment version number to 1.0.7
Oct 31, 2023
e75c2d0
update files to new function name
Oct 31, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions .github/workflows/python-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
name: Publish nwmurl to PyPi

on:
push:
branches:
- main

jobs:
deploy:
runs-on: ubuntu-latest
environment: production
steps:
- name: var APP_ID
run: echo " My App ID is ${{ vars.APP_KEY }}"

- name: SECRET API_KEY
run: echo " My secret ID is ${{ secrets.PYPI_API_TOKEN }}"

- name: Checkout code
uses: actions/checkout@v2

- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.x

- name: Print Repository URL
run: |
REPO_URL="https://github.com/${{ github.repository }}"
echo "Repository URL: $REPO_URL"

- name: Install dependencies
run: pip install setuptools wheel
- name: List Secrets
run: |
echo "Available secrets:"
env
- name: Build and deploy package
run: |
python setup.py sdist bdist_wheel
pip install twine==4.0.2
echo "PYPI_API_TOKEN: ${{ secrets.SEPEHR }}" # Print the API token
twine upload -u '__token__' -p ${{ secrets.PYPI_API_TOKEN }} dist/*
41 changes: 41 additions & 0 deletions .github/workflows/secrets.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: Secrets and Environment Variables

on:
workflow_dispatch

jobs:
deploy:
runs-on: ubuntu-latest
environment: production
steps:
- name: var APP_ID
run: echo " My App ID is ${{ vars.APP_KEY }}"

- name: SECRET API_KEY
run: echo " My secret ID is ${{ secrets.PYPI_API_TOKEN }}"

- name: Checkout code
uses: actions/checkout@v2

- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.x

- name: Print Repository URL
run: |
REPO_URL="https://github.com/${{ github.repository }}"
echo "Repository URL: $REPO_URL"

- name: Install dependencies
run: pip install setuptools wheel
- name: List Secrets
run: |
echo "Available secrets:"
env
- name: Build and deploy package
run: |
python setup.py sdist bdist_wheel
pip install twine==4.0.2
echo "PYPI_API_TOKEN: ${{ secrets.SEPEHR }}" # Print the API token
twine upload -u '__token__' -p ${{ secrets.PYPI_API_TOKEN }} dist/*
64 changes: 0 additions & 64 deletions .ipynb_checkpoints/README-checkpoint.md

This file was deleted.

97 changes: 91 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@

# nwmurl

This library contains utility functions to generate National Water Model data URLs
Expand All @@ -9,7 +10,7 @@ Developed by CIROH 2023

2. The code will generate a list of JSON header URLs tailored to your specified parameters using the `generate_urls` function.

## Customize Your Data Retrieval
## Customize Your Data Retrieval for Operational Dataset
- `start_date`: A string representing the starting date in the format \"YYYYMMDDHHMM".
- `end_date`: A string representing the ending date in the same format.
- `fcst_cycle`: A list of integers specifying forecast cycle numbers, e.g., `[0, 1, 2, 3, 4]`. These cycles represent specific points in time for which URLs will be generated.
Expand Down Expand Up @@ -47,18 +48,102 @@ Developed by CIROH 2023
- `8`: "s3://noaa-nwm-pds/".
- `9`: "https://ciroh-nwm-zarr-copy.s3.amazonaws.com/national-water-model/".
- `meminput `: An integer representing the ensemble member designation ranging from 0 to 7
- `write_to_file`: A Boolean variable that saves the output urls into a .txt file if set 'True'


## Customize Your Data Retrieval for Retrospective Dataset
- `start_date`: A string representing the starting date in the format \"YYYYMMDDHHMM".
- `end_date`: A string representing the ending date in the same format.
- `urlbaseinput `: An integer representing the NWM dataset. Available options include:
- `1`: "https://noaa-nwm-retrospective-2-1-pds.s3.amazonaws.com/".
- `2`: "s3://noaa-nwm-retrospective-2-1-pds/model_output/".
- `selectet_object_type`: An integer representing the object type. Available options include:
- `1` for forcing data
- `2` for model_output

- `Selectet_var_types`: An integer or string representing the variable of interest within the NWM data. Available options include:
- `1`: ".CHRTOUT_DOMAIN1.comp"
- `2`: ".GWOUT_DOMAIN1.comp"
- `3`: ".LAKEOUT_DOMAIN1.comp"
- `4`: ".LDASOUT_DOMAIN1.comp"
- `5`: ".RTOUT_DOMAIN1.comp"
- `6`: ".LDASIN_DOMAIN1.comp"
- `write_to_file`: A Boolean A Boolean variable that saves the output urls into a .txt file if set `True`
## Examples of how to use



1. Clone the repository to your local machine:
1. For Operational dataset:
```
import nwmurl

start_date = "202201120000"
end_date = "202201130000"
fcst_cycle = [0,8]
lead_time = [1,18]
varinput = 1
geoinput = 1
runinput = 1
urlbaseinput = 2
meminput = 1
write_to_file = False

file_list = nwmurl.generate_urls_operational(
start_date, end_date, fcst_cycle,
lead_time,
varinput,
geoinput,
runinput,
urlbaseinput,
meminput,
write_to_file
)
```

2. For Retrospective dataset:
```
git clone https://github.com/CIROH-UA/data_access_example.git
import nwmurl

start_date = "200701010000"
end_date = "200701030800"
urlbaseinput = 2
selected_var_types = [1, 2]
selected_object_types = [1]
write_to_file = True

file_list = nwmurl.generate_urls_retro(
start_date,
end_date,
urlbaseinput,
selected_object_types,
selected_var_types,
write_to_file
)
```

2. Navigate to the repository folder:
## How to Contribute

We welcome contributions to nwmurl! To contribute to the development of this library, please follow these steps:

1. Fork the repository on GitHub.

2. Clone your fork to your local machine:`

`git clone https://github.com/your-username/nwmurl.git`

3. Create a new branch for your contribution:`

`git checkout -b feature/your-feature-name`


4. Make your code changes and improvements.

5. Before submitting a pull request, make sure to update the package version in `setup.py` if necessary.

6. Commit your changes with descriptive commit messages.

7. Push your changes to your fork:``


8. Open a pull request on the main repository, describing your changes and why they should be merged.

3. Launch the Jupyter Notebook to execute the example
We appreciate your contributions and will review your pull request as soon as possible. Thank you for helping improve nwmurl!
Loading