Updated RDA addresses and dataset names #790

kdraeder · 2025-01-08T16:38:20Z

Description:

NCAR's Research Data Archive changed their dataset naming convention,
so all the references in DART became outdated.
The "local" (/glade) versions were also moved to a new location (/glade/campaign) when derecho replaced cheyenne.

Did not update
./CHANGELOG.rst
./_build/html/CHANGELOG.html
./_build/html/_sources/CHANGELOG.rst.txt

These substitutions were made:
sed \
-e "s#glade/collections#glade/campaign/collections#" \
-e "s#edu/ds/#edu/datasets/ds#g" \
-e "s#ds345.0#d345000#g" \
-e "s#ds199.1#d199001#g" \
-e "s#ds090.0#d090000#g" \
-e "s#ds337.0#d337000#g" \
-e "s#ds285.0#d285000#g" \

@hkershaw-brown I updated some .html files too, and would like confirmation that that was
the right thing to do.

@braczka Please check some of the CLM files, but I think that you don't need check all of them.

I did not add DOIs for these datasets, which would be more robust, but more work.

Fixes issue

#755

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update

Documentation changes needed?

My change requires a change to the documentation.
- I have updated the documentation accordingly.

Tests

I checked the changes in
./models/POP/shell_scripts/cesm2_1/user_datm.streams.txt.CPLHISTForcing.Solar_template

I did not test any of the testable files.

Checklist for merging

Updated changelog entry
Documentation updated
Update conf.py

Checklist for release

Merge into main
Create release from the main branch with appropriate tag
Delete feature-branch

Testing Datasets

Dataset needed for testing available upon request
Dataset download instructions included
No dataset needed

Did not update ./CHANGELOG.rst ./_build/html/CHANGELOG.html ./_build/html/_sources/CHANGELOG.rst.txt These substitutions were made: sed \ -e "s#glade\/collections#glade\/campaign\/collections#" \ -e "s#edu\/ds\/#edu\/datasets\/ds#g" \ -e "s#ds345.0#d345000#g" \ -e "s#ds199.1#d199001#g" \ -e "s#ds090.0#d090000#g" \ -e "s#ds337.0#d337000#g" \ -e "s#ds285.0#d285000#g" \

hkershaw-brown · 2025-01-08T17:01:44Z

@hkershaw-brown I updated some .html files too, and would like confirmation that that was
the right thing to do.

Does not look like any html files in this pull request. Do you mean the .rst files?

kdraeder · 2025-01-08T17:25:47Z

@hkershaw-brown I updated some .html files too, and would like confirmation that that was
the right thing to do.

Does not look like any html files in this pull request. Do you mean the .rst files?

Apparently the files in _build/html/... are not under git control. I'm guessing those are exclude intentionally.

hkershaw-brown · 2025-01-08T17:29:35Z

Apparently the files in _build/html/... are not under git control. I'm guessing those are exclude intentionally.

Yes those are files from the local build of the documentation.

hkershaw-brown · 2025-01-08T18:07:59Z

How about for the .rst having one reference file for all the data sets so they only need to be changes in one place (when the rda links change, or change to dois).

.. CAM6rean :: https://rda.ucar.edu/datasets/d345000/

.. CAM4rean :: https://rda.ucar.edu/datasets/whatever

.. NCEPrda :: https://rda.ucar.edu/datasets/blahblah

kdraeder · 2025-01-08T19:15:24Z

How about for the .rst having one reference file for all the data sets so they only need to be changes in one place (when the rda links change, or change to dois).

.. CAM6rean :: https://rda.ucar.edu/datasets/d345000/

.. CAM4rean :: https://rda.ucar.edu/datasets/whatever

.. NCEPrda :: https://rda.ucar.edu/datasets/blahblah

That would tidy some of the files nicely. But most of the files are stream files.
I'll work on it.

hkershaw-brown · 2025-01-08T19:25:00Z

Add this to conf.py them you can have a guide/references.rst (or whatever you want to call it) included in every file. This means you can use CAM6rean_ anywhere in the docs.

 89 # include references
 90 with open(os.path.join(os.path.dirname(__file__), 'guide/references.rst')) as f:
 91     references_content = f.read()
 92 
 93 rst_prolog = references_content

kdraeder · 2025-01-17T21:23:10Z

In my conf.py I needed to activate the os package to make the references lines work.

I also added .. include:: lines into the rst files that need to use the references.rst file,
but Sphinx(?) is changing the path names in a way that I haven't parsed yet.
On my mac the full path name I give it gets converted to
../Manhattan/Users/raeder/DAI/Manhattan/guide/references.rst
so it can't find definitions for the rda web site references.

hkershaw-brown · 2025-01-17T21:26:30Z

@kdraeder do you want me to fix it or help you fix it?

hkershaw-brown · 2025-01-21T17:28:55Z

I'll just go ahead and fix this Kevin

for RDA datasets: - cam6 rean d345 - cam 4 rean d199 - prepbufr d09 - wod d285

hkershaw-brown

I switched the https://rda.ucar.edu/datasets links for targets in guide/references.rst.

I'm happy for this to be merged and released.

@braczka do you have any objections?

kdraeder · 2025-01-21T19:40:33Z

I'm still trying to figure out what you changed and how it compares with the work I did Friday, but didn't commit yet. My git skills are not the quickest.
My list of references is longer, and I removed "anonymous" links, since they seem to be less robust.
If you want to be done with it, go ahead. If you want to see my changes, then I need to figure out how to include the references.rst file into the files that need it. So far I don't see an include in your cam-fv/readme.rst to use as a template.

I switched the https://rda.ucar.edu/datasets links for targets in guide/references.rst.

I'm happy for this to be merged and released.

@braczka do you have any objections?

hkershaw-brown · 2025-01-21T19:44:13Z

these lines in the conf.py includes the guide/refrences.rst in every file

# include references
with open(os.path.join(os.path.dirname(__file__), 'guide/references.rst')) as f:
    references_content = f.read()

rst_prolog = references_content

hkershaw-brown · 2025-01-21T19:46:15Z

Here is what I changed:
a67de0e

guide/references.rst Collected frequently used web addresses for convenient updating. conf.py Added lines to make references.rst available in all .rst files. models/POP/readme.rst models/cam-fv/readme.rst observations/obs_converters/WOD/WOD.rst Updated to use references.rst links.

WARNING: there are still many non-link references (d######) to RDA datasets. These will not be updated by changes to references.rst, if, for example, the RDA changes its dataset name convention again. Trying to continue the merge. I resolved the conflicts, ran `make html`, which noted undefined links to the entries in refs, because they used Helen's abbreviations. I fixed those ({clm,wrf}/tutorial/README.rst and prep_bufr.rst), and added them explicitly, since they weren't part of the 2 previous references.rst commits and didn't have any conflicts.

I pulled this commit already, but there were conflicts. I resolved them by editing all the conflicted files, committing them, and 'continue'ing the merge. It appeared to work. I did an explicit merge again and got the message that there was nothing to merge. I tried to push the changes, but was told again that there are upstream changes that need to be merged. But they are all the same as what I just merged (based on the time stamps. So I'm trying to pull them again, with a different command; `git pull -s ort -X ours upstream rda_refs`. I have little hope.

hkershaw-brown · 2025-01-22T00:12:12Z

ok @kdraeder I guess you are doing the changes, I will redo the review.

models/POP/readme.rst

models/cam-fv/readme.rst

hkershaw-brown · 2025-01-22T00:29:00Z

models/cam-fv/readme.rst


 The variety of research can be sampled on the DART  
-`Publications <https://dart.ucar.edu/pages/Publications.html>`__ page.
+`Publications <https://dart.ucar.edu/pages/Publications.html>`_ page.


what's the goal of making this non-anonymous?

In my limited reading I got the impression that anonymous is less robust, but now I see that it's an "embedded" URI and there are no other references to it (in this document), so anonymous is not any riskier than a named link or pointer. I'll revert it.

hkershaw-brown · 2025-01-22T00:33:13Z

models/cam-fv/readme.rst

 is described in the `CESM readme <../CESM/readme.html>`_
-This document focuses on the several `atmospheric models <http://www2.cesm.ucar.edu/models>`__
+This document focuses on the several `atmospheric models <http://www2.cesm.ucar.edu/models>`_


Are you updating all the links in this document?

If so, <../CESM/readme.html>`_ should be changed to not be html so it builds correctly with latex, etc. builds of the documentation.

My goal was not to update all the links, but it seemed like I should fix link-related things that I noticed. My intent was the same as the Publications change (previous comment), but you've brought up a new issue - latex - so I'm wondering whether the html form is fine in an anonymous link (__) but not in a named link (_).

:doc:`CESM readme <../CESM/readme>`

will create a link that works in html (or pdf, or ps, or whatever is built)

The x.html links show up as broken links when you run a link checker (e.g. #799 ~355 broken links), so they are good to fix. That way it is easier to spot actually broken links (like an external site has moved)

if you want to get super tidy, you could get rid of the relative path (which will break if/when someone reorganizes the docs) you could add a target to the CESM/readme.rst file

.. _ CESMreadme :ref:`CESMreade`

We have loads of relative links, so not super necessary at the moment.

The x.html links show up as broken links when you run a link checker, so they are good to fix

I'd probably enjoy making it super tidy and maintainable, but I don't want to add that mission creep to this issue. I've opted for the :doc: form (in 3 places). Thanks for the tips!

models/cam-fv/readme.rst

hkershaw-brown · 2025-01-22T00:37:39Z

models/cam-fv/readme.rst

-`DS345.0 <https://rda.ucar.edu/datasets/ds345.0/#!description>`__ .
-(See the |CAM6_Rean|_ ).
+`d345000 <CAM6rean_rda_>`_ .
+(See the `CAM6rean_wiki`_ for details).


You've changed the name of the link here. Main branch:
(See the 1 degree reanalysis wiki ).

This pull request:
(See the CAM6rean_wiki for details).

That was a reference formatting error. But the doc still refers to the wiki by different words. Is that a problem if the rst name of the link is consistent?

no not a problem just checking that you were ok with having the link test be CAM6rean_wiki vs 1 degree reanalysis wiki. Either seems ok to me

edit: link text, not test

models/cam-fv/readme.rst

hkershaw-brown · 2025-01-22T00:39:14Z

models/cam-fv/readme.rst

 Versions of these files, which also have the results of the reanalysis in them,
-are available from the RDA ds345.0 linked above.
+are available from the RDA d345000 linked above.


why not link here?

When I first wrote it they were closer, and the extra typing didn't seem worthwhile. I've made it a link now.

Removed mistaken `include` rst commands. models/cam-fv/readme.rst There is a "Publications" link, so Removed the related ISSUE. models/POP/readme.rst

hkershaw-brown

Nice work Kevin!

kdraeder · 2025-01-23T19:00:52Z

Nice work Kevin!

Thanks! and thanks for your suggestions.

kdraeder added Bug Something isn't working Documentation Improvements or additions to documentation CAM Community Atmosphere Model CLM Community Land Model wrf Weather Research & Forecasting Model POP Parallel Ocean Program cice Sea Ice models labels Jan 8, 2025

kdraeder requested review from braczka and hkershaw-brown January 8, 2025 16:38

kdraeder self-assigned this Jan 8, 2025

hkershaw-brown and others added 2 commits January 21, 2025 12:29

Merge branch 'main' into rda_refs

37ce388

doc: reference page, rda links

a67de0e

for RDA datasets: - cam6 rean d345 - cam 4 rean d199 - prepbufr d09 - wod d285

hkershaw-brown force-pushed the rda_refs branch from 6e736a5 to a67de0e Compare January 21, 2025 19:19

hkershaw-brown approved these changes Jan 21, 2025

View reviewed changes

hkershaw-brown added the release! bundle with next release label Jan 21, 2025

kdraeder added 2 commits January 21, 2025 13:49

hkershaw-brown requested changes Jan 22, 2025

View reviewed changes

hkershaw-brown removed the release! bundle with next release label Jan 22, 2025

Tidied comments and link formatting and naming based on review.

56e01d2

Removed mistaken `include` rst commands. models/cam-fv/readme.rst There is a "Publications" link, so Removed the related ISSUE. models/POP/readme.rst

hkershaw-brown added the release! bundle with next release label Jan 23, 2025

hkershaw-brown and others added 3 commits January 23, 2025 13:36

Merge branch 'main' into rda_refs

1768bc9

fix: link missing _

a705c56

doc: copy button for documnentation

4c49aea

hkershaw-brown approved these changes Jan 23, 2025

View reviewed changes

hkershaw-brown merged commit d042596 into main Jan 23, 2025
4 checks passed

hkershaw-brown deleted the rda_refs branch January 23, 2025 19:15

hkershaw-brown mentioned this pull request Jan 23, 2025

bug: Research Data Archive links are broken by new naming convention #755

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updated RDA addresses and dataset names #790

Updated RDA addresses and dataset names #790

kdraeder commented Jan 8, 2025

hkershaw-brown commented Jan 8, 2025

kdraeder commented Jan 8, 2025

hkershaw-brown commented Jan 8, 2025

hkershaw-brown commented Jan 8, 2025

kdraeder commented Jan 8, 2025

hkershaw-brown commented Jan 8, 2025

kdraeder commented Jan 17, 2025

hkershaw-brown commented Jan 17, 2025

hkershaw-brown commented Jan 21, 2025

hkershaw-brown left a comment

kdraeder commented Jan 21, 2025

hkershaw-brown commented Jan 21, 2025 •

edited

Loading

hkershaw-brown commented Jan 21, 2025

hkershaw-brown commented Jan 22, 2025

hkershaw-brown Jan 22, 2025

kdraeder Jan 22, 2025

hkershaw-brown Jan 22, 2025

kdraeder Jan 22, 2025

hkershaw-brown Jan 22, 2025

kdraeder Jan 22, 2025

hkershaw-brown Jan 22, 2025

kdraeder Jan 22, 2025

hkershaw-brown Jan 22, 2025

hkershaw-brown Jan 22, 2025

hkershaw-brown Jan 22, 2025

kdraeder Jan 22, 2025

hkershaw-brown left a comment

kdraeder commented Jan 23, 2025

Updated RDA addresses and dataset names #790

Updated RDA addresses and dataset names #790

Conversation

kdraeder commented Jan 8, 2025

Description:

Fixes issue

Types of changes

Documentation changes needed?

Tests

Checklist for merging

Checklist for release

Testing Datasets

hkershaw-brown commented Jan 8, 2025

kdraeder commented Jan 8, 2025

hkershaw-brown commented Jan 8, 2025

hkershaw-brown commented Jan 8, 2025

kdraeder commented Jan 8, 2025

hkershaw-brown commented Jan 8, 2025

kdraeder commented Jan 17, 2025

hkershaw-brown commented Jan 17, 2025

hkershaw-brown commented Jan 21, 2025

hkershaw-brown left a comment

Choose a reason for hiding this comment

kdraeder commented Jan 21, 2025

hkershaw-brown commented Jan 21, 2025 • edited Loading

hkershaw-brown commented Jan 21, 2025

hkershaw-brown commented Jan 22, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hkershaw-brown left a comment

Choose a reason for hiding this comment

kdraeder commented Jan 23, 2025

hkershaw-brown commented Jan 21, 2025 •

edited

Loading