If Aeris throws an exception Belchertown doesn't update

[dkbarto]

First step
install weewx, belchertown, and the forecast extension.

Describe the bug
It appears that belchertown doesn't expect the exception from aeris and just aborts page generation, rather it should put up some kind of error entry "Aeris forecast not available", as an example.

Link to your website (IMPORTANT)
http://www.kdbarto.org/weather/belchertown

Version of the skin you're using
1.3.1

To Reproduce
Steps to reproduce the behavior:
Run and re-run aeris forecast until you have used all your free connections in a day.
Restart weewx
Note that Belchertown no longer updates because of the thrown exception getting the aeris forecast.

Expected behavior
What would be nice is to have a default "Aeris forecast is unavailable "
In this cast the would say "Too Many Requests"

Screenshots

LOOP:   Traceback (most recent call last):
  File "/Users/Shared/weewx-4.10.2/bin/user/belchertown.py", line 1273, in get_extension_list
    response = urlopen(req)
               ^^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 429: Too Many Requests

Device Information (please complete the following information):
MacOS 12.6.8
Weewx 4.10.2
Python 3.11.5
Forecast extension from https://github.com/chaunceygardiner/weewx-forecast

[michaelundwd]

I don't know if this helps you but I wonder if the problem is caused by your use of the forecast extension. You don’t need it to be able to access Aeris forecasts from the Belchertown skin. I am not familiar with the forecast extension, the number of calls it makes to Aeris or the limit set by your access method to Aeris.
The number of calls made by the Belchertown skin directly is well within the limit of 1000/day set by the PWS contributor plan, as it limits forecast updates to one per hour. See the Belchertown readme.md for further details.
Michael

[dkbarto]

No, it isn't related to the use of the forecast extension. It was because I had a bug in the refresh rate on Belchertown that was fetching the forecast once per minute. Once fixed the problem doesn't reproduce and still, I didn't expect Belchertown to just fail so completely when the forecast from Aeris failed. That is what I consider the root problem here.

[michaelundwd]

It's unlikely to occur under normal conditions, but otherwise fair point. Let's see if anyone with better python skills than me can provide a fix for the error conditions that Aeris can produce.

[dkbarto]

I agree 100%. I've not got the python chops to fix it either.

[dkbarto]

Um - this code looks funky to me.

                    if (
                        time.strftime("%M") < "05"
                        and int(time.time()) - int(os.path.getmtime(forecast_file))
                    ) > int(300):

Specifically a boolean appears to be checked for being greater than 300
And the strftime string is being compared to the string "05".

from an C++ programmers POV wouldn't you check the time difference to be > 300, and the strftime to be cast to an int and then compared to the value 5?

Could someone explain this logic please?

[swfrx]

I had this recently, wish I had seen this report before lots of fiddling around with the Pi & WeeWx to try to nail the issue.

I also am not good enough at Python to fix it!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[uajqq]

Um - this code looks funky to me.

                    if (
                        time.strftime("%M") < "05"
                        and int(time.time()) - int(os.path.getmtime(forecast_file))
                    ) > int(300):

Specifically a boolean appears to be checked for being greater than 300 And the strftime string is being compared to the string "05".

from an C++ programmers POV wouldn't you check the time difference to be > 300, and the strftime to be cast to an int and then compared to the value 5?

Could someone explain this logic please?

If the first statement in parentheses evaluates to false, the entire if (...) block returns as false. False > int(300) is false, so that's the final answer.
If the first statement evaluates to true, then the entire if (...) block returns the integer result of the equation on the second line, which is then evaluated against int(300). That can be false or true, depending.

[michaelundwd]

As I modified the original code, let me answer your question. The original code ensured the forecast downloaded at every archive interval to reduce the load on the server (now AERIS). When I introduced the 1 hour and 3 hour options for the forecast (the original version was every 24 hours), I inserted the check on the minutes past the hour (05) to ensure that in a steady state the forecast was downloaded on the hour. So if the first forecast downloaded at 35 mins past the hour, this code would force the next forecast on the hour and then the original code thereafter, That looked tidier for the 1 hour forecasts. Does that help? I was finding my way with Python (still am!) so that may explain the coding.

In retrospect, the 300 should have been replaced by forecast_stale_timer. Feel free to change it.

Michael

[michaelundwd]

Apologies, there are a couple of errors in my last response
1 The original code ensured that the forecast downloaded at hourly intervals, (rather than archive interval).
2 300 should have been replaced by archive_interval.

[uajqq]

I can replicate this bug by turning off API access in Aeris/XWeather (so it returns "403: Forbidden"), so I think having a fix in place would be a good idea, even though rate-limiting is unlikely to occur under normal conditions, for situations when the API isn't working for some reason. I'm opening a pull request.