Issue/n calls per hour (#381)

* default, rate limit to 60

* add "current_running" in cache

* better logging

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* docs

* lint

* rename QUERY_WAIT_SECONDS

* docs

* add LOGLEVEL, default wait time 30, set route to not running after waiting 30 seconds

* safer handling on keys, set not running after 30 seconds - bug fix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* lint

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

README.md

@@ -36,6 +36,7 @@ You will need to set the following environmental variables:
 - `QUERY_WAIT_SECONDS` - The number of seconds to wait for an on going query
 - `CACHE_TIME_SECONDS` - The time in seconds to cache the data is used for
 - `DELETE_CACHE_TIME_SECONDS` - The time in seconds to after which the cache is delete
+- `LOGLEVEL` - The log level for the application.
 
 Note you will need a database set up at `DB_URL`. This should use the datamodel in [nowcasting_datamodel](https://github.com/openclimatefix/nowcasting_datamodel)
 

src/cache.py

@@ -96,7 +96,10 @@ def wrapper(*args, **kwargs):  # noqa
 
         last_updated, response = remove_old_cache(last_updated, response)
 
-        # make route_variables into a string
+        # make route_variables into a string\
+        # TODO add url
+        # TODO sort route variables alphabetically
+        # url = request.url
         route_variables = json.dumps(route_variables)
 
         # use case
@@ -126,13 +129,20 @@ def wrapper(*args, **kwargs):  # noqa
                         return response[route_variables]
                     else:
                         logger.warning(
-                            f"Waited {QUERY_WAIT_SECONDS} seconds but response not "
-                            f"in cache. Setting this route as not running, "
-                            f"and continuing"
+                            "Process finished running but response not "
+                            "in cache. Setting this route as not running, "
+                            "and continuing"
                         )
                         currently_running[route_variables] = False
                         break
 
+            logger.warning(
+                f"Waited {QUERY_WAIT_SECONDS} seconds but response not "
+                f"in cache. Setting this route as not running, "
+                f"and continuing"
+            )
+            currently_running[route_variables] = False
+
         # 1.1 check if its been called before and not currently running
         if (route_variables not in last_updated) and (
             not currently_running.get(route_variables, False)
@@ -149,7 +159,9 @@ def wrapper(*args, **kwargs):  # noqa
 
         # 1.2 rerun if cache time out is up and not currently running
         now = datetime.now(tz=timezone.utc)
-        if now - timedelta(seconds=cache_time_seconds) > last_updated[route_variables] and (
+        if route_variables not in last_updated:
+            pass
+        elif now - timedelta(seconds=cache_time_seconds) > last_updated.get(route_variables) and (
             not currently_running.get(route_variables, False)
         ):
             logger.debug(

src/main.py

@@ -202,7 +202,7 @@ async def add_process_time_header(request: Request, call_next):
     start_time = time.time()
     response = await call_next(request)
     process_time = str(time.time() - start_time)
-    logger.debug(f"Process Time {process_time} {request.url}")
+    logger.info(f"Process Time {process_time} {request.url}")
     response.headers["X-Process-Time"] = process_time
 
     return response