Skip to content

Commit

Permalink
consumer: rollback database transaction in case of errors
Browse files Browse the repository at this point in the history
Rollback the database `Session` when an error occurs while handling a
status update message.

This fixes an issue where, if the current transaction fails for any
reason, then the consumer keeps consuming the queue without actually
updating the status of workflows and deleting the related `run-batch`
pods. This is caused by the fact that messages are acknowledged, but
querying the database fails, as the transaction was not rolled back.
This causes REANA to stop scheduling workflows, as finished workflows
are still considered as running.
  • Loading branch information
mdonadoni committed Aug 11, 2023
1 parent 764d26a commit 990e1a8
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions reana_workflow_controller/consumer.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# -*- coding: utf-8 -*-
#
# This file is part of REANA.
# Copyright (C) 2018, 2019, 2020, 2021, 2022 CERN.
# Copyright (C) 2018, 2019, 2020, 2021, 2022, 2023 CERN.
#
# REANA is free software; you can redistribute it and/or modify it
# under the terms of the MIT License; see LICENSE file for more details.
Expand Down Expand Up @@ -126,14 +126,14 @@ def on_message(self, body, message):
f"Event for workflow {workflow_uuid} that doesn't exist in DB received:\n"
f"{body}\nIgnoring..."
)
except REANAWorkflowControllerError as rwce:
logging.error(rwce, exc_info=True)
except SQLAlchemyError as sae:
Session.rollback()

Check warning on line 130 in reana_workflow_controller/consumer.py

View check run for this annotation

Codecov / codecov/patch

reana_workflow_controller/consumer.py#L130

Added line #L130 was not covered by tests
logging.error(
f"Something went wrong while querying the database for workflow: {workflow_uuid}"
)
logging.error(sae, exc_info=True)
except Exception as e:
Session.rollback()
logging.error(
f"Unexpected error while processing workflow: {e}", exc_info=True
)
Expand Down

0 comments on commit 990e1a8

Please sign in to comment.