CASMINST-7172 add simple retry logic to postgresql_check_SyncFailed t…

…est (#680) CASMINST-7172 add simple retry logic to postgresql_check_SyncFailed test detected in auto-triage, this script failed, but worked upon manual execution by a human. this adds a simple retry logic as an additional safety net to help prevent false auto-triage tickets. Signed-off-by: Jacob Salmela <[email protected]>
Cray-HPE · Feb 12, 2025 · 9221a35 · 9221a35
1 parent 3ef5a07
commit 9221a35
Showing 1 changed file with 14 additions and 2 deletions.
diff --git a/goss-testing/tests/ncn/goss-postgresql-syncfailed.yaml b/goss-testing/tests/ncn/goss-postgresql-syncfailed.yaml
@@ -32,8 +32,20 @@ command:
             desc: Checks if any posgresql have status SyncFailed. If this fails, investigate postgresql statuses. Refer to 'operations/kubernetes/Troubleshoot_Postgres_Database.md' in the CSM documentation for more information on diagnosing and fixing clusters in a SyncFailed state.
             sev: 0
         exec: |-
-            "{{$logrun}}" -l "{{$testlabel}}" \
-                "{{$kubectl}}" get postgresql -A -o jsonpath='{.items[*].status.PostgresClusterStatus}'
+            retries=3
+            count=0
+            while [[ $count -lt $retries ]]; do
+                result=$("{{$logrun}}" -l "{{$testlabel}}" \
+                    "{{$kubectl}}" get postgresql -A -o jsonpath='{.items[*].status.PostgresClusterStatus}')
+                if [[ $result != *"SyncFailed"* ]]; then
+                    echo "$result"
+                    exit 0
+                fi
+                count=$((count + 1))
+                sleep 2
+            done
+            echo "$result"
+            exit 1
         stdout:
         - "!SyncFailed"
         exit-status: 0