Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] snapshot/status.yaml is flaky #756

Closed
dblock opened this issue Dec 31, 2024 · 3 comments · Fixed by #768
Closed

[BUG] snapshot/status.yaml is flaky #756

dblock opened this issue Dec 31, 2024 · 3 comments · Fixed by #768
Labels
bug Something isn't working flaky test untriaged

Comments

@dblock
Copy link
Member

dblock commented Dec 31, 2024

What is the bug?

FAILED  snapshot/status.yaml (tests/snapshot/snapshot/status.yaml)
    FAILED  CHAPTERS
        PASSED  Get status of all running snapshots.
            PASSED  REQUEST BODY
            PASSED  RESPONSE STATUS
            PASSED  RESPONSE PAYLOAD BODY
            PASSED  RESPONSE PAYLOAD SCHEMA
            SKIPPED RESPONSE OUTPUT VALUES
        FAILED  Get status of all running snapshots in specific repository.
            PASSED  PARAMETERS
                PASSED  repository
            PASSED  REQUEST BODY
            PASSED  RESPONSE STATUS
            FAILED  RESPONSE PAYLOAD BODY (missing snapshots[0]='[object Object]')
            PASSED  RESPONSE PAYLOAD SCHEMA
            SKIPPED RESPONSE OUTPUT VALUES
        PASSED  Get status of specific snapshot in specific repository.
            PASSED  PARAMETERS
                PASSED  repository
                PASSED  snapshot
            PASSED  REQUEST BODY
            PASSED  RESPONSE STATUS
            PASSED  RESPONSE PAYLOAD BODY
            PASSED  RESPONSE PAYLOAD SCHEMA
            SKIPPED RESPONSE OUTPUT VALUES

How can one reproduce the bug?

https://github.com/opensearch-project/opensearch-api-specification/actions/runs/12559264739/job/35014742950

What is the expected behavior?

Tests to pass reliably.

What is your host/environment?

GHA
@dblock dblock added bug Something isn't working flaky test labels Dec 31, 2024
@dblock
Copy link
Member Author

dblock commented Jan 6, 2025

Full log.

[INFO] Evaluating snapshot/status.yaml ...
[INFO] => PUT /_snapshot/my-fs-repository ({}) [application/json] {
  "type": "fs",
  "settings": {
    "location": "/tmp/opensearch/repo"
  }
}
[INFO] <= 200 (application/json; charset=UTF-8) | {
  "acknowledged": true
}
[INFO] => PUT /_snapshot/my-fs-repository/my-test-snapshot ({}) [application/json] {
  "indices": "*",
  "ignore_unavailable": true,
  "include_global_state": false,
  "partial": true
}
[INFO] <= 200 (application/json; charset=UTF-8) | {
  "accepted": true
}
[INFO] => GET /_snapshot/_status ({}) [application/json] 
[INFO] <= 200 (application/json; charset=UTF-8) | {
  "snapshots": [
    {
      "snapshot": "my-test-snapshot",
      "repository": "my-fs-repository",
      "uuid": "1-K65GWuQmOsGGwfp3oq7A",
      "state": "STARTED",
      "include_global_state": false,
      "shards_stats": {
        "initializing": 0,
        "started": 0,
        "finalizing": 3,
        "done": 1,
        "failed": 0,
        "total": 4
      },
      "stats": {
        "incremental": {
          "file_count": 0,
          "size_in_bytes": 0
        },
        "total": {
          "file_count": 79,
          "size_in_bytes": 258458
        },
        "start_time_in_millis": 1736186935298,
        "time_in_millis": 0
      },
      "indices": {
        ".opensearch-observability": {
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 1,
            "done": 0,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "incremental": {
              "file_count": 0,
              "size_in_bytes": 0
            },
            "total": {
              "file_count": 1,
              "size_in_bytes": 208
            },
            "start_time_in_millis": 1736186935298,
            "time_in_millis": 0
          },
          "shards": {
            "0": {
              "stage": "FINALIZE",
              "stats": {
                "incremental": {
                  "file_count": 0,
                  "size_in_bytes": 0
                },
                "total": {
                  "file_count": 1,
                  "size_in_bytes": 208
                },
                "start_time_in_millis": 1736186935298,
                "time_in_millis": 0
              }
            }
          }
        },
        ".opensearch-sap-log-types-config": {
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 1,
            "done": 0,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "incremental": {
              "file_count": 0,
              "size_in_bytes": 0
            },
            "total": {
              "file_count": 25,
              "size_in_bytes": 135727
            },
            "start_time_in_millis": 1736186935298,
            "time_in_millis": 0
          },
          "shards": {
            "0": {
              "stage": "FINALIZE",
              "stats": {
                "incremental": {
                  "file_count": 0,
                  "size_in_bytes": 0
                },
                "total": {
                  "file_count": 25,
                  "size_in_bytes": 135727
                },
                "start_time_in_millis": 1736186935298,
                "time_in_millis": 0
              }
            }
          }
        },
        "security-auditlog-2025.01.06": {
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 0,
            "done": 1,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "incremental": {
              "file_count": 0,
              "size_in_bytes": 0
            },
            "total": {
              "file_count": 22,
              "size_in_bytes": 38138
            },
            "start_time_in_millis": 1736186935298,
            "time_in_millis": 0
          },
          "shards": {
            "0": {
              "stage": "DONE",
              "stats": {
                "incremental": {
                  "file_count": 0,
                  "size_in_bytes": 0
                },
                "total": {
                  "file_count": 22,
                  "size_in_bytes": 38138
                },
                "start_time_in_millis": 1736186935298,
                "time_in_millis": 0
              }
            }
          }
        },
        ".opendistro_security": {
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 1,
            "done": 0,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "incremental": {
              "file_count": 0,
              "size_in_bytes": 0
            },
            "total": {
              "file_count": 31,
              "size_in_bytes": 84385
            },
            "start_time_in_millis": 1736186935298,
            "time_in_millis": 0
          },
          "shards": {
            "0": {
              "stage": "FINALIZE",
              "stats": {
                "incremental": {
                  "file_count": 0,
                  "size_in_bytes": 0
                },
                "total": {
                  "file_count": 31,
                  "size_in_bytes": 84385
                },
                "start_time_in_millis": 1736186935298,
                "time_in_millis": 0
              }
            }
          }
        }
      }
    }
  ]
}
[INFO] {
  "snapshots": [
    {
      "snapshot": "my-test-snapshot",
      "repository": "my-fs-repository",
      "uuid": "1-K65GWuQmOsGGwfp3oq7A",
      "state": "STARTED",
      "include_global_state": false,
      "shards_stats": {
        "initializing": 0,
        "started": 0,
        "finalizing": 3,
        "done": 1,
        "failed": 0,
        "total": 4
      },
      "stats": {
        "incremental": {
          "file_count": 0,
          "size_in_bytes": 0
        },
        "total": {
          "file_count": 79,
          "size_in_bytes": 258458
        },
        "start_time_in_millis": 1736186935298,
        "time_in_millis": 0
      },
      "indices": {
        ".opensearch-observability": {
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 1,
            "done": 0,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "incremental": {
              "file_count": 0,
              "size_in_bytes": 0
            },
            "total": {
              "file_count": 1,
              "size_in_bytes": 208
            },
            "start_time_in_millis": 1736186935298,
            "time_in_millis": 0
          },
          "shards": {
            "0": {
              "stage": "FINALIZE",
              "stats": {
                "incremental": {
                  "file_count": 0,
                  "size_in_bytes": 0
                },
                "total": {
                  "file_count": 1,
                  "size_in_bytes": 208
                },
                "start_time_in_millis": 1736186935298,
                "time_in_millis": 0
              }
            }
          }
        },
        ".opensearch-sap-log-types-config": {
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 1,
            "done": 0,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "incremental": {
              "file_count": 0,
              "size_in_bytes": 0
            },
            "total": {
              "file_count": 25,
              "size_in_bytes": 135727
            },
            "start_time_in_millis": 1736186935298,
            "time_in_millis": 0
          },
          "shards": {
            "0": {
              "stage": "FINALIZE",
              "stats": {
                "incremental": {
                  "file_count": 0,
                  "size_in_bytes": 0
                },
                "total": {
                  "file_count": 25,
                  "size_in_bytes": 135727
                },
                "start_time_in_millis": 1736186935298,
                "time_in_millis": 0
              }
            }
          }
        },
        "security-auditlog-2025.01.06": {
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 0,
            "done": 1,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "incremental": {
              "file_count": 0,
              "size_in_bytes": 0
            },
            "total": {
              "file_count": 22,
              "size_in_bytes": 38138
            },
            "start_time_in_millis": 1736186935298,
            "time_in_millis": 0
          },
          "shards": {
            "0": {
              "stage": "DONE",
              "stats": {
                "incremental": {
                  "file_count": 0,
                  "size_in_bytes": 0
                },
                "total": {
                  "file_count": 22,
                  "size_in_bytes": 38138
                },
                "start_time_in_millis": 1736186935298,
                "time_in_millis": 0
              }
            }
          }
        },
        ".opendistro_security": {
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 1,
            "done": 0,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "incremental": {
              "file_count": 0,
              "size_in_bytes": 0
            },
            "total": {
              "file_count": 31,
              "size_in_bytes": 84385
            },
            "start_time_in_millis": 1736186935298,
            "time_in_millis": 0
          },
          "shards": {
            "0": {
              "stage": "FINALIZE",
              "stats": {
                "incremental": {
                  "file_count": 0,
                  "size_in_bytes": 0
                },
                "total": {
                  "file_count": 31,
                  "size_in_bytes": 84385
                },
                "start_time_in_millis": 1736186935298,
                "time_in_millis": 0
              }
            }
          }
        }
      }
    }
  ]
}
[INFO] => GET /_snapshot/my-fs-repository/_status ({}) [application/json] 
[INFO] <= 200 (application/json; charset=UTF-8) | {
  "snapshots": []
}
[INFO] {
  "snapshots": []
}
[INFO] => GET /_snapshot/my-fs-repository/my-test-snapshot/_status ({}) [application/json] 
[INFO] <= 200 (application/json; charset=UTF-8) | {
  "snapshots": [
    {
      "snapshot": "my-test-snapshot",
      "repository": "my-fs-repository",
      "uuid": "1-K65GWuQmOsGGwfp3oq7A",
      "state": "SUCCESS",
      "include_global_state": false,
      "shards_stats": {
        "initializing": 0,
        "started": 0,
        "finalizing": 0,
        "done": 4,
        "failed": 0,
        "total": 4
      },
      "stats": {
        "incremental": {
          "file_count": 0,
          "size_in_bytes": 0
        },
        "total": {
          "file_count": 79,
          "size_in_bytes": 258458
        },
        "start_time_in_millis": 1736186935298,
        "time_in_millis": 0
      },
      "indices": {
        ".opensearch-observability": {
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 0,
            "done": 1,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "incremental": {
              "file_count": 0,
              "size_in_bytes": 0
            },
            "total": {
              "file_count": 1,
              "size_in_bytes": 208
            },
            "start_time_in_millis": 1736186935298,
            "time_in_millis": 0
          },
          "shards": {
            "0": {
              "stage": "DONE",
              "stats": {
                "incremental": {
                  "file_count": 0,
                  "size_in_bytes": 0
                },
                "total": {
                  "file_count": 1,
                  "size_in_bytes": 208
                },
                "start_time_in_millis": 1736186935298,
                "time_in_millis": 0
              }
            }
          }
        },
        ".opensearch-sap-log-types-config": {
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 0,
            "done": 1,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "incremental": {
              "file_count": 0,
              "size_in_bytes": 0
            },
            "total": {
              "file_count": 25,
              "size_in_bytes": 135727
            },
            "start_time_in_millis": 1736186935298,
            "time_in_millis": 0
          },
          "shards": {
            "0": {
              "stage": "DONE",
              "stats": {
                "incremental": {
                  "file_count": 0,
                  "size_in_bytes": 0
                },
                "total": {
                  "file_count": 25,
                  "size_in_bytes": 135727
                },
                "start_time_in_millis": 1736186935298,
                "time_in_millis": 0
              }
            }
          }
        },
        "security-auditlog-2025.01.06": {
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 0,
            "done": 1,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "incremental": {
              "file_count": 0,
              "size_in_bytes": 0
            },
            "total": {
              "file_count": 22,
              "size_in_bytes": 38138
            },
            "start_time_in_millis": 1736186935298,
            "time_in_millis": 0
          },
          "shards": {
            "0": {
              "stage": "DONE",
              "stats": {
                "incremental": {
                  "file_count": 0,
                  "size_in_bytes": 0
                },
                "total": {
                  "file_count": 22,
                  "size_in_bytes": 38138
                },
                "start_time_in_millis": 1736186935298,
                "time_in_millis": 0
              }
            }
          }
        },
        ".opendistro_security": {
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 0,
            "done": 1,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "incremental": {
              "file_count": 0,
              "size_in_bytes": 0
            },
            "total": {
              "file_count": 31,
              "size_in_bytes": 84385
            },
            "start_time_in_millis": 1736186935298,
            "time_in_millis": 0
          },
          "shards": {
            "0": {
              "stage": "DONE",
              "stats": {
                "incremental": {
                  "file_count": 0,
                  "size_in_bytes": 0
                },
                "total": {
                  "file_count": 31,
                  "size_in_bytes": 84385
                },
                "start_time_in_millis": 1736186935298,
                "time_in_millis": 0
              }
            }
          }
        }
      }
    }
  ]
}
[INFO] {
  "snapshots": [
    {
      "snapshot": "my-test-snapshot",
      "repository": "my-fs-repository",
      "uuid": "1-K65GWuQmOsGGwfp3oq7A",
      "state": "SUCCESS",
      "include_global_state": false,
      "shards_stats": {
        "initializing": 0,
        "started": 0,
        "finalizing": 0,
        "done": 4,
        "failed": 0,
        "total": 4
      },
      "stats": {
        "incremental": {
          "file_count": 0,
          "size_in_bytes": 0
        },
        "total": {
          "file_count": 79,
          "size_in_bytes": 258458
        },
        "start_time_in_millis": 1736186935298,
        "time_in_millis": 0
      },
      "indices": {
        ".opensearch-observability": {
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 0,
            "done": 1,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "incremental": {
              "file_count": 0,
              "size_in_bytes": 0
            },
            "total": {
              "file_count": 1,
              "size_in_bytes": 208
            },
            "start_time_in_millis": 1736186935298,
            "time_in_millis": 0
          },
          "shards": {
            "0": {
              "stage": "DONE",
              "stats": {
                "incremental": {
                  "file_count": 0,
                  "size_in_bytes": 0
                },
                "total": {
                  "file_count": 1,
                  "size_in_bytes": 208
                },
                "start_time_in_millis": 1736186935298,
                "time_in_millis": 0
              }
            }
          }
        },
        ".opensearch-sap-log-types-config": {
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 0,
            "done": 1,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "incremental": {
              "file_count": 0,
              "size_in_bytes": 0
            },
            "total": {
              "file_count": 25,
              "size_in_bytes": 135727
            },
            "start_time_in_millis": 1736186935298,
            "time_in_millis": 0
          },
          "shards": {
            "0": {
              "stage": "DONE",
              "stats": {
                "incremental": {
                  "file_count": 0,
                  "size_in_bytes": 0
                },
                "total": {
                  "file_count": 25,
                  "size_in_bytes": 135727
                },
                "start_time_in_millis": 1736186935298,
                "time_in_millis": 0
              }
            }
          }
        },
        "security-auditlog-2025.01.06": {
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 0,
            "done": 1,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "incremental": {
              "file_count": 0,
              "size_in_bytes": 0
            },
            "total": {
              "file_count": 22,
              "size_in_bytes": 38138
            },
            "start_time_in_millis": 1736186935298,
            "time_in_millis": 0
          },
          "shards": {
            "0": {
              "stage": "DONE",
              "stats": {
                "incremental": {
                  "file_count": 0,
                  "size_in_bytes": 0
                },
                "total": {
                  "file_count": 22,
                  "size_in_bytes": 38138
                },
                "start_time_in_millis": 1736186935298,
                "time_in_millis": 0
              }
            }
          }
        },
        ".opendistro_security": {
          "shards_stats": {
            "initializing": 0,
            "started": 0,
            "finalizing": 0,
            "done": 1,
            "failed": 0,
            "total": 1
          },
          "stats": {
            "incremental": {
              "file_count": 0,
              "size_in_bytes": 0
            },
            "total": {
              "file_count": 31,
              "size_in_bytes": 84385
            },
            "start_time_in_millis": 1736186935298,
            "time_in_millis": 0
          },
          "shards": {
            "0": {
              "stage": "DONE",
              "stats": {
                "incremental": {
                  "file_count": 0,
                  "size_in_bytes": 0
                },
                "total": {
                  "file_count": 31,
                  "size_in_bytes": 84385
                },
                "start_time_in_millis": 1736186935298,
                "time_in_millis": 0
              }
            }
          }
        }
      }
    }
  ]
}
[INFO] => DELETE /_snapshot/my-fs-repository/my-test-snapshot ({}) [application/json] 
[INFO] <= 200 (application/json; charset=UTF-8) | {
  "acknowledged": true
}
[INFO] => DELETE /_snapshot/my-fs-repository ({}) [application/json] 
[INFO] <= 200 (application/json; charset=UTF-8) | {
  "acknowledged": true
}

This was referenced Jan 6, 2025
Closed
Merged
@Xtansia
Copy link
Collaborator

Xtansia commented Jan 6, 2025

Also replied on the OpenSearch repo issue, but copying here:

I believe GET /_snapshot/_status and GET /_snapshot/{repository}/_status only return actively running snapshots, where as GET /_snapshot/{repository}/{snapshot}/_status returns the snapshot regardless of status. So I think it might just be that the snapshot completed between request 1. and 2..

@dblock
Copy link
Member Author

dblock commented Jan 6, 2025

@Xtansia thanks, you are right, I removed the payload check in #768.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working flaky test untriaged
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix flaky snapshot test. dblock/opensearch-api-specification
2 participants
@dblock @Xtansia