CLOWarden applied team changes before a PR was approved.

[austinlparker]

See open-telemetry/community#2356

In short, we were testing CLOWarden for the OpenTelemetry organization. We were performing a staged rollout where we created a new team in the config file to make sure that we could slowly roll out CLOWarden to existing teams. Our expectation was that it would not touch items that weren't defined in the config file or that it wouldn't change anything until we merged the PR. What happened, though, is that existing teams in the organization were deleted while CLOWarden was checking the configuration.

I've attached the relevant log output of the clowarden server below.

check-db-ready otel-clowarden-postgresql.default:5432 - accepting connections
server {"timestamp":"2024-09-18T15:44:50.585247Z","level":"INFO","fields":{"message":"server started"},"target":"clowarden_server"}
server {"timestamp":"2024-09-18T15:44:50.585568Z","level":"INFO","fields":{"message":"listening","addr":"0.0.0.0:9000"},"target":"clowarden_server"}
server {"timestamp":"2024-09-18T15:44:50.585242Z","level":"DEBUG","fields":{"message":"reconciling state","service_name":"github"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
Stream closed EOF for default/server-6784fc4dbc-ztwr4 (check-db-ready)
server {"timestamp":"2024-09-18T15:49:12.181907Z","level":"DEBUG","fields":{"message":"changes between the actual and the desired state","changes":"Changes { directory: [TeamRemoved(\"ebpf-profiler-maintainers\"), TeamRemoved(\"php-triagers\"), TeamRemoved(\"configuration-approvers\"), TeamRemoved(\"dotnet-instrumentation-triagers\"), TeamRemoved(\"swift-maintainers\"), TeamRemoved(\"go-instrumentation-approvers\"), TeamRemoved(\"sig-mainframe-approvers\"), TeamRemoved(\"configuration-maintainers\"), TeamRemoved(\"sig-developer-experience-approvers\"), TeamRemoved(\"android-approvers\"), TeamRemoved(\"php-maintainers\"), TeamRemoved(\"proto-go-approvers\"), TeamRemoved(\"android-maintainers\"), TeamRemoved(\"java-instrumentation-maintainers\"), TeamRemoved(\"docs-es-maintainers\"), TeamRemoved(\"sig-end-user-triagers\"), TeamRemoved(\"java-triagers\"), TeamRemoved(\"erlang-contrib-maintainers\"), TeamRemoved(\"cpp-contrib-approvers\"), TeamRemoved(\"cpp-maintainers\"), TeamRemoved(\"specs-semconv-maintainers\"), TeamRemoved(\"specs-triagers\"), TeamRemoved(\"network-approvers\"), TeamRemoved(\"ruby-approvers\"), TeamRemoved(\"sig-contributor-experience-maintainers\"), TeamRemoved(\"docs-ja-maintainers\"), TeamRemoved(\"technical-committee\"), TeamRemoved(\"semconv-system-approvers\"), TeamRemoved(\"collector-triagers\"), TeamRemoved(\"semconv-llm-approvers\"), TeamRemoved(\"collector-maintainers\"), TeamRemoved(\"lambda-extension-approvers\"), TeamRemoved(\"otel-elections\"), TeamRemoved(\"ebpf-profiler-approvers\"), TeamRemoved(\"docs-ja-approvers\"), TeamRemoved(\"go-maintainers\"), TeamRemoved(\"sig-mainframe-maintainers\"), TeamRemoved(\"swift-triagers\"), TeamRemoved(\"opamp-spec-approvers\"), TeamRemoved(\"ruby-contrib-approvers\"), TeamRemoved(\"sig-end-user-maintainers\"), TeamRemoved(\"semconv-messaging-approvers\"), TeamRemoved(\"rust-maintainers\"), TeamRemoved(\"cpp-approvers\"), TeamRemoved(\"assign-reviewers-action-maintainers\"), TeamRemoved(\"docs-pt-approvers\"), TeamRemoved(\"operator-maintainers\"), TeamRemoved(\"go-instrumentation-maintainers\"), TeamRemoved(\"java-contrib-triagers\"), TeamRemoved(\"go-instrumentation-triagers\"), TeamRemoved(\"operator-approvers\"), TeamRemoved(\"docs-pt-maintainers\"), TeamRemoved(\"java-approvers\"), TeamRemoved(\"spec-sponsors\"), TeamRemoved(\"javascript-maintainers\"), TeamRemoved(\"javascript-triagers\"), TeamRemoved(\"semconv-db-approvers\"), TeamRemoved(\"go-approvers\"), TeamRemoved(\"dotnet-approvers\"), TeamRemoved(\"semconv-feature-flag-approvers\"), TeamRemoved(\"opentelemetry-python-contrib-approvers\"), TeamRemoved(\"sqlcommenter-approvers\"), TeamRemoved(\"docs-approvers\"), TeamRemoved(\"opentelemetry-python-contrib-maintainers\"), TeamRemoved(\"java-contrib-maintainers\"), TeamRemoved(\"javascript-approvers\"), TeamRemoved(\"opamp-java-maintainers\"), TeamRemoved(\"opamp-spec-maintainers\"), TeamRemoved(\"arrow-maintainers\"), TeamRemoved(\"build-tools-approvers\"), TeamRemoved(\"dotnet-maintainers\"), TeamRemoved(\"weaver-maintainers\"), TeamRemoved(\"ruby-contrib-maintainers\"), TeamRemoved(\"dotnet-contrib-maintainers\"), TeamRemoved(\"erlang-maintainers\"), TeamRemoved(\"semconv-mobile-approvers\"), TeamRemoved(\"dotnet-triagers\"), TeamRemoved(\"sig-contributor-experience-approvers\"), TeamRemoved(\"instr-wg\"), TeamRemoved(\"ruby-maintainers\"), TeamRemoved(\"operator-ta-maintainers\"), TeamRemoved(\"semconv-security-approvers\"), TeamRemoved(\"opamp-go-approvers\"), TeamRemoved(\"java-instrumentation-triagers\"), TeamRemoved(\"docs-zh-maintainers\"), TeamRemoved(\"semconv-cicd-approvers\"), TeamRemoved(\"java-maintainers\"), TeamRemoved(\"opamp-java-triagers\"), TeamRemoved(\"helm-maintainers\"), TeamRemoved(\"cpp-contrib-maintainers\"), TeamRemoved(\"docs-zh-approvers\"), TeamRemoved(\"wg-prometheus\"), TeamRemoved(\"sig-mainframe-triagers\"), TeamRemoved(\"network-triagers\"), TeamRemoved(\"profiling-approvers\"), TeamRemoved(\"docs-es-approvers\"), TeamRemoved(\"demo-approvers\"), TeamRemoved(\"docs-cn-maintainers\"), TeamRemoved(\"docs-maintainers\"), TeamRemoved(\"arrow-approvers\"), TeamRemoved(\"opamp-java-approvers\"), TeamRemoved(\"helm-triagers\"), TeamRemoved(\"opamp-go-maintainers\"), TeamRemoved(\"collector-contrib-triagers\"), TeamRemoved(\"blog-approvers\"), TeamRemoved(\"erlang-contrib-approvers\"), TeamRemoved(\"collector-contrib-maintainers\"), TeamRemoved(\"dotnet-contrib-approvers\"), TeamRemoved(\"semconv-http-approvers\"), TeamRemoved(\"profiling-triagers\"), TeamRemoved(\"python-approvers\"), TeamRemoved(\"semconv-event-approvers\"), TeamRemoved(\"sig-developer-experience-maintainers\"), TeamRemoved(\"docs-cn-approvers\"), TeamRemoved(\"semconv-k8s-approvers\"), TeamRemoved(\"semconv-jvm-approvers\"), TeamRemoved(\"sandbox-web-js-maintainers\"), TeamRemoved(\"go-triagers\"), TeamRemoved(\"collector-approvers\"), TeamRemoved(\"swift-approvers\"), TeamRemoved(\"rust-approvers\"), TeamRemoved(\"network-maintainers\"), TeamRemoved(\"dotnet-contrib-triagers\"), TeamRemoved(\"collector-contrib-approvers\"), TeamRemoved(\"semconv-dotnet-approver\"), TeamRemoved(\"proto-go-maintainers\"), TeamRemoved(\"governance-committee\"), TeamRemoved(\"dotnet-instrumentation-approvers\"), TeamRemoved(\"sig-security-maintainers\"), TeamRemoved(\"lambda-extension-maintainers\"), TeamRemoved(\"php-approvers\"), TeamRemoved(\"java-contrib-approvers\"), TeamRemoved(\"sqlcommenter-maintainers\"), TeamRemoved(\"semconv-container-approvers\"), TeamRemoved(\"demo-maintainers\"), TeamRemoved(\"sig-end-user-approvers\"), TeamRemoved(\"erlang-approvers\"), TeamRemoved(\"specs-semconv-approvers\"), TeamRemoved(\"helm-approvers\"), TeamRemoved(\"python-maintainers\"), TeamRemoved(\"profiling-maintainers\"), TeamRemoved(\"dotnet-instrumentation-maintainers\"), TeamRemoved(\"java-instrumentation-approvers\"), TeamAdded(Team { name: \"project-infra\", display_name: None, maintainers: [\"austinlparker\", \"trask\"], members: [\"adrielp\", \"jaronoff97\"], annotations: {} })], repositories: [TeamAdded(\"people\", \"project-infra\", Admin)] }"},"target":"clowarden_core::services::github","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.633766Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"ebpf-profiler-maintainers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634039Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"php-triagers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634048Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"configuration-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634055Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"dotnet-instrumentation-triagers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634061Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"swift-maintainers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634066Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"go-instrumentation-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634071Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"sig-mainframe-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634076Z","level":"DEBUG","fields":{"message":"something went wrong applying change","service":"github","kind":"team-removed","extra":"{\"team_name\":\"configuration-maintainers\"}","error":"HTTP Error. Code: 404 Not Found, message: {\"message\":\"Not Found\",\"documentation_url\":\"https://docs.github.com/rest/teams/teams#delete-a-team\",\"status\":\"404\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634083Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"sig-developer-experience-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634088Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"android-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634093Z","level":"DEBUG","fields":{"message":"something went wrong applying change","service":"github","kind":"team-removed","extra":"{\"team_name\":\"php-maintainers\"}","error":"HTTP Error. Code: 404 Not Found, message: {\"message\":\"Not Found\",\"documentation_url\":\"https://docs.github.com/rest/teams/teams#delete-a-team\",\"status\":\"404\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634099Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"proto-go-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634104Z","level":"DEBUG","fields":{"message":"something went wrong applying change","service":"github","kind":"team-removed","extra":"{\"team_name\":\"android-maintainers\"}","error":"HTTP Error. Code: 404 Not Found, message: {\"message\":\"Not Found\",\"documentation_url\":\"https://docs.github.com/rest/teams/teams#delete-a-team\",\"status\":\"404\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634110Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"java-instrumentation-maintainers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634115Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"docs-es-maintainers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634120Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"sig-end-user-triagers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634126Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"java-triagers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634134Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"erlang-contrib-maintainers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634140Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"cpp-contrib-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634146Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"cpp-maintainers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634151Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"specs-semconv-maintainers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634156Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"specs-triagers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634164Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"network-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634169Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"ruby-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634174Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"sig-contributor-experience-maintainers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634180Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"docs-ja-maintainers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634185Z","level":"DEBUG","fields":{"message":"something went wrong applying change","service":"github","kind":"team-removed","extra":"{\"team_name\":\"technical-committee\"}","error":"HTTP Error. Code: 404 Not Found, message: {\"message\":\"Not Found\",\"documentation_url\":\"https://docs.github.com/rest/teams/teams#delete-a-team\",\"status\":\"404\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634191Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"semconv-system-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634196Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"collector-triagers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634200Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"semconv-llm-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634205Z","level":"DEBUG","fields":{"message":"something went wrong applying change","service":"github","kind":"team-removed","extra":"{\"team_name\":\"collector-maintainers\"}","error":"HTTP Error. Code: 404 Not Found, message: {\"message\":\"Not Found\",\"documentation_url\":\"https://docs.github.com/rest/teams/teams#delete-a-team\",\"status\":\"404\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634212Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"lambda-extension-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634217Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"otel-elections\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634222Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"ebpf-profiler-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634227Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"docs-ja-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634232Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"go-maintainers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634238Z","level":"DEBUG","fields":{"message":"something went wrong applying change","service":"github","kind":"team-removed","extra":"{\"team_name\":\"sig-mainframe-maintainers\"}","error":"HTTP Error. Code: 404 Not Found, message: {\"message\":\"Not Found\",\"documentation_url\":\"https://docs.github.com/rest/teams/teams#delete-a-team\",\"status\":\"404\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634244Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"swift-triagers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634249Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"opamp-spec-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634254Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"ruby-contrib-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634259Z","level":"DEBUG","fields":{"message":"something went wrong applying change","service":"github","kind":"team-removed","extra":"{\"team_name\":\"sig-end-user-maintainers\"}","error":"HTTP Error. Code: 404 Not Found, message: {\"message\":\"Not Found\",\"documentation_url\":\"https://docs.github.com/rest/teams/teams#delete-a-team\",\"status\":\"404\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634266Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"semconv-messaging-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634271Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"rust-maintainers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634276Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"cpp-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634281Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"assign-reviewers-action-maintainers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634286Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"docs-pt-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634291Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"operator-maintainers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634297Z","level":"DEBUG","fields":{"message":"something went wrong applying change","service":"github","kind":"team-removed","extra":"{\"team_name\":\"go-instrumentation-maintainers\"}","error":"HTTP Error. Code: 404 Not Found, message: {\"message\":\"Not Found\",\"documentation_url\":\"https://docs.github.com/rest/teams/teams#delete-a-team\",\"status\":\"404\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634303Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"java-contrib-triagers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634309Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"go-instrumentation-triagers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634314Z","level":"DEBUG","fields":{"message":"change applied","service":"github","kind":"team-removed","extra":"{\"team_name\":\"operator-approvers\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634319Z","level":"DEBUG","fields":{"message":"something went wrong applying change","service":"github","kind":"team-removed","extra":"{\"team_name\":\"docs-pt-maintainers\"}","error":"HTTP Error. Code: 404 Not Found, message: {\"message\":\"Not Found\",\"documentation_url\":\"https://docs.github.com/rest/teams/teams#delete-a-team\",\"status\":\"404\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}
server {"timestamp":"2024-09-18T15:50:29.634326Z","level":"DEBUG","fields":{"message":"something went wrong applying change","service":"github","kind":"team-removed","extra":"{\"team_name\":\"java-approvers\"}","error":"HTTP Error. Code: 404 Not Found, message: {\"message\":\"Not Found\",\"documentation_url\":\"https://docs.github.com/rest/teams/teams#delete-a-team\",\"status\":\"404\"}"},"target":"clowarden_server::jobs","span":{"org":"open-telemetry","name":"handle_reconcile_job"},"spans":[{"org":"open-telemetry","name":"handle_reconcile_job"}]}

[tegioz]

Hi @austinlparker

This is something expected. As mentioned in the how it works section of the repository's readme file, there are on-demand and periodic reconciliation jobs. On-demand reconciliation jobs take care of applying changes in PRs, whereas periodic reconciliation jobs ensure the current state matches the desired state. Without them, we wouldn't be able to correct changes applied directly in GitHub. And as soon as the service starts, the first periodic reconciliation job is run.

CLOWarden includes a CLI tool that can assist with the initial setup. You can use the diff subcommand to see the changes CLOWarden will apply as soon as it runs based on the configuration provided (this would have helped catch this problem in advance), or the generate one to automatically prepare a configuration file from the current state of the service (which should produce a configuration file with minimal or no changes to be applied, that can be reviewed with a diff).

In any case, I'm sorry you run into some problems. We'll make this clearer in the README file (repository and chart) to help other users that may encounter the same issue.

[austinlparker]

When we tried this in a separate org, it didn't remove existing teams -- you're just saying that there was some sort of, like, timing issue where the periodic reconciliation didn't happen because the service wasn't restarted?

I think you should make this significantly more clear, yes, because it's not obvious at all.

[tegioz]

Periodic reconciliations happen every hour, so it could be indeed a timing issue.

We'll do 👍

[trask]

Our expectation was that ... it wouldn't change anything until we merged the PR

This is something expected

I'm confused, so anyone can send a PR to the repo and without us reviewing or approving it blow away all our org teams?

the docs say:

changes are applied immediately once the pull request is approved and merged

[tegioz]

I'm confused, so anyone can send a PR to the repo and without us reviewing or approving it blow away all our org teams?

the docs say:

changes are applied immediately once the pull request is approved and merged

No, changes proposed in a pull request need to be approved and merged.

But this is not what caused this incident. The reason is that CLOWarden periodically ensures that the desired state as declared in the config file matches the current state in GitHub. The problem was that the config file used initially did not include all the resources available in GitHub. So when CLOWarden run a periodic reconciliation job (when it starts and every hour), it applied the necessary changes to make the desired state match the current. Unfortunately in this case, it deleted content. But it has nothing to do with the fact that a PR was created, that didn't matter.

Using the CLI tool, you can always run a diff (basically a dry-run) before launching the service to understand what will it do. You can also generate a config file from the current state using the generate subcommand to help you get started.

This is mentioned in a couple of places in the README file:

Sometimes, however, this may not be enough. Changes can be applied manually to the service bypassing the configuration files (i.e. from the GitHub settings UI), and CLOWarden still needs to make sure that the actual state matches the desired state documented in the configuration files. So in addition to on-demand reconciliation jobs, CLOWarden runs periodic ones to ensure everything is all right all the time.

We also have a full section for the CLI tool in that README, and it's referenced from the Helm chart README as well.

But in any case, we'll make this clearer, of course.