Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is it possible to require deploys in a specific order? #303

Closed
johnseekins-pathccm opened this issue Sep 12, 2024 · 30 comments
Closed

Is it possible to require deploys in a specific order? #303

johnseekins-pathccm opened this issue Sep 12, 2024 · 30 comments
Assignees
Labels
enhancement New feature or request question Further information is requested or the issue is a question

Comments

@johnseekins-pathccm
Copy link

Details

Let's say we have two environments: environment_targets: Staging,Production. This works fine and deploys work against both environments just fine. But what if we want to ensure that a user deploys to Staging before they're allowed to deploy to Production? Essentially, is there a way to, similar to requiring reviews, require an environment be successful before another environment can be deployed to?

@johnseekins-pathccm johnseekins-pathccm added the question Further information is requested or the issue is a question label Sep 12, 2024
@GrantBirki
Copy link
Member

👋 Hey @johnseekins-pathccm, thanks for opening this issue! This is not the first time I have heard this request and I think its something that warrants a deep dive now. I will schedule some time to investigate if this can be done, and if so... implement it. Keep some 👀 on this issue and I'll update it as I go! It might not be right away as my schedule is pretty tight but I'll get to it when I'm able. Thank you! 🙇

@GrantBirki GrantBirki self-assigned this Sep 18, 2024
@GrantBirki GrantBirki added the enhancement New feature or request label Sep 18, 2024
@GrantBirki
Copy link
Member

GrantBirki commented Sep 20, 2024

@johnseekins-pathccm I was actually able to complete this faster than expected. I have published a pre-release v9.8.0. Would you be able to test this out and let me know how it works? Thanks! 🙇

@johnseekins-pathccm
Copy link
Author

Sorry for the bad message. This works just fine. Thanks for the quick turn-around!

@johnseekins-pathccm
Copy link
Author

Actually...no. The order enforcing works too well:
image

@johnseekins-pathccm
Copy link
Author

For some additional clarification:
sha matches in both the successful Staging deploy and the failed Production deploy.

@GrantBirki
Copy link
Member

@johnseekins-pathccm would you be able to copy/paste your actions config for the branch-deploy step here or re-run your job with debug logs and paste those too? Feel free to redact anything that might be considered sensitive and I'll take a look to see what's up.

@GrantBirki
Copy link
Member

One thing that I am noticing is that I don't see one of the little green rockets (🚀) in your screenshot. Usually if a branch successfully deploys it will say so on your pull request just below that "Deployment Results ✅" comment. I don't see that in your case so it would make sense that the deployment to Production would fail because Staging was never fully deployed.

Here is an example of it working for me:

Image

@johnseekins-pathccm
Copy link
Author

Fair point, but here's that rocket:
Screenshot 2024-09-23 at 10 45 51 AM

@GrantBirki
Copy link
Member

Hmm that is quite strange then.. Do you have logs or your action config that you can copy/paste here? 🙇

@johnseekins-pathccm
Copy link
Author

johnseekins-pathccm commented Sep 23, 2024

Here's the action config:

- uses: github/[email protected]
   id: branch-deploy
   with:
     environment_targets: Staging,Production
     enforced_deployment_order: Staging,Production
     # default deployment environment
     environment: Staging
     production_environments: Production
     skip_reviews: Staging
     skip_completing: true
     sticky_locks: true

@GrantBirki
Copy link
Member

Ah! I see. So it looks like you are completing the deployments via a custom step with skip_completing: true then?

Are you able to share more details about how you are setting your deployment to active? I bet there is a bug somewhere in there and when branch-deploy goes to check its missing some metadata that's causing the new checks to fail...

@johnseekins-pathccm
Copy link
Author

johnseekins-pathccm commented Sep 23, 2024

Here's more of the code around "finishing" a deploy.

    - if: "inputs.noop != 'true'"
      name: Set Deployment Status
      id: set-status
      shell: bash
      env:
        GH_TOKEN: ${{ github.token }}
      run: |
        gh api --method POST \
          "repos/${{ inputs.repository }}/deployments/${{ inputs.deployment-id }}/statuses" \
          -f environment=${{ inputs.environment }} \
          -f state=${{ inputs.deployment-status }}

    # Remove the trigger reaction added to the user's comment.
    - name: Remove Trigger Reaction
      id: remove-reaction
      shell: bash
      env:
        GH_TOKEN: ${{ github.token }}
      run: |
        gh api --method DELETE \
          "repos/${{ inputs.repository }}/issues/comments/${{ inputs.comment-id }}/reactions/${{ inputs.initial_reaction_id }}"

    # Add a new reaction based on if the deployment succeeded or failed.
    - name: Add Reaction
      id: add-reaction
      uses: GrantBirki/comment@v2
      env:
        REACTION: ${{ inputs.deployment-status == 'success' && 'rocket' || '-1' }}
        GH_TOKEN: ${{ github.token }}
      with:
        comment-id: ${{ inputs.comment-id }}
        reactions: ${{ inputs.deployment-status == 'success' && 'rocket' || '-1' }}

    # Add a success comment, including the plan/apply output (if present).
    - if: "inputs.deployment-status == 'success'"
      name: Add Success Comment
      id: success-comment
      uses: actions/github-script@v7
      env:
        GH_TOKEN: ${{ github.token }}
      with:
        script: |
          await github.rest.issues.createComment({
            issue_number: context.issue.number,
            owner: context.repo.owner,
            repo: context.repo.repo,
            body: `### Deployment Results :white_check_mark:

          **${{ inputs.actor }}** successfully ${ ${{ inputs.noop }} === 'true' ? '**noop** deployed' : 'deployed' } branch \`${{ inputs.ref }}\` to **${{ inputs.environment }}**`
          })

    # Add a failure comment, including the plan/apply output (if present).
    - if: "inputs.deployment-status == 'failure'"
      name: Add Failure Comment
      id: failure-comment
      uses: actions/github-script@v7
      env:
        GH_TOKEN: ${{ github.token }}
      with:
        script: |
          await github.rest.issues.createComment({
            issue_number: context.issue.number,
            owner: context.repo.owner,
            repo: context.repo.repo,
            body: `### Deployment Results :x:

          **${{ inputs.actor }}** had a failure when ${ inputs.noop === 'true' ? '**noop** deploying' : 'deploying' } branch \`${{ inputs.ref }}\` to **${{ inputs.environment }}**`
          })

    # If the deployment failed, fail the workflow.
    - if: "inputs.deployment-status == 'failure'"
      name: Fail Workflow
      shell: bash
      id: fail-workflow
      env:
        GH_TOKEN: ${{ github.token }}
      run: |
        echo "There was a deployment problem...failing the workflow!"
        exit 1

Which is essentially following the docs https://github.com/github/branch-deploy/blob/main/docs/examples.md#multiple-jobs-with-github-environments

@johnseekins-pathccm
Copy link
Author

Here's (some of) the output of the staging run:
Screenshot 2024-09-23 at 11 00 52 AM

And the similar prod output:
Screenshot 2024-09-23 at 10 59 02 AM

I think you're right that something isn't getting set somewhere...

@johnseekins-pathccm
Copy link
Author

It looks like (based again on the docs) that I'm setting state to success or failure, never active.

@GrantBirki
Copy link
Member

It looks like (based again on the docs) that I'm setting state to success or failure, never active.

I think you have found the problem! But this leads me to more questions... what is the difference between ACTIVE and SUCCESS for a deployment and which one should we actually be using by this action? I'm going to do some digging to see if I can find out

@GrantBirki
Copy link
Member

The GraphQL docs (what the action uses to fetch but not set) indicate that there is indeed an ACTIVE field and that makes the most sense to use: https://docs.github.com/en/graphql/reference/enums#deploymentstate

@johnseekins-pathccm
Copy link
Author

Well that's fixable. Is success used at all? Or should I do res == "success" ? 'ACTIVE' : 'anythingelse'?

@GrantBirki
Copy link
Member

Alrighty, I did some investigating and I think there may be a problem in your workflow somewhere because I cannot replicate this issue in my own dev workflows.

Here are some details that I have found...

The success state is indeed the correct state that should be sent to the GitHub REST API when completing a successful deployment. For failed deployments, the failure state is also the correct one to send. So all is fine here and that isn't the problem.

The ACTIVE value is returned from GitHub's GraphQL API endpoints and is used to determine which deployment for a given environment is the "active" one. All other deployments are marked as "inactive" by comparison. So this value is also fine (used internally by the branch-deploy action here).

I have also done some debugging with helpful comments on this pull request: GrantBirki/actions-sandbox#115. You may find this debugging information useful and also helpful to reference the branch-deploy workflow that I am using to compare it to your to look for bugs.

TL;DR: I actually think things are working as expected and I'm not able to recreate the bug that you are running into.

@johnseekins-pathccm
Copy link
Author

I'm getting a very different result in API returns:

$ gh api repos/pathccm/noop_tool/deployments/1820147308/statuses 
[
  {
    "url": "https://api.github.com/repos/<org>/deployments/1820147308/statuses/4617553256",
    "id": 4617553256,
    "node_id": "DES_kwDOHjNddM8AAAABEzpFaA",
    "state": "success",
    "creator": {
      "login": "github-actions[bot]",
      "id": 41898282,
      "node_id": "MDM6Qm90NDE4OTgyODI=",
      "avatar_url": "https://avatars.githubusercontent.com/in/15368?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/github-actions%5Bbot%5D",
      "html_url": "https://github.com/apps/github-actions",
      "followers_url": "https://api.github.com/users/github-actions%5Bbot%5D/followers",
      "following_url": "https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}",
      "gists_url": "https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/github-actions%5Bbot%5D/subscriptions",
      "organizations_url": "https://api.github.com/users/github-actions%5Bbot%5D/orgs",
      "repos_url": "https://api.github.com/users/github-actions%5Bbot%5D/repos",
      "events_url": "https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}",
      "received_events_url": "https://api.github.com/users/github-actions%5Bbot%5D/received_events",
      "type": "Bot",
      "site_admin": false
    },
    "description": "",
    "environment": "Staging",
    "target_url": "",
    "created_at": "2024-09-23T14:31:33Z",
    "updated_at": "2024-09-23T14:31:33Z",
    "deployment_url": "https://api.github.com/repos/<org>/deployments/1820147308",
    "repository_url": "https://api.github.com/repos/<org>",
    "environment_url": "",
    "log_url": "",
    "performed_via_github_app": null
  },
  {
    "url": "https://api.github.com/repos/<org>/deployments/1820147308/statuses/4617473865",
    "id": 4617473865,
    "node_id": "DES_kwDOHjNddM8AAAABEzkPSQ",
    "state": "in_progress",
    "creator": {
      "login": "github-actions[bot]",
      "id": 41898282,
      "node_id": "MDM6Qm90NDE4OTgyODI=",
      "avatar_url": "https://avatars.githubusercontent.com/in/15368?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/github-actions%5Bbot%5D",
      "html_url": "https://github.com/apps/github-actions",
      "followers_url": "https://api.github.com/users/github-actions%5Bbot%5D/followers",
      "following_url": "https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}",
      "gists_url": "https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/github-actions%5Bbot%5D/subscriptions",
      "organizations_url": "https://api.github.com/users/github-actions%5Bbot%5D/orgs",
      "repos_url": "https://api.github.com/users/github-actions%5Bbot%5D/repos",
      "events_url": "https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}",
      "received_events_url": "https://api.github.com/users/github-actions%5Bbot%5D/received_events",
      "type": "Bot",
      "site_admin": false
    },
    "description": "",
    "environment": "Staging",
    "target_url": "https://github.com/<org>/actions/runs/10996318031",
    "created_at": "2024-09-23T14:23:22Z",
    "updated_at": "2024-09-23T14:23:22Z",
    "deployment_url": "https://api.github.com/repos/<org>/deployments/1820147308",
    "repository_url": "https://api.github.com/repos/<org>",
    "environment_url": "",
    "log_url": "https://github.com/<org>/actions/runs/10996318031",
    "performed_via_github_app": null
  }
]

@johnseekins-pathccm
Copy link
Author

$ gh api repos/<org>/deployments/1820147308 
{
  "url": "https://api.github.com/repos/<org>/deployments/1820147308",
  "id": 1820147308,
  "node_id": "DE_kwDOHjNddM5sfT5s",
  "task": "deploy",
  "original_environment": "Staging",
  "environment": "Staging",
  "description": null,
  "created_at": "2024-09-23T14:23:21Z",
  "updated_at": "2024-09-23T14:31:33Z",
  "statuses_url": "https://api.github.com/repos/<org>/deployments/1820147308/statuses",
  "repository_url": "https://api.github.com/repos/<org>",
  "creator": {
    "login": "github-actions[bot]",
    "id": 41898282,
    "node_id": "MDM6Qm90NDE4OTgyODI=",
    "avatar_url": "https://avatars.githubusercontent.com/in/15368?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/github-actions%5Bbot%5D",
    "html_url": "https://github.com/apps/github-actions",
    "followers_url": "https://api.github.com/users/github-actions%5Bbot%5D/followers",
    "following_url": "https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}",
    "gists_url": "https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/github-actions%5Bbot%5D/subscriptions",
    "organizations_url": "https://api.github.com/users/github-actions%5Bbot%5D/orgs",
    "repos_url": "https://api.github.com/users/github-actions%5Bbot%5D/repos",
    "events_url": "https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}",
    "received_events_url": "https://api.github.com/users/github-actions%5Bbot%5D/received_events",
    "type": "Bot",
    "site_admin": false
  },
  "sha": "2263c48cdb096b195b5b2bf82ae27d47ca518422",
  "ref": "test-ordered",
  "payload": {
    "type": "branch-deploy",
    "sha": "2263c48cdb096b195b5b2bf82ae27d47ca518422"
  },
  "transient_environment": false,
  "production_environment": false,
  "performed_via_github_app": null
}

@johnseekins-pathccm
Copy link
Author

Kinda feels like I need to say ACTIVE instead of success.

@johnseekins-pathccm
Copy link
Author

johnseekins-pathccm commented Sep 23, 2024

ACTIVE is not supported. success is clearly the correct input to the API call. This is baffling. @GrantBirki Do you have the actual graph query you ran so I can validate things on my side?

Edit: Never mind. Found the query in the code.

@GrantBirki
Copy link
Member

GrantBirki commented Sep 23, 2024

GraphQL Query:

query ($repo_owner: String!, $repo_name: String!, $environment: String!) {
      repository(owner: $repo_owner, name: $repo_name) {
        deployments(environments: [$environment], first: 1, orderBy: { field: CREATED_AT, direction: DESC }) {
          nodes {
            createdAt
            environment
            updatedAt
            id
            payload
            state
            ref {
              name
            }
            creator {
              login
            }
            commit {
              oid
            }
          }
        }
      }
    }

Variables:

{
    "repo_owner": "<owner>",
    "repo_name": "<repo>",
    "environment": "production"
}

Example results:

{
  "data": {
    "repository": {
      "deployments": {
        "nodes": [
          {
            "createdAt": "2024-09-23T18:08:28Z",
            "environment": "production",
            "updatedAt": "2024-09-23T18:08:43Z",
            "id": "DE_kwDOID9x8M5shzsE",
            "payload": "\"{\\\"type\\\":\\\"branch-deploy\\\",\\\"sha\\\":\\\"2a000a896fb9a6b2c1bbd69608d60865be22c515\\\"}\"",
            "state": "ACTIVE",
            "ref": {
              "name": "testing"
            },
            "creator": {
              "login": "github-actions"
            },
            "commit": {
              "oid": "2a000a896fb9a6b2c1bbd69608d60865be22c515"
            }
          }
        ]
      }
    }
  }
}

@GrantBirki
Copy link
Member

GrantBirki commented Sep 23, 2024

When you mentioned "getting a very different result in API returns" I was actually getting the same results in my experiment and it all looks as expected so that should be good 👍

@johnseekins-pathccm
Copy link
Author

I'm getting a different result:

$ curl -Ss -d @graphql.query -H "Authorization: bearer <>" https://api.github.com/graphql  | jq
{
  "data": {
    "repository": {
      "deployments": {
        "nodes": [
          {
            "createdAt": "2024-09-23T19:27:00Z",
            "environment": "Staging",
            "updatedAt": "2024-09-23T19:39:38Z",
            "id": "DE_kwDOHjNddM5sikiB",
            "payload": null,
            "state": "INACTIVE",
            "ref": {
              "name": "main"
            },
            "creator": {
              "login": "johnseekins-pathccm"
            },
            "commit": {
              "oid": "267c72adce090e2972f05ce8f9deab70401f43e0"
            }
          },
          {
            "createdAt": "2024-09-23T19:26:39Z",
            "environment": "Staging",
            "updatedAt": "2024-09-23T19:31:45Z",
            "id": "DE_kwDOHjNddM5sikU8",
            "payload": null,
            "state": "INACTIVE",
            "ref": {
              "name": "main"
            },
            "creator": {
              "login": "johnseekins-pathccm"
            },
            "commit": {
              "oid": "267c72adce090e2972f05ce8f9deab70401f43e0"
            }
          },
          {
            "createdAt": "2024-09-23T19:24:39Z",
            "environment": "Staging",
            "updatedAt": "2024-09-23T19:26:40Z",
            "id": "DE_kwDOHjNddM5sijND",
            "payload": null,
            "state": "INACTIVE",
            "ref": {
              "name": "main"
            },
            "creator": {
              "login": "johnseekins-pathccm"
            },
            "commit": {
              "oid": "267c72adce090e2972f05ce8f9deab70401f43e0"
            }
          },
          {
            "createdAt": "2024-09-23T19:24:39Z",
            "environment": "Staging",
            "updatedAt": "2024-09-23T19:27:00Z",
            "id": "DE_kwDOHjNddM5sijMu",
            "payload": null,
            "state": "INACTIVE",
            "ref": {
              "name": "main"
            },
            "creator": {
              "login": "johnseekins-pathccm"
            },
            "commit": {
              "oid": "267c72adce090e2972f05ce8f9deab70401f43e0"
            }
          },
          {
            "createdAt": "2024-09-23T19:24:33Z",
            "environment": "Staging",
            "updatedAt": "2024-09-23T19:39:37Z",
            "id": "DE_kwDOHjNddM5sijIf",
            "payload": "\"{\\\"type\\\":\\\"branch-deploy\\\",\\\"sha\\\":\\\"c3ef9875fcdb78177963039c2943e44a6431a328\\\"}\"",
            "state": "ACTIVE",
            "ref": {
              "name": "test-deploy-again"
            },
            "creator": {
              "login": "github-actions"
            },
            "commit": {
              "oid": "c3ef9875fcdb78177963039c2943e44a6431a328"
            }
          },
          {
            "createdAt": "2024-09-23T18:52:39Z",
            "environment": "Staging",
            "updatedAt": "2024-09-23T18:57:17Z",
            "id": "DE_kwDOHjNddM5siPaJ",
            "payload": null,
            "state": "INACTIVE",
            "ref": {
              "name": "main"
            },
            "creator": {
              "login": "johnseekins-pathccm"
            },
            "commit": {
              "oid": "5b6ddca45595519f1a3aee9b98b92d3717e242a2"
            }
          },
          {
            "createdAt": "2024-09-23T18:52:17Z",
            "environment": "Staging",
            "updatedAt": "2024-09-23T18:57:08Z",
            "id": "DE_kwDOHjNddM5siPNI",
            "payload": null,
            "state": "INACTIVE",
            "ref": {
              "name": "main"
            },
            "creator": {
              "login": "johnseekins-pathccm"
            },
            "commit": {
              "oid": "5b6ddca45595519f1a3aee9b98b92d3717e242a2"
            }
          },
          {
            "createdAt": "2024-09-23T18:51:32Z",
            "environment": "Staging",
            "updatedAt": "2024-09-23T18:52:17Z",
            "id": "DE_kwDOHjNddM5siOvK",
            "payload": null,
            "state": "INACTIVE",
            "ref": {
              "name": "main"
            },
            "creator": {
              "login": "johnseekins-pathccm"
            },
            "commit": {
              "oid": "5b6ddca45595519f1a3aee9b98b92d3717e242a2"
            }
          },
          {
            "createdAt": "2024-09-23T18:51:31Z",
            "environment": "Staging",
            "updatedAt": "2024-09-23T18:52:39Z",
            "id": "DE_kwDOHjNddM5siOuf",
            "payload": null,
            "state": "INACTIVE",
            "ref": {
              "name": "main"
            },
            "creator": {
              "login": "johnseekins-pathccm"
            },
            "commit": {
              "oid": "5b6ddca45595519f1a3aee9b98b92d3717e242a2"
            }
          },
          {
            "createdAt": "2024-09-23T18:51:23Z",
            "environment": "Staging",
            "updatedAt": "2024-09-23T19:26:11Z",
            "id": "DE_kwDOHjNddM5siOqK",
            "payload": "\"{\\\"type\\\":\\\"branch-deploy\\\",\\\"sha\\\":\\\"2263c48cdb096b195b5b2bf82ae27d47ca518422\\\"}\"",
            "state": "INACTIVE",
            "ref": null,
            "creator": {
              "login": "github-actions"
            },
            "commit": {
              "oid": "2263c48cdb096b195b5b2bf82ae27d47ca518422"
            }
          }
        ]
      }
    }
  }
}

You'll notice only some of the values (and never the most recent one) come back as ACTIVE.

I had to slightly change the query to get these results:

query { 
      repository(owner: \"org\", name: \"repo\") {
        deployments(environments: [\"Staging\", \"Production\"], first: 10, orderBy: { field: CREATED_AT, direction: DESC }) { 
          nodes { 
            createdAt
            environment
            updatedAt
            id
            payload
            state
            ref {
              name
            }
            creator {
              login
            }
            commit {
              oid
            }
          }
        }
      }
    }

@GrantBirki
Copy link
Member

You wouldn't happen to have an environment tag floating around in your workflow would ya?...

If you did, there could be the possibility that it is marking your deployment as inactive after the workflow run finishes:

name: Deployment

on:
  push:
    branches:
      - main

jobs:
  deployment:
    runs-on: ubuntu-latest
    environment: production # <--- this value right here, it doesn't play well with branch-deploy
    steps:
      - name: deploy
        # ...

@GrantBirki
Copy link
Member

Perhaps the better solution would be to fetch more results and then paginate through them. As it looks like your ACTIVE deployment isn't the very first one returned so it will never be found with the current logic. I'm not entirely sure what flows would place it lower down in the list but I think that is something we should account for and guard against

@johnseekins-pathccm
Copy link
Author

It's absolutely the case that we were stomping on environment. We wanted those fancy little progress bars in our deployment steps...
I'm testing the deploy with environment: corrected.

@johnseekins-pathccm
Copy link
Author

This was the problem. All along. Sorry for all the churn. These changes work great!

Now to figure out deploy trains with branch deploys...

@GrantBirki
Copy link
Member

We wanted those fancy little progress bars in our deployment steps...

I totally hear ya... I wish that there was more control when using the environment: input as well. If you checkout this comment, you'll find a linked issue with a large discussion around the topic as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request question Further information is requested or the issue is a question
Projects
None yet
Development

No branches or pull requests

2 participants