ADO API responds with GitObjectDoesNotExistException when pushing file changes to branch

[grixxie]

Hi,

I am facing the following problem with my dependabot (v2) pipeline. My pipeline looks like so:

stages:
- stage: dependabot
  jobs:
  - job: dependabot
    pool:
      name: my-selfhosted-pool
    steps:
    - task: dependabot@2
      displayName: 'Run Dependabot'
      inputs:
        setAutoComplete: true
        abandonUnwantedPullRequests: false
        azureDevOpsAccessToken: $(System.AccessToken)
        gitHubAccessToken: $(GHTOKEN)

My dependabot.yml config looks like so:

version: 2
updates:
  - package-ecosystem: "nuget"
    directory: "/"
    open-pull-requests-limit: 10
    target-branch: 'dependabot/releases/release-15'
    schedule:
      interval: "daily"

My pipeline is erroring with the following error snippet, when it tries to call the /pushes endpoint of my repo via the ADO API to push the file changed. It is complaining that an object does not exist; I am not sure what this object is refering to. My release-15 branch mentioned in my dependabot.yml definitely exists and is a different ID to the one mentioned in the error...
The error:

Creating pull request 'Bump Microsoft.AspNetCore.SpaServices.Extensions from 8.0.8 to 8.0.10'...
 - Pushing 1 file change(s) to branch 'dependabot/nuget/dependabot/releases/release-15/Microsoft.AspNetCore.SpaServices.Extensions-8.0.10'...
🌎 🠊 [POST] https://dev.azure.com/ORG/PROJECT/_apis/git/repositories/REPO/pushes
🌎 🠈 [500] Internal Server Error
{"$id":"1","innerException":null,"message":"TF401035: The object '5eae8e97b8105c3b55acb7ed3d41233062e2bcf7' does not exist.","typeName":"Microsoft.TeamFoundation.Git.Server.GitObjectDoesNotExistException, Microsoft.TeamFoundation.Git.Server","typeKey":"GitObjectDoesNotExistException","errorCode":0,"eventId":3000}
##[error]Failed to create pull request: Error: Request to 'https://dev.azure.com/ORG/PROJECT/_apis/git/repositories/REPO/pushes' failed: 500 Internal Server Error
Error: Request to 'https://dev.azure.com/ORG/PROJECT/_apis/git/repositories/REPO/pushes' failed: 500 Internal Server Error
    at AzureDevOpsWebApiClient.restApiRequest (/agent/_work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1000/utils/azure-devops/AzureDevOpsWebApiClient.js:521:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async AzureDevOpsWebApiClient.restApiPost (/agent/_work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1000/utils/azure-devops/AzureDevOpsWebApiClient.js:498:16)
    at async AzureDevOpsWebApiClient.createPullRequest (/agent/_work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1000/utils/azure-devops/AzureDevOpsWebApiClient.js:156:26)
    at async DependabotOutputProcessor.process (/agent/_work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1000/utils/dependabot-cli/DependabotOutputProcessor.js:67:42)
    at async DependabotCli.update (/agent/_work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1000/utils/dependabot-cli/DependabotCli.js:113:51)
    at async run (/agent/_work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1000/index.js:75:57)

Please advise if I am doing something wrong. If you need any more information then please let me know and I will provide. Thank you.
If anyone else is also experiencing this issue please respond!

[rhyskoedijk]

@grixxie thanks for the report;
do you have an existing branch named dependabot/releases/release-15? If so, I would guess that this is causing problems as Dependabot is wanting to use the dependabot/ prefix for the pull requests it is trying to create. If not, do you know what branch the commit 5eae8e97b8105c3b55acb7ed3d41233062e2bcf7 belongs to? Is it your target branch or another branch?

If you haven't already tried and it is possible to do so, I'd recommend renaming any branches starting with dependabot/; I'll look at adding a config option to change this prefix as it sounds like there are a few people who are already using this prefix.

EDIT: Branch name is not the cause of the issue; tested below

[rhyskoedijk]

just a follow up to this; I've tried to reproduce your issue by doing the following:

• Run update targeting default branch when an unrelated branch named dependabot/releases/release-15 already exists;
• Run update targeting dependabot/releases/release-15 when dependabot/releases/release-15 already exists;
• Run update targeting dependabot/releases/release-15 when dependabot/releases/release-15 does not exist;

All of these scenarios work without error; I was not able to reproduce the issue.

Are you able to provide any more information that would help reproduce your issue? Can you identify what branch git commit 5eae8e97b8105c3b55acb7ed3d41233062e2bcf7 belongs to?

[grixxie]

Hi - thanks for the response.

I also tested rerunning v2 with changes to the target branch name and ran into my same issue.

To answer your question, I have no idea where that commit is coming from, I could not find it when I was doing my testing yesterday. I will be doing some more testing later today though so will let you know the outcome of that.

You not being able to reproduce this issue definitely makes it seem like I'm doing something not quite right though...

[DaleMckeown]

I have also just tried to use v2 for the first time and and running into this issue.

Creating pull request 'Bump the npm group in /{folderName} with 17 updates'...
 - Pushing 1 file change(s) to branch 'dependabot/npm_and_yarn/dependabot/dependabot_grouping_v2/{folderName}/npm-d631ece976'...
🌎 🠊 [POST] https://dev.azure.com/{orgName}/{projectName}/_apis/git/repositories/{repoName}/pushes
🌎 🠈 [500] Internal Server Error
{"$id":"1","innerException":null,"message":"TF401035: The object 'abdceb5590885eeb4810012ade73d8125ae03864' does not exist.","typeName":"Microsoft.TeamFoundation.Git.Server.GitObjectDoesNotExistException, Microsoft.TeamFoundation.Git.Server","typeKey":"GitObjectDoesNotExistException","errorCode":0,"eventId":3000}
##[error]Failed to create pull request: Error: Request to 'https://dev.azure.com/{orggName}/{projectName}/_apis/git/repositories/{projectName}/pushes' failed: 500 Internal Server Error
##[debug]Processed: ##vso[task.issue type=error;source=TaskInternal;]Failed to create pull request: Error: Request to 'https://dev.azure.com/{orggName}/{projectName}/_apis/git/repositories/{repoName}/pushes' failed: 500 Internal Server Error
Error: Request to 'https://dev.azure.com/{orgName}/{projectName}/_apis/git/repositories/{repoName}/pushes' failed: 500 Internal Server Error
    at AzureDevOpsWebApiClient.restApiRequest (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1003/utils/azure-devops/AzureDevOpsWebApiClient.js:521:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async AzureDevOpsWebApiClient.restApiPost (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1003/utils/azure-devops/AzureDevOpsWebApiClient.js:498:16)
    at async AzureDevOpsWebApiClient.createPullRequest (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1003/utils/azure-devops/AzureDevOpsWebApiClient.js:156:26)
    at async DependabotOutputProcessor.process (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1003/utils/dependabot-cli/DependabotOutputProcessor.js:67:42)
    at async DependabotCli.update (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1003/utils/dependable-cli/DependabotCli.js:113:51)
    at async run (/home/vsts/work/_tasks/dependabot_d98b873d-cf18-41eb-8ff5-234f14697896/2.36.1003/index.js:75:57)

I also cannot find that commit ID at all, it must be one being created locally by Dependabot? I've ran the job multiple times and each time the objectId is different.

The branch dependabot/dependabot_grouping_v2 definitely exists.

When running this, I have two package ecosystems. NuGet, which runs successfully, and then NPM, which fails.

In my dependabot config file, I am attempting to use groups:

version: 2
registries:
  nuget-internal-packages:
    type: nuget-feed
    url: https://pkgs.dev.azure.com/{orgName}/_packaging/{projectName}/nuget/v3/index.json
    token: 'PAT:${{PAT}}'
  nuget-public:
    type: nuget-feed
    url: https://api.nuget.org/v3/index.json
  npm-internal-packages:
    type: npm-registry
    url: https://pkgs.dev.azure.com/{orgName}/_packaging/{projectName}/npm/registry/
    token: 'PAT:${{PAT}}'
  npm-public:
    type: npm-registry
    url: https://registry.npmjs.org

updates:
  - package-ecosystem: nuget
    directory: '/'
    registries:
      - nuget-internal-packages
      - nuget-public
    schedule:
      interval: weekly
    open-pull-requests-limit: 50
    target-branch: targetBranchPlaceholder
    ignore:
      - dependency-name: 'Microsoft.AspNetCore'
        update-types: ['version-update:semver-major']
      - dependency-name: 'Microsoft.AspNetCore.*'
        update-types: ['version-update:semver-major']
      - dependency-name: 'UoL.*'
        update-types: ['version-update:semver-major']

  - package-ecosystem: npm
    directory: npmDirectoryPlaceholder
    registries:
      - npm-internal-packages
      - npm-public
    schedule:
      interval: weekly
    open-pull-requests-limit: 50
    target-branch: targetBranchPlaceholder
    groups:
      npm:
        applies-to: version-updates
        patterns:
        - "*"
        update-types:
        - "minor"
        - "patch"

[grixxie]

@rhyskoedijk I have found the same as @DaleMckeown post above; running the command git branch -a --contains <SHA> returns no results. I also think that this commit is one being created locally by Dependabot, but has failed to push to remote. This would explain why ADO API returns 500 when trying to create the PR.

[DaleMckeown]

If I flip the order of my package ecosystems to run NPM first and then NuGet, NPM runs successfully and NuGet errors instead. So it seems to be an issue related to processing multiple package ecosystems in the same Dependabot run.

Also, removing the grouping configuration doesn't have any effect, the same error is thrown.

Just tested again with the NPM config individually, and it works fine. So it's definitely related to having multiple package ecosystems in my case.

[rhyskoedijk]

If I flip the order of my package ecosystems to run NPM first and then NuGet, NPM runs successfully and NuGet errors instead. So it seems to be an issue related to processing multiple package ecosystems in the same Dependabot run.

Thanks @DaleMckeown, that is a good lead and something that I have not tested yet; I will investigate.

[grixxie]

If I flip the order of my package ecosystems to run NPM first and then NuGet, NPM runs successfully and NuGet errors instead. So it seems to be an issue related to processing multiple package ecosystems in the same Dependabot run.

Thanks @DaleMckeown, that is a good lead and something that I have not tested yet; I will investigate.

This may be a good lead to find the root of the problem. However, just to point out my dependabot.yml config in the description of this issue, I'm only using a single package ecosystem (NuGet) and still running into this issue!

[rhyskoedijk]

This may be a good lead to find the root of the problem. However, just to point out my dependabot.yml config in the description of this issue, I'm only using a single package ecosystem (NuGet) and still running into this issue!

Understood; At this point I'm just looking for a way to reproduce the error, even if it isn't your original scenario.
I have just tried using a run with multiple package ecosystems (NPM and NuGet), but unfortunately this didn't fail for me either.

I am out of ideas for now; If anybody is able to produce a minimal reproduction of their dependabot.yml and *.csproj that causes the error, I will revisit this.

[DaleMckeown]

@rhyskoedijk I've scoured the logs again, not sure if this helps at all, but here is the first time the errant commit sha appears in the logs:

updater | 2024/10/22 11:19:08 DEBUG Initializing the background worker with 2 threads
updater | 2024/10/22 11:19:10 INFO <job_update_1_npm_and_yarn_all> Starting job processing
updater | 2024/10/22 11:19:10 INFO <job_update_1_npm_and_yarn_all> Job definition: {"job":{"package-manager":"npm_and_yarn","allowed-updates":[{"dependency-type":"all"}],"debug":true,"dependency-groups":[{"name":"npm","applies-to":"version-updates","rules":{"patterns":["*"],"update-types":["minor","patch"]}}],"dependencies":null,"dependency-group-to-refresh":null,"existing-pull-requests":[],"existing-group-pull-requests":[],"experiments":null,"ignore-conditions":[],"lockfile-only":false,"requirements-update-strategy":null,"security-advisories":[],"security-updates-only":false,"source":{"provider":"azure","repo":"UoLIct/UniLincoln/_git/UniLincoln","directory":"/UniLincolnUI","branch":"dependabot/dependabot_grouping_v2","hostname":"dev.azure.com","api-endpoint":"[https://dev.azure.com:/"},"update-subdependencies":true,"updating-a-pull-request":false,"vendor-dependencies":false,"reject-external-code":false,"repo-private":false,"commit-message-options":null,"credentials-metadata":[{"host":"dev.azure.com","type":"git_source"},{"type":"nuget_feed","url":"https://pkgs.dev.azure.com/UoLIct/_packaging/Packages/nuget/v3/index.json"},{"type":"nuget_feed","url":"https://api.nuget.org/v3/index.json"},{"registry":"pkgs.dev.azure.com/UoLIct/_packaging/Packages/npm/registry/","type":"npm_registry"},{"registry":"registry.npmjs.org","type":"npm_registry"}],"max-updater-run-time":0}}](https://dev.azure.com/%22%7D,%22update-subdependencies%22:true,%22updating-a-pull-request%22:false,%22vendor-dependencies%22:false,%22reject-external-code%22:false,%22repo-private%22:false,%22commit-message-options%22:null,%22credentials-metadata%22:[%7B%22host%22:%22dev.azure.com%22,%22type%22:%22git_source%22%7D,%7B%22type%22:%22nuget_feed%22,%22url%22:%22https://pkgs.dev.azure.com/UoLIct/_packaging/Packages/nuget/v3/index.json%22%7D,%7B%22type%22:%22nuget_feed%22,%22url%22:%22https://api.nuget.org/v3/index.json%22%7D,%7B%22registry%22:%22pkgs.dev.azure.com/UoLIct/_packaging/Packages/npm/registry/%22,%22type%22:%22npm_registry%22%7D,%7B%22registry%22:%22registry.npmjs.org%22,%22type%22:%22npm_registry%22%7D],%22max-updater-run-time%22:0%7D%7D)
updater | 2024/10/22 11:19:11 INFO <job_update_1_npm_and_yarn_all> Base commit SHA: abdceb5590885eeb4810012ade73d8125ae03864
updater | 2024/10/22 11:19:11 INFO <job_update_1_npm_and_yarn_all> Finished job processing
updater | 2024/10/22 11:19:11 DEBUG Killing session flusher
updater | 2024/10/22 11:19:11 DEBUG Shutting down background worker
updater | 2024/10/22 11:19:12 DEBUG Initializing the background worker with 2 threads

It then appears after finding the first package that needs an update. In my case, Quasar:

updater | 2024/10/22 11:19:40 INFO <job_update_1_npm_and_yarn_all> Updating @quasar/extras from 1.16.11 to 1.16.12
updater | 2024/10/22 11:19:40 DEBUG <job_update_1_npm_and_yarn_all> Dependencies updated:
updater | 2024/10/22 11:19:40 DEBUG <job_update_1_npm_and_yarn_all>  - @quasar/extras ( 1.16.11 to 1.16.12 )
updater | 2024/10/22 11:19:40 DEBUG <job_update_1_npm_and_yarn_all> Dependency files updated:
updater | 2024/10/22 11:19:40 DEBUG <job_update_1_npm_and_yarn_all>   - /UniLincolnUI/package.json ( Changed 1 times )
updater | 2024/10/22 11:19:40 DEBUG <job_update_1_npm_and_yarn_all>   - /UniLincolnUI/.npmrc 
updater | 2024/10/22 11:19:40 DEBUG <job_update_1_npm_and_yarn_all> Storing change to workspace: Updating @quasar/extras
updater | 2024/10/22 11:19:40 DEBUG <job_update_1_npm_and_yarn_all> [workspace] store_change - before: Initial SHA: abdceb5590885eeb4810012ade73d8125ae03864
updater | 2024/10/22 11:19:40 DEBUG <job_update_1_npm_and_yarn_all> [workspace] store_change - after: 90f51d7e Updating @quasar/extras

It then runs create_pull_request using this base commit successfully:

  proxy | 2024/10/22 11:23:08 [315] POST http://host.docker.internal:42793/update_jobs/update_1_npm_and_yarn_all/create_pull_request
{"data":{"base-commit-sha":"abdceb5590885eeb4810012ade73d8125ae03864","dependencies": ...}}
  proxy | 2024/10/22 11:23:08 [315] 200 http://host.docker.internal:42793/update_jobs/update_1_npm_and_yarn_all/create_pull_request

It repeats the above for each NPM package that needs an update.

It then marks the commit as processed, and outputs the changes detected:

{"data":{"base-commit-sha":"abdceb5590885eeb4810012ade73d8125ae03864"},"type":"mark_as_processed"}
  proxy | 2024/10/22 11:24:59 [290] PATCH http://host.docker.internal:42793/update_jobs/update_1_npm_and_yarn_all/mark_as_processed
  proxy | 2024/10/22 11:24:59 [290] 200 http://host.docker.internal:42793/update_jobs/update_1_npm_and_yarn_all/mark_as_processed
updater | 2024/10/22 11:24:59 INFO <job_update_1_npm_and_yarn_all> Finished job processing
updater | 2024/10/22 11:24:59 INFO Results:
updater | +------------------------------------------------------------------------------------------------------------------------------------+
updater | |                                                Changes to Dependabot Pull Requests                                                 |
updater | +---------+--------------------------------------------------------------------------------------------------------------------------+
updater | | created | @quasar/extras ( from 1.16.11 to 1.16.12 ), @uol/cwd ( from 7.0.0 to 7.0.1 ), @uol/cwd-fontawesome-pro ( from 7.0.0 t... |
updater | | created | @vueuse/core ( from 10.11.1 to 11.1.0 )                                                                                  |
updater | | created | vue-shepherd ( from 3.0.0 to 4.1.0 )                                                                                     |
updater | | created | @quasar/app-vite ( from 2.0.0-beta.14 to 2.0.0-beta.24 )                                                                 |
updater | | created | @types/node ( from 20.14.1 to 22.7.8 )                                                                                   |
updater | | created | @typescript-eslint/eslint-plugin ( from 7.12.0 to 8.11.0 )                                                               |
updater | | created | @typescript-eslint/parser ( from 7.12.0 to 7.18.0 )                                                                      |
updater | +---------+--------------------------------------------------------------------------------------------------------------------------+

It then outputs the create_pull_request response:

Processing output 'create_pull_request' with data: {
  'base-commit-sha': 'abdceb5590885eeb4810012ade73d8125ae03864',
...

Then it errors:

Creating pull request 'Bump the npm group in /UniLincolnUI with 17 updates'...
 - Pushing 1 file change(s) to branch 'dependabot/npm_and_yarn/dependabot/dependabot_grouping_v2/UniLincolnUI/npm-d631ece976'...
🌎 🠊 [POST] https://dev.azure.com/{orgName}/{projectName}/_apis/git/repositories/{repoName}/pushes
🌎 🠈 [500] Internal Server Error
{"$id":"1","innerException":null,"message":"TF401035: The object 'abdceb5590885eeb4810012ade73d8125ae03864' does not exist.","typeName":"Microsoft.TeamFoundation.Git.Server.GitObjectDoesNotExistException, Microsoft.TeamFoundation.Git.Server","typeKey":"GitObjectDoesNotExistException","errorCode":0,"eventId":3000}

[beytun]

I am facing the exact issue. My pipeline was successful on 14th October. The next execution on 21st October failed with this issue. I see a new version of this task has been released meanwhile.

Successful pipeline

Task : Dependabot
Description : Automatically update dependencies and vulnerabilities in your code using Dependabot CLI
Version : 2.35.955
Author : Tingle Software
Help : https://github.com/tinglesoftware/dependabot-azure-devops/issues

Failed pipeline

Task : Dependabot
Description : Automatically update dependencies and vulnerabilities in your code using Dependabot CLI
Version : 2.36.1003
Author : Tingle Software
Help : https://github.com/tinglesoftware/dependabot-azure-devops/issues

Update:
I just tried pinning the version 2.35.955 in my pipeline and it worked fine. This can be used a workaround till the issue is found out and fixed.