feat(datasource/azure-pipelines-tasks): Azure DevOps API based datasource

[bdovaz]

Changes

Initially as I do not have much experience of the Renovate API, I tried (looking at other code examples in this repository) that only changes the behavior for the case that you are running it from platform: azure because this way there is no need to create any new variable and it can use the values of RENOVATE_PLATFORM, RENOVATE_ENDPOINT and RENOVATE_TOKEN.

This PR does not solve the case that platform is a value different to azure (in that case it makes fallback to what it was already doing) because for that case I suppose that new variables will be needed and I consider that this can go in another future PR because it requires more analysis.

Context

The need comes from the fact that we currently take as a source of data the static files that are generated from this repository: https://github.com/renovatebot/azure-devops-marketplace

The problem is that task updates are not propagated at the same speed to all Azure Devops organizations so there are times when renovate mistakenly suggests updating to versions of tasks that do not exist in the organization and if you do not know why it happens it is very confusing.

There is an open discussion: #24820

Documentation (please check one with an [x])

• I have updated the documentation, or
• No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

• Code inspection only, or
• Newly added/modified unit tests, or
• No unit tests but ran on a real repository, or
• Both unit tests + ran on a real repository

[CLAassistant]

All committers have signed the CLA.

[rarkins]

Are you sure that this can replace the existing functionality in a backwards-compatible way?

How efficient is this approach? e.g. how many API calls and how many MB should it require each run?

Any caching possible?

[bdovaz]

Are you sure that this can replace the existing functionality in a backwards-compatible way?

How efficient is this approach? e.g. how many API calls and how many MB should it require each run?

Any caching possible?

Doesn't the caching already do it with the code that was there?

https://github.com/bdovaz/renovate/blob/620fa90803e995e404e29c478b1cfcd32ad028f8/lib/modules/datasource/azure-pipelines-tasks/index.ts#L81

And the backwards-compatible thing is for when these conditions are not met that it at least does what it did before:

https://github.com/bdovaz/renovate/blob/620fa90803e995e404e29c478b1cfcd32ad028f8/lib/modules/datasource/azure-pipelines-tasks/index.ts#L36

I've also added more tests to compare function so we have 100% code coverage.

[rarkins]

So it caches once per 24 hours, if the user has persistent datasource caching? (note: most people running Renovate in pipelines do not have such persistence)

I don't understand your answer to my backwards-compatibility question. Is it replacing existing logic, or adding to it? If it's replacing it, is it fully backwards compatible?

How many API calls and how many MB should it require?

[bdovaz]

So it caches once per 24 hours, if the user has persistent datasource caching? (note: most people running Renovate in pipelines do not have such persistence)

@rarkins is there any documentation on how to implement persistence? Without documentation or examples I will hardly be able to do it since it is the first time I do a PR. As I said, the persistence there was already implemented, I haven't touched anything in this PR.

I don't understand your answer to my backwards-compatibility question. Is it replacing existing logic, or adding to it? If it's replacing it, is it fully backwards compatible?

If these conditions are met: https://github.com/renovatebot/renovate/pull/32966/files#diff-2ef2acf656366f4dd2036d71f106b4d8d0b41cd546e1349942a4e40efffde808R36

It will use the new logic, otherwise it will do what it did before. The difference is that if you work against Azure DevOps and provide a PAT (personal access token), it will resolve the task versions according to your organization and not to a static file as now (https://github.com/renovatebot/azure-devops-marketplace/blob/main/azure-pipelines-builtin-tasks.json). Not all organizations propagate task updates at the same time and with the current implementation the problem is that it suggests updates that have not yet been propagated in your organization (= do not exist yet) and it is very confusing.

In the discussion of where this PR comes from that I mention in my first post of this PR #24820 puts you in context of what I am trying to solve. One of the keys is here: renovatebot/azure-devops-marketplace#42 (comment)

How many API calls and how many MB should it require?

The API (https://dev.azure.com/{org}/_apis/distributedtask/tasks/) resolves which are the latest versions of each task so that this call can be cached and used equally for all subsequent requests. In the tests that I add you can see an example of response.

The MB depends on each organization, because in my organization I can have 5 third party extensions and another organization 50. This API lists the latest versions of the official/built-in tasks (same for all organizations) + those of third parties/marketplace (these are the ones that can not be measured, it depends on each organization).

Example API result in my organization (built-in + third party installed): 1.93 MB

[bdovaz]

@rarkins @zharinov I think it's all done, I just need you to answer some doubts you had to see if any more changes are needed or not.

Besides, I don't understand this workflow, it fails by adding files to the tests? https://github.com/renovatebot/renovate/actions/runs/12457890306/job/34782564100

[rarkins]

We prefer inline test fixtures instead of dedicated file fixtures, so the test checks for that. It's possible to override and still merge

[bdovaz]

We prefer inline test fixtures instead of dedicated file fixtures, so the test checks for that. It's possible to override and still merge

I suppose that in this case, as it stands, it is correct and the inline is only for when there is little content.

[rarkins]

Needs RENOVATE_TOKEN fix

[bdovaz]

Needs RENOVATE_TOKEN fix

I just replied to you in the comment.

[bdovaz]

@rarkins I see that it has been removed from the “merge queue” because of the “Undesirable Code” check but as we already commented, although ideally it is better that the input of the tests is inline, in this case it is a large json file and so I guess it will be justified but there is no way to pass that test, I guess you will have to ignore it?

[renovate-release]

🎉 This PR is included in version 39.96.0 🎉

The release is available on:

• GitHub release
• npm package (@latest dist-tag)
• 39.96.0

Your semantic-release bot 📦🚀