Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GitHub App to notify OpenTofu registry when new releases are published. #1279

Open
skyzyx opened this issue Nov 11, 2024 · 7 comments
Open
Labels
enhancement New feature or request

Comments

@skyzyx
Copy link
Contributor

skyzyx commented Nov 11, 2024

Description of the feature you are looking for.

One of the things I appreciate about the Terraform registry that I’m unaware of in the OpenTofu registry is the GitHub app that receives a ping whenever I publish a new release, so that the Terraform registry updates nearly instantly.

Enabling such functionality could streamline the registry's ability to know when to update the versions and index for a provider/module. Push vs pull, as it were.

@skyzyx skyzyx added the enhancement New feature or request label Nov 11, 2024
@abstractionfactory
Copy link
Contributor

Hey @skyzyx is this, by any chance, similar to opentofu/opentofu#1236 ? At any rate, I believe it should be possible to create an RSS feed per provider that contains the versions on api.opentofu.org once the search interface is fully operational.

@skyzyx
Copy link
Contributor Author

skyzyx commented Nov 11, 2024

It is not.

The Terraform registry supports receiving webhook pings. This flips the approach from pull (registry periodically polls the endpoints to see if there are new releases) to push (the provider release notifies the registry).

CleanShot 2024-11-11 at 11 26 23@2x

CleanShot 2024-11-11 at 11 27 28@2x

I misspoke when I suggested it was a GitHub app. It's even simpler than that.

@skyzyx
Copy link
Contributor Author

skyzyx commented Nov 11, 2024

Not that we can necessarily remove support for polling/pulling data for provider releases (e.g., I don't expect HashiCorp to ping the OpenTofu registry on releases). However, for the providers who opt-in to pushing notifications via webhooks, the registry may be able to rely on that instead, which would opt-out that provider from polling/pulling.

  1. It would be a more efficient way to scale-up the system to support more providers.
  2. It would significantly reduce the delay in identifying new provider releases.

@abstractionfactory
Copy link
Contributor

So if I'm understanding this correctly, you would like to ping the OpenTofu registry if your provider has a new version? We purposefully avoided this by design since it requires us to run a web service so the provider author can log in and the provider author to trust us to some degree.

This can be reevaluated, but I wonder if we can address the root cause. What are you experiencing in terms of delay that would require such a solution?

@skyzyx
Copy link
Contributor Author

skyzyx commented Nov 11, 2024

I was unaware that the system was designed this way intentionally.

Last week, I was noticing that the official AWS provider listing was a solid 1-2 months behind.

And presently, I'm experiencing an issues described in opentofu/registry-ui#241.

I wonder if a Lambda function (or equivalent) could be written as something which can receive the webhook payload, then trigger the rebuilding of the index? Or just the index for that provider?

@abstractionfactory
Copy link
Contributor

The OpenTofu search is lagging behind because we are currently still fixing indexing issues. Once it's up and running, it should update within the hour.

@cam72cam
Copy link
Member

The current latency between the github released and the registry deployment is worst case 15 minutes. It averages around 5 to 10.

If that is sufficient, I believe we can close this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants