[DRAFT] web: Extension source upload

[kmeisthax]

This PR attempts to add support for source upload on addons.mozilla.org, which is a necessary prerequisite to automating our extension submissions again.

The reason why this wasn't done before is that:

• We use Mozilla's sign-addon tool to upload the extension, and that uses API v4
• Source upload was only added in API v5 and sign-addon does not implement that API endpoint
• Nobody at Mozilla is currently interested in updating sign-addon for API v5 (in lieu of a major refactor)

We hack around that by manually issuing the v5 requests using the same Client class that sign-addon provides. I'm still not 100% sure that this is correct, however; because the v5 API endpoint requires a version ID and I'm not sure the v4 (or v5) signing endpoint returns a compatible ID. In fact, the logs from the most recent extension upload seem to imply that we DON'T get a version ID as part of the initial request at all. Hence why this is a draft - I need to do more analysis.

[diox]

Hi,

I work on AMO so I can provide some context.

sign-addon and the old v3/v4 submission endpoint suffer from various limitations & design issues, so we have decided to not port new features to them.

Instead we built a new, completely separate and fully featured submission API in v5, made it available in web-ext behind a --use-submission-api parameter. It will eventually replace the old one.

So you can't just replace the version number of the old endpoint, that will not get you anything.

web-ext will eventually do everything - follow mozilla/web-ext#2503 and probably mozilla/web-ext#2543 for that. In the meantime, you likely need to talk to the new API directly. I'd start with https://addons-server.readthedocs.io/en/latest/topics/api/addons.html#version-create.

A possible option if you don't want to fully port submission to the new API yet is to keep using sign-addon (or web-ext) with the legacy v4 API while using the new API just to submit source code immediately after submission.

Edit: oh, and as of last week, the new submission API returns a version field with information about the version that was uploaded, which can be useful to then poll the detail endpoint to download the signed XPI once it gets approved.

Keep in mind though, approval is never guaranteed to be immediate, add-ons can be subject to human review at any time, so in some cases it can take a few days to be approved.

[kmeisthax]

@diox The "keep using sign-addon but hack around it with v5 API endpoints" thing you mentioned is what this PR does.

You are correct that the best approach would be to wait for web-ext to have the features we need. But that's blocked on mozilla/web-ext#2497. For now this will at least ensure I'm not constantly having to mad-dash copy-paste everything to addons.mozilla.org before a reviewer gets to it.

I do have one question, though: because we're mixing v4 and v5 API calls, I have to grab the version ID from the v5 API (with filter=all_with_unlisted). Does the v5 /addons/addon/<id>/versions endpoint return newly-submitted versions immediately or is there a time lag I should be factoring into this code?

[diox]

For authenticated requests the /addons/addon/<id>/versions should return new versions immediately. Couple things to note however:

• The new API does return a version object that includes the id
• A few hours from now, the versions API detail will also work with version numbers. So if you know the version number, you will be able to use /api/v5/addons/addon/<addon_id>/versions/<version_number>/, without needing to query the list

[kmeisthax]

Oh wow! That's exactly what I needed. I'll hold off on merging until version number is ready to go though - it should let me get rid of some of the extra requests I'm currently doing in this PR.

[torokati44]

FYI mozilla/web-ext#2503 is now done. But I think the current setup is fine as is. It works at least.