Build an on premeses solution for secure environments

[denelon]

Description of the new feature / enhancement

Some environments do not have Internet connectivity. They need a way to host a private source that doesn't rely on Azure access.

microsoft/winget-cli#2361 (comment)

Proposed technical implementation details

No response

[FabianNiesen]

As @zachcarp wished, a bit more details for my use case from my issue #195
Our problem is that we have customers with very high-security requirements. This means no access to the internet and all data must pass through so-called data locks. The first level is the most complex. This means from one level via physical media and offline scans to the next level. These environments have staggered levels.
A practical option would be to create a corresponding repo of the packages in the secure environment and then distribute it to a corresponding web server at various levels. For this, the created repo would have to be static.
Most of the software that would have to be distributed with it are applications and scripts that are created within the environment and must not be exported. So creating the repo with Azure is not an option.

Proposed technical implementation details

My suggestion would be a PowerShell script that reads in all manifests in a subfolder and creates a corresponding feed. This folder can then be transferred to different environments. With dynamic destinations within the link, the hostname of the web server would not be relevant.
The alternatives would be to create your own software distribution that comes with clear versions or possibly Chocolaty. But I wanted to avoid the latter because WinGet is more future-proof.

[zachcarp]

@FabianNiesen - From my experience with closed networks, they still have some form of server/client model all of which are on the closed net.

If you, hypothetically, had a stand alone REST server that could run on generic hardware (such as a light-weight python based server or a docker container) would it be feasible for you to move the REST Server (source and/or binaries), installers, JSON manifests, and some PowerShell utilities through your locks - and after which you could instantiate the REST server, move your binaries to some sort of shared storage, update your JSON manifests with the new location, and upload them to your REST server?

[FabianNiesen]

We have a complete Active Directory on each stage, so there is some client server infrastructure 😉
But we have no SCCM. The question would be, are still talking about WinGet or is the Rest approach more complete custom thing.
If we are still talking about WinGet, I would like to get a hint to the right document and I would take a look.

[denelon]

It would still be WinGet calling into a fully supported "source".

Today, we have a "PreIndexed" package, and a "REST" source. Both of them technically can be configured to point to essentially any "URL". We're debating whether we build a "lighter-weight" source that could run on the local host. This could either be fully local like IIS or it could be in a hosted environment like a container or a VM.

I think the potentially tricky bits are how we reason about how the installers are referenced. This could be in the same logical HTTP server as the REST source, or potentially via UNC path.

Then we look at does this "source" itself move through the environments where it's "location" for winget source add is predetermined in the secured environment(s), or is this more about moving the manifests and the "installers" to different locations in the secured / hardened environment(s).

[denelon]

We had discussed a "file-based" source on a couple of different occasions, but the search performance becomes very poor as the number of packages increases.

There are also the questions around how frequently does one "update" their source, and what do they want to use for the "upstream" feed.

We've had some "event source" discussions in the past where the upstream feed could automatically "track" how many mutations had been processed so external parties could more easily identify what changes had been made since some previous state. That could help reduce the need to run a "full" scan across all packages in the upstream and somewhat limit it to what's changed since the upstream had last been checked.

There are many different considerations.

[jantari]

If someone here knows a little bit of go, I'd be very happy to take contributions to https://github.com/jantari/rewinged

It's exactly this: an offline or internal winget REST source server that can be run as a standalone portable executable on either Windows or Linux or in docker as well. It already works but there's for sure some improvements to be made.

I guess at some point we'll see more complex and featureful self-hostable winget REST sources, but for now rewinged is at least in a state where it's still simple but works. Sorry if this comes off like an ad but I put the project under the very permissive MIT license and like I said, since everyone here is interested in a self-hostable or "offline" winget source I would genuinely appreciate testers, ideas and code contributions :)

It's all based on the most important subset of the winget REST source API specification.

[Swamination]

There are several threads on this of varying needs. On one hand (1), there is a need for full RESTful + db environment. On another hand (2) is simply the ability to manage manifests and installation packages via internal UNC or URL (non-http/s).

A shorter resolution would be to provide a wingetcreate + winget install -m combination that fully supports UNC and URL paths. This would solve scenario 2 above, only being devoid of search functions in an environment that likely has other visibility into the source manifest and installer repos, being managed independently.

Use case example:

• Wingetcreate, generate local manifests in standard structure on \privatefqdn\packages
• Winget install -m, install from local manifests referenced on \privatefqdn\manifests
• Can never use winget search/show/list functions unless targeting local machine or exposed URL sources
• Can later convert to RESTful source since manifests in standardized format
• Can deploy combination of RESTful and manual UNC internal custom applications

Winget "source" option says it supports UNC paths, but it does not appear to actually support this as a simple share. Wingetcreate does not support UNC presently. Http/s generally assumes RESTful in these dev discussions, where http/s does provide some functionality as direct access for manifest management and direct --manifest installs.

I think a stopgap would be to assure consistency in Wingetcreate's ability to access UNC package paths and Winget's install ability to deploy those UNC paths. Then it comes down to management desire: want full search/management? Install RESTful environment. Want dumbfire? Deploy managed shares and script installs directly (which generally works if everything is URL).

[denelon]

I think that ask should go to wingetcreate along with the:

• Support publishing to REST API source winget-create#3

[byjrack]

I know I want to avoid running an app if possible. Likely could do something in Azure Functions and keep it light, but still dbs etc to handle. Ideally it would be easier for me to use static content similar to https://formulae.brew.sh/docs/api/ and existing resources we have for that type of hosting.

So in the rif on restsource I would say an ARM/Bicep for Functions backed by Cosmos might be pretty low overhead. While much less efficient, but when your pool of manifests is pretty small a bunch of static structured files likely will fit the bill.

[jantari]

@byjrack Azure Function + static files (e.g. in Azure blob storage?) should work. Unless you need to scale to 10.000s of packages I doubt a DB would be necessary.

But, an alternative to consider would be to run an app in Azure Container Apps. That'd work today and also be very light and low overhead.

[byjrack]

@byjrack Azure Function + static files (e.g. in Azure blob storage?) should work. Unless you need to scale to 10.000s of packages I doubt a DB would be necessary.

But, an alternative to consider would be to run an app in Azure Container Apps. That'd work today and also be very light and low overhead.

Yup but right now the restsource is an active app backed by cosmos and the pre-compiled is a bit of black magic (and not "official") on the sqlite db structure. It's not complicated but would take a bit of work to take a repo of YAMLs and spit out a working MSIX that might change w/o notice.

So 100% lots of options for static files. -m could support URLs as last i checked it still didn't. And similar to homebrew if there was a simple metadata file of all the manifests you could so a search, latest, etc. with it as a "source". i get i am handwaving some of the complexity, but static content is much easier to deal with.