Add steps to configure a PGP key for agent upgrade

[kilfoyle]

This adds instructions for air-gapped environments on how to set server.pgp.upstream_url so that Elastic Agents can access the PGP/GPG key required for upgrade.

See docs preview page.

Please let me know if I've missed anything, especially in the "Note that" section under the steps.

Closes: #980
Rel: https://github.com/elastic/sdh-beats/issues/4496

[github-actions]

A documentation preview will be available soon.

• 🔨 Buildkite builds
• 📚 HTML diff
• 📙 Preview page

Request a new doc build by commenting

• Rebuild this PR: run docs-build
• Rebuild this PR and all Elastic docs: run docs-build rebuild

_{run docs-build is much faster than run docs-build rebuild. A rebuild should only be needed in rare situations.}

_{If your PR continues to fail for an unknown reason, the doc build pipeline may be broken. Elastic employees can check the pipeline status here.}

[kilfoyle]

@lucabelluccini When you have a chance, please let me know if this new section looks okay. Thanks!

[kilfoyle]

@lucabelluccini trying again. Please have a look if you have a chance. Thanks!

[kilfoyle]

@michel-laterman , @michalpristas , @lucabelluccini
Can any of you please review and confirm that these added Configure Elastic Agents to download a PGP/GPG key from Fleet Server steps are correct?

[kilfoyle]

@elasticmachine run docs-build

[lucabelluccini]

I still think a developer should review this.

[kilfoyle]

Thanks @lucabelluccini

I still think a developer should review this.

@michel-laterman or @michalpristas Would either of you mind taking a look at this? Please let me know if you're not the right people to ask.

[michel-laterman]

I think it looks good, do we want to add anything to explain how to use the fleet-server endpoint as an alternative when distributing gpg keys (issue elastic/fleet-server#2887)?

[kilfoyle]

do we want to add anything to explain how to use the fleet-server endpoint as an alternative when distributing gpg keys (issue elastic/fleet-server#2887)?

@michel-laterman I've add the steps below, but I'm way out of my depth here so please let me know whatever needs fixing up.

---

[michel-laterman]

I think my reading comprehension on my 1st pass wasn't great.

Just to be clear fleet-server serves PGP keys at GET /api/agents/upgrades/{major}.{minor}.{patch}/pgp-public-key
It will serve the key in FLEETSERVER_BINARY_DIR/elastic-agent-upgrade-keys/default.pgp by default, if there is no default.pgp key in the dir it will look upstream to retrieve a key; server.pgp.upstream_url can be used to configure to a custom URL for fleet-server to use.

IIRC agents will try to get the upstream key, then the fleet-server endpoint (@michalpristas please confirm), they do not use the url we add to fleet-server config

[kilfoyle]

Thanks @michel-laterman! That's really clear. Here's my attempt to fix things up:

---

[lucabelluccini]

Suggested change

	Accessing the endpoint requires a valid API key and the endpoint is rate limited.
	Accessing the endpoint requires a valid API key and the endpoint is rate limited.

I am not sure we're protecting the key via API key - can we confirm?
If yes, are Elastic Agent adding the API Key to the requests?

[lucabelluccini]

Suggested change

	If there isn't a `default.pgp` key in the `FLEETSERVER_BINARY_DIR/elastic-agent-upgrade-keys/default.pgp` directory, {fleet-server} instead will attempt to retrieve a PGP/GPG key from the URL that you can specify with the `server.pgp.upstream_url` setting.
	If there isn't a `default.pgp` key in the `FLEETSERVER_BINARY_DIR/elastic-agent-upgrade-keys/default.pgp` directory, {fleet-server} instead will attempt to retrieve a PGP/GPG key from the URL that you can specify with the `server.pgp.upstream_url` setting.

Few points to clarify:

• Is the default.pgp bundled with Fleet Server?

If it's bundled, air-gapped users are good to go with it.

• Is the default.pgp downloaded lazily from https://artifacts.elastic.co/GPG-KEY-elastic-agent by default?

If it is the case, then air-gapped users can grant access to it (HTTP Proxy or open the firewall).

• Is the default.php downloaded lazily from the URL provided via server.pgp.upstream_url setting?

If it is the case, then air-gapped users can specify a URL so that it will be downloaded from there.

[michel-laterman]

It's a lazy download from server.pgp.upstream_url

[michel-laterman]

To be very clear:

• Is the default.pgp bundled with Fleet Server?

No, on install this file does not exist; but a user can add it manually write the file on the instance if there is 0 internet access in order to distribute it to the agents

• Is the default.pgp downloaded lazily from https://artifacts.elastic.co/GPG-KEY-elastic-agent by default?

Yes, the default value of server.pgp.upstream_url is https://artifacts.elastic.co/GPG-KEY-elastic-agent; if default.pgp is not on-disk when the key is requested, fleet-server will download the file from the upstream URL, save it to disk and serve it

• Is the default.php downloaded lazily from the URL provided via server.pgp.upstream_url setting?

yes, see above 😄

[kilfoyle]

@michel-laterman I'm sorry to bring this one up again (I lost track of it and am doing some housekeeping now).

Is there anything I need to clarify in the text above, or do you think this is okay to merge as is?

[michel-laterman]

I don't think so

[lucabelluccini]

Hello @kilfoyle - I've left some comments. We might need clarifications by the developers.
I've not taken the time to setup an environment to try it out yet.

[kilfoyle]

@lucabelluccini I need to merge any remaining open docs PRs by end of day on Wednesday. Are there any remaining changes that you'd like for this PR? I'd suggest that we merge it as is, and if we need to refine it we can do that later. What do you think?

[lucabelluccini]

Is there a setting where one can provide the CA to trust the TLS Certificate of the Fleet server?

[kilfoyle]

@michel-laterman Can you help with Luca's question?

[michel-laterman]

I believe that it would be the standard CA options

[lucabelluccini]

This is Fleet Server integration connecting to server.pgp.upstream_url.
Would it use the --certificate-authorities (the global one for EA)?

[kilfoyle]

@michel-laterman Tagging you just in case you haven't seen Luca's question. Thanks!

[michel-laterman]

Yes, it should

[kilfoyle]

Thanks! I've changed the text to:

For an https endpoint, the CA for {fleet-server} to connect to server.pgp.upstream_url must be trusted by {fleet-server} using the --certificate-authorities setting that is used globally for {agent}.

[lucabelluccini]

It must be the full url or just the host?
E.g. http://my-web:8080/default.pgp or http://my-web:8080/?

[kilfoyle]

@michel-laterman Can you help with Luca's question?

[michel-laterman]

The full URL

[kilfoyle]

Updated to:

In the **Custom fleet-server configurations** field, add the setting `server.pgp.upstream_url` 
with the full URL where the PGP/GPG key can be accessed. For example:
[source,yaml]
----
server.pgp.upstream_url: <http://my-web-server:8080/default.pgp>

[karenzone]

LGTM

[kilfoyle]

I'm going to merge this since the docs freeze is tonight and any unmerged work will be lost.