Add nebula_ca plugin #539

cclaudio · 2024-10-19T06:31:22Z

This PR adds the nebula_ca plugin.

How to build and start the plugin

Build trustee with the 'nebula-ca-plugin' cargo feature enabled and build a KBS image that has the nebula-cert binary

docker compose build --build-arg NEBULA_CA_PLUGIN=true --no-cache

Add the lines below to kbs/config/docker-compose/kbs-config.toml

    [[plugins]]
    name = "nebula-ca"

Start trustee

docker compose up

How to test the credential service

Ultimately, the plugin will be called from the CDH (confidential-containers/guest-components#763). In the meantime, we can build and use the kbs-client I have in the branch https://github.com/cclaudio/trustee/tree/nebula-ca-plugin-test

$ git clone https://github.com/cclaudio/trustee.git -b nebula-ca-plugin-test
$ cd trustee/kbs && make cli ATTESTER=snp-attester
$ sudo make install-cli

Request a Nebula credential

$ kbs-client config --auth-private-key kbs/config/private.key set-resource-policy --policy-file  kbs/sample_policies/allow_all.rego
$ kbs-client get-resource --plugin-name "nebula-ca" --resource-path "credential?name=podA&ip=10.9.8.2/21" | base64 -d

That should return a CredentialServiceOut structure like:

{
  "node_crt":[45,45,45,45,45,66,69,71,73,78,32,78,69,66,85,76,65,32,67,69,82,84,73,70,73,67,65,84,69,45,45,45,45,45,10,67,110,85,75,66,72,66,118,90,69,69,83,67,89,75,81,112,70,67,65,56,80,47,47,68,120,111,83,103,73,67,73,67,73,68,119,47,47,56,80,103,73,67,73,67,73,68,119,47,47,56,80,75,76,80,47,43,55,115,71,77,76,88,80,10,120,115,99,71,79,105,68,48,99,57,104,75,115,99,73,119,69,84,116,118,74,54,48,99,53,122,119,69,104,119,101,49,48,74,55,111,73,55,111,101,117,68,102,97,77,89,117,87,67,85,111,103,76,70,48,75,99,85,79,112,81,73,114,106,10,110,74,51,101,72,116,68,120,103,84,67,68,69,54,116,106,76,82,50,87,111,90,98,54,103,53,50,89,71,56,85,83,81,70,118,115,85,87,69,89,83,108,83,110,88,97,53,102,48,52,88,78,82,67,109,88,84,85,66,122,51,73,90,67,10,86,121,120,86,88,68,101,50,85,70,84,76,72,105,83,82,121,82,112,69,48,82,122,97,51,100,107,113,117,70,118,106,52,70,83,70,88,72,70,80,89,122,108,107,79,104,43,107,106,90,108,121,83,103,65,61,10,45,45,45,45,45,69,78,68,32,78,69,66,85,76,65,32,67,69,82,84,73,70,73,67,65,84,69,45,45,45,45,45,10],
  "node_key":[45,45,45,45,45,66,69,71,73,78,32,78,69,66,85,76,65,32,88,50,53,53,49,57,32,80,82,73,86,65,84,69,32,75,69,89,45,45,45,45,45,10,104,74,67,43,105,85,86,56,113,109,122,85,105,57,75,87,118,73,110,83,110,114,53,65,104,108,80,43,108,52,118,70,105,69,117,101,84,77,76,47,73,76,111,61,10,45,45,45,45,45,69,78,68,32,78,69,66,85,76,65,32,88,50,53,53,49,57,32,80,82,73,86,65,84,69,32,75,69,89,45,45,45,45,45,10],
  "ca_crt":[45,45,45,45,45,66,69,71,73,78,32,78,69,66,85,76,65,32,67,69,82,84,73,70,73,67,65,84,69,45,45,45,45,45,10,67,107,115,75,71,85,53,108,89,110,86,115,89,83,66,68,81,83,66,109,98,51,73,103,86,72,74,49,99,51,82,108,90,83,66,76,81,108,77,111,116,117,106,66,117,65,89,119,116,115,47,71,120,119,89,54,73,79,48,86,104,52,122,84,10,83,81,121,52,108,117,54,105,122,68,98,72,102,114,72,122,65,69,117,66,120,112,108,53,88,70,65,102,115,114,47,108,85,88,98,105,81,65,69,83,81,77,77,99,108,110,105,101,117,84,100,98,97,55,56,48,80,102,82,76,101,109,77,117,10,104,90,73,89,69,50,89,97,86,112,100,105,105,72,50,75,119,81,87,43,54,113,112,89,115,105,113,119,66,98,106,53,85,72,56,100,116,114,78,101,49,43,90,122,75,86,69,76,74,65,109,80,115,79,83,105,89,84,100,115,71,81,103,61,10,45,45,45,45,45,69,78,68,32,78,69,66,85,76,65,32,67,69,82,84,73,70,73,67,65,84,69,45,45,45,45,45,10]
}

cclaudio · 2025-01-08T23:53:52Z

This PR is ready for review. I updated the PR description with the steps to build and test it.

portersrc

lgtm
I reviewed this by going through the code and looking for bugs -- I didn't spot anything.
I also tried to build and run. docker compose up fails for me but for reasons unrelated to your work, I think. If you eventually lift those instructions and put them in docs or a tutorial, we can add some more details.

fitzthum

Looks clean. A few suggestions but nothing too significant.

kbs/docs/config.md

kbs/src/plugins/implementations/nebula_ca.rs

kbs/Cargo.toml

cclaudio · 2025-01-29T16:47:22Z

All feedbacks applied and added some improvements

Created macros to convert structs to command line arguments for the nebula-cert tool
Got rid of the Vec<OsString> parameter for the functions implemented by NebulaCertBin
Verify the created certificate before returning it
Print the Nebula CA certificate to the KBS log
Documented the plugin in kbs/docs/plugins/nebula_ca.md . I would suggest we use this directory for plugins documentation. Currently, we have documentation for configuration, but plugins can provide a diversity of runtime services and may have build dependencies
The plugin configurations are all optional, only the name field is required now.
Updated the PR description

fitzthum

LGTM

There is small problem with link checker.

Let's get another approval from @confidential-containers/trustee-maintainers

This plugin calls the nebula-cert binary to provide some of its functionality for Nebula nodes (e.g. CoCo PODs or confidential VMs) that want to join an encrypted Nebula overlay network (ENON). The Nebula CA is created when the KBS is started. Currently, the only runtime service it provides is the credential service which returns the following structure for the caller to join an ENON. struct CredentialServiceOut { node_crt: Vec<u8>, // Self-signed certificate created node_key: Vec<u8>, // Key created ca_crt: Vec<u8>, // CA certificate } Since the CredentialServiceOut is returned only after the caller is attested, the ENON is expected to have only attested nodes. Some use cases for an ENON: - The workload does not trust the network interface provided by the CSP - The workload does not support data encryption over the network For further information on configuration or usage, please refer to the kbs/docs/plugins/nebula_ca.md or the official Nebula website https://github.com/slackhq/nebula Signed-off-by: Claudio Carvalho <[email protected]>

Xynnn007

Thanks for the dry code.

There is a question about the design. Do you think it better for users to manually generate the nebula certs out of band instead of generate them automatically inside code?

Users might reuse some certs for deployments rather than generate them during each deployment.

Xynnn007 · 2025-02-05T02:59:46Z

kbs/docs/config.md

@@ -283,6 +316,13 @@ policy_engine = "opa"
 name = "resource"
 type = "LocalFs"
 dir_path = "/opt/confidential-containers/kbs/repository"
+
+[[plugins]]


Shall we add a separate config example for nebula and leave this only for CoCoAS example?

Xynnn007 · 2025-02-05T03:00:09Z

kbs/docs/config.md

+nebula_cert_bin_path = "/usr/local/bin/nebula-cert"
+work_dir = "/opt/confidential-containers/kbs/nebula-ca"
+    [plugins.settings]
+    name = "Nebula CA for Trustee KBS"


Same suggestion as before.

Xynnn007 · 2025-02-05T03:06:34Z

kbs/src/plugins/implementations/nebula_ca.rs

+/// Nebula CA plugin.
+/// More information can be found in the KBS plugin documentation
+/// and in the [Nebula](https://github.com/slackhq/nebula) website.


We could use inner line doc here.

Suggested change

/// Nebula CA plugin.

/// More information can be found in the KBS plugin documentation

/// and in the [Nebula](https://github.com/slackhq/nebula) website.

//! Nebula CA plugin.

//! More information can be found in the KBS plugin documentation

//! and in the [Nebula](https://github.com/slackhq/nebula) website.

An Example
If we publish kbs to crates.io
Behavior: https://github.com/serde-rs/serde/blob/master/serde/src/ser/mod.rs
Source code: https://github.com/serde-rs/serde/blob/master/serde/src/ser/mod.rs

Xynnn007 · 2025-02-05T03:09:51Z

kbs/src/plugins/implementations/nebula_ca.rs

+            "1.2.3.5/21,1.2.3.6/21".into(),
+        ])
+    )]
+


Suggested change

Let 's delete the empty line

Xynnn007 · 2025-02-05T03:18:16Z

kbs/src/plugins/implementations/nebula_ca.rs

+                .await
+                .context(format!("{} sign {:?}", self.path.display(), &args))?;
+        if !status.success() {
+            return Err(anyhow!("{} sign {:?}", self.path.display(), &args));


Suggested change

return Err(anyhow!("{} sign {:?}", self.path.display(), &args));

bail!("{} sign {:?}", self.path.display(), &args);

Xynnn007 · 2025-02-05T03:19:04Z

kbs/src/plugins/implementations/nebula_ca.rs

+            .status()
+            .context(format!("{} verify {:?}", self.path.display(), &args))?;
+        if !status.success() {
+            return Err(anyhow!("{} verify {:?}", self.path.display(), &args));


Suggested change

return Err(anyhow!("{} verify {:?}", self.path.display(), &args));

bail!("{} verify {:?}", self.path.display(), &args);

Xynnn007 · 2025-02-05T03:21:09Z

kbs/src/plugins/implementations/nebula_ca.rs

+    ffi::OsString,
+    fs,
+    path::{Path, PathBuf},
+    process::Command,


How about using tokio:: process::Command here? The async version has better efficiency. https://docs.rs/tokio/1.43.0/tokio/process/index.html

portersrc · 2025-02-05T14:42:11Z

There is a question about the design. Do you think it better for users to manually generate the nebula certs out of band instead of generate them automatically inside code?

You're thinking there's a pool of certs, for example, that you draw from whenever a pod tries to join the network?

Xynnn007 · 2025-02-06T06:22:25Z

@portersrc I mean only the CA cert, not the node certs

portersrc · 2025-02-06T06:53:19Z

@portersrc I mean only the CA cert, not the node certs

@cclaudio has some logic related to this (nebula_ca.rs line 125).

        // Create self-signed certificate authority
        if !crt.exists() && !key.exists() {
            nebula.ca(
                &config.ca_config.unwrap_or_default(),
                crt.as_path(),
                key.as_path(),
            )?;
            log::info!("Self-signed Nebula CA created");
        }

        // Check the provided or created Nebula CA exists.
        if !crt.exists() || !key.exists() {
            bail!("Nebula CA not found");
        }

And after browsing more, I see he mentions it in the config.md:

When the KBS is starting, the plugin will try to load the CA credential from `${work_dir}/ca/ca.{key,crt}`, otherwise, it will create a new self-signed CA credential based on the `[plugins.self_signed_ca]` properties provided. The default value will be considered if a `[plugins.self_signed_ca]` property is not provided.

He'd have to comment on how the user is expected to do this, though. I assume they rebuild the container with those CA files included. It might be good to document how to generate and then include them.

cclaudio requested a review from a team as a code owner October 19, 2024 06:31

cclaudio mentioned this pull request Oct 19, 2024

Extend KBS to provide the resources required to create an encrypted overlay network #451

Closed

portersrc mentioned this pull request Oct 23, 2024

Add mesh VPN support to the CDH confidential-containers/guest-components#763

Open

cclaudio marked this pull request as draft November 12, 2024 21:44

cclaudio force-pushed the nebula-plugin branch from 4359304 to a48c6de Compare January 8, 2025 23:16

cclaudio marked this pull request as ready for review January 8, 2025 23:17

portersrc approved these changes Jan 9, 2025

View reviewed changes

fitzthum reviewed Jan 9, 2025

View reviewed changes

kbs/docs/config.md Outdated Show resolved Hide resolved

kbs/docs/config.md Outdated Show resolved Hide resolved

kbs/src/plugins/implementations/nebula_ca.rs Outdated Show resolved Hide resolved

kbs/Cargo.toml Outdated Show resolved Hide resolved

salmanyam mentioned this pull request Jan 18, 2025

Add splitapi resource plugin #585

Open

cclaudio force-pushed the nebula-plugin branch 2 times, most recently from cc40b9a to 604a793 Compare January 29, 2025 16:45

fitzthum approved these changes Jan 29, 2025

View reviewed changes

cclaudio force-pushed the nebula-plugin branch from 604a793 to 49ed388 Compare January 29, 2025 17:47

Xynnn007 reviewed Feb 5, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add nebula_ca plugin #539

Add nebula_ca plugin #539

cclaudio commented Oct 19, 2024 •

edited

Loading

cclaudio commented Jan 8, 2025

portersrc left a comment •

edited

Loading

fitzthum left a comment

cclaudio commented Jan 29, 2025 •

edited

Loading

fitzthum left a comment

Xynnn007 left a comment •

edited

Loading

Xynnn007 Feb 5, 2025

Xynnn007 Feb 5, 2025

Xynnn007 Feb 5, 2025

Xynnn007 Feb 5, 2025

Xynnn007 Feb 5, 2025

Xynnn007 Feb 5, 2025

Xynnn007 Feb 5, 2025

portersrc commented Feb 5, 2025 •

edited

Loading

Xynnn007 commented Feb 6, 2025

portersrc commented Feb 6, 2025 •

edited

Loading

	return Err(anyhow!("{} sign {:?}", self.path.display(), &args));
	bail!("{} sign {:?}", self.path.display(), &args);

	return Err(anyhow!("{} verify {:?}", self.path.display(), &args));
	bail!("{} verify {:?}", self.path.display(), &args);

Add nebula_ca plugin #539

Are you sure you want to change the base?

Add nebula_ca plugin #539

Conversation

cclaudio commented Oct 19, 2024 • edited Loading

How to build and start the plugin

How to test the credential service

cclaudio commented Jan 8, 2025

portersrc left a comment • edited Loading

Choose a reason for hiding this comment

fitzthum left a comment

Choose a reason for hiding this comment

cclaudio commented Jan 29, 2025 • edited Loading

fitzthum left a comment

Choose a reason for hiding this comment

Xynnn007 left a comment • edited Loading

Choose a reason for hiding this comment

Xynnn007 Feb 5, 2025

Choose a reason for hiding this comment

Xynnn007 Feb 5, 2025

Choose a reason for hiding this comment

Xynnn007 Feb 5, 2025

Choose a reason for hiding this comment

Xynnn007 Feb 5, 2025

Choose a reason for hiding this comment

Xynnn007 Feb 5, 2025

Choose a reason for hiding this comment

Xynnn007 Feb 5, 2025

Choose a reason for hiding this comment

Xynnn007 Feb 5, 2025

Choose a reason for hiding this comment

portersrc commented Feb 5, 2025 • edited Loading

Xynnn007 commented Feb 6, 2025

portersrc commented Feb 6, 2025 • edited Loading

cclaudio commented Oct 19, 2024 •

edited

Loading

portersrc left a comment •

edited

Loading

cclaudio commented Jan 29, 2025 •

edited

Loading

Xynnn007 left a comment •

edited

Loading

portersrc commented Feb 5, 2025 •

edited

Loading

portersrc commented Feb 6, 2025 •

edited

Loading