Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

satisfy markdown linter, remove aspecto references #5

Merged
merged 3 commits into from
Jul 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,9 +71,9 @@ members of the project's leadership.
## Attribution

This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
available at <https://www.contributor-covenant.org/version/1/4/code-of-conduct.html>

[homepage]: https://www.contributor-covenant.org

For answers to common questions about this code of conduct, see
https://www.contributor-covenant.org/faq
<https://www.contributor-covenant.org/faq>
19 changes: 17 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Path Warden

This project looks to improve the security in cloud systems. Specifically, it demonstrates:

1) A system for tracing the Lineage of data moving through services
2) A strategy for storing the Lineage information for a piece of data
3) An enforcement point for pieces of data based on their Lineage
Expand All @@ -8,6 +10,7 @@ This project looks to improve the security in cloud systems. Specifically, it de
This design looks at solving these issues in dynamically constructed edge systems.

## Overview

So far, there are 4 key components to this system:

1. Lineage Propagation: This refers to the propagation of labels related to a piece of data moving through a set of services. The entirety of this set of labels is the `Data Lineage`. The original source of the data is the `Data Provenance`. The goal of lineage propagation is to save the original Data Provenance and concatenate onto it each processing step the data undergoes to generate a Data Lineage that can be evaluated at each proceeding step and ultimately stored for future reference.
Expand All @@ -18,14 +21,17 @@ So far, there are 4 key components to this system:

4. Policy Evaluation: At the enforcement points, not-yet-evaluated labels are sent off to a separate service to be evaluated for their pass/fail status. Otherwise, previously cached values are simply retrieved.

#### Key Technologies & Concepts
### Key Technologies & Concepts

- Label Propagation built using OpenTelemetry
- Enforcement done in Istio Sidecars using Go Wasm Plugin
- Policies written in Rego & evaluated using OPA
- It is necessary that developers instrument their applications with our lineage propagation (tracing) libraries. However, these libraries desire to be incredibly lightweight and easy to use.

### Current State

#### Summary of System

1. Label Propagation is achieved using Open Telemetry's Baggage Concept. We store a Label Set in JSON format at the baggage labeled `lineage_label_set`.
2. Label Storage is achieved in myqsql by creating a separate table whose primary key is equivalent to the primary key of the table one is labeling. The current example shows this being done using a small library of functions in python. This allows enabling/disabling the labeling of data in existing systems without updating/destroying existing tables.
3. We enforce data label policies in Istio's Service Mesh sidecars using a Go-Wasm Plugin. Reference Istio & the Go Wasm SDK for more information on those.
Expand All @@ -38,40 +44,49 @@ So far, there are 4 key components to this system:
- wasm-lineage-headers contains all files relevant to the development of the plugin written for the Istio sidecar which parses, validates & caches LabelSets.

##### (Provided as Reference)

- OTel Basic contains a Series of Services used to Initially Develop & Test OpenTelemetry. Generally, the functionality created here is less mature than that in account-CRUD. These files are provided for general reference.

- OPA contains files relevant to testing & developing the OPA implementations. Ultimately, the plug-and-play solution of OPA for Istio was not used however these files are provided as reference.

#### Implementation Summary

As mentioned before, account-CRUD contains the demo of this system. See the README in that directory.

### Pre-reqs

- minikube installed on system
- Istio installed on Minikube cluster
- gsutils installed
- Go installed
- tinygo installed

#### Recommended Additional Software

- VSCode Server on instance for remote IDE access

#### Getting Up and Running with Cluster on EC2 instance

1. Create Tunnel from terminal: `ssh -L 8080:localhost:8080 <remote-host>`
2. Launch VSCode server: `code-server --auth none`
3. Open new Terminal on local machine & ssh: `ssh <remote-host>`
4. Create Tunnel from EC2 instance to minikube gateway `minikube tunnel`
5. Launch New Terminal & ssh which serves as your working cli

#### Accounts Required For:
#### Accounts Required For

- GoogleCloud: Remotely storing & Deploying the Wasm Plugin using a GC bucket. Archived files show how it can be deployed with a local file (see envoyFilter.yaml). Deploying EnvoyFilter with GoogleCloud means saving it to a ConfigMap and injecting ConfigMap into app deployment. See docs for more information.
- Docker: if you want to push images. Not necessary.
- Use Jaeger for OpenTelemetry trace visualization

##### Early Steps

In addition to connecting & establishing the minikube tunnel, early on you will need to do the following things:

1. Should you ever need to edit/redeploy the wasm-lineage-headers plugins (which is very likely), you'll need to make a google cloud account, and a bucket to contain the wasm file. Reference the git linked at the top of `WASM-Label-Lineage.md`.

### Future Work

- More Languages supported for Label Propagation
- More Databases supported for Label Storage
- Integrate OPAL for simpler Policy distribution
Expand Down
22 changes: 15 additions & 7 deletions apiserver/README.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,30 @@
# Flask API

## 1. Description

Base application for API Server

## 2. Deployment

In order to ensure the good functioning of this component service the following steps should be undertaken in the specified order:
- Deploy MySQL database (see _db/README.md_)
- kubectl apply -f _../OTel_Basic/otel-agent-config.yml_
- docker build . -t _<devacct>/flask-api-server_
- kubectl apply -f _./flaskapi-deployment.yml_
- kubectl apply -f _./laskapi-configmap.yml_
- kubectl apply -f _./flaskapi-service.yml_

- Deploy MySQL database (see _db/README.md_)
- `kubectl apply -f ../OTel_Basic/otel-agent-config.yml`
- `docker build . -t <devacct>/flask-api-server`
- `kubectl apply -f ./flaskapi-deployment.yml`
- `kubectl apply -f ./laskapi-configmap.yml`
- `kubectl apply -f ./flaskapi-service.yml`

## 3. Automated Build

Please check the flaskapi-deployment.yml, line 21, and fill in with your account username by which you're going to build the docker image for the flask api server.
The default _devacct_ value is _$USER_.
The default _devacct_ value is _$USER_.
The image can be automatically build by running:

- ./build.sh _devacct_

## 4. Automated Start

This service can be brought up automatically by running; the script doesn't include OpenTelemetry Agent Configmap:

- ./start.sh
36 changes: 25 additions & 11 deletions docs/DeployEnvoyFilterIstio.md
Original file line number Diff line number Diff line change
@@ -1,55 +1,68 @@
# Deploying an EnvoyFilter resource into istio

The following instructions are for deploying an EnvoyFilter resource into istio. This is generally useful as Istio's custom WasmPlugin resources is missing a number of features/configuration options for Wasm plugins.
Notably:

- The default WasmPlugin resource doesn't support TCP filters
- The default WasmPlugin resource doesn't allow for configuring vm_id

We use an EnvoyFilter resource to deploy the WASM plugin in the account-CRUD demo.

Furthermore, this tutorial focuses on deploying using a local wasm file and a ConfigMap. While not recommended due to the fact this approach requires modifying the manifest of the pod you are deploying into, we provide it here as reference.
Furthermore, this tutorial focuses on deploying using a local wasm file and a ConfigMap. While not recommended due to the fact this approach requires modifying the manifest of the pod you are deploying into, we provide it here as reference

---

Learnings based off:
https://sirishagopigiri-692.medium.com/deploying-envoy-filter-on-istio-ce2d2573b981
<https://sirishagopigiri-692.medium.com/deploying-envoy-filter-on-istio-ce2d2573b981>

See Tutorial Above for Full explanation. Below are some useful/notable steps and commands

## Pre-Reqs

- Istio Installed
- Namespaces Labelled: `kubectl label namespace default istio-injection=enabled`
- Create & Write Wasm Plugin (See WASMPluginFromScratch.md)

## Deploying

### Create Config Map in Default Namespace: only done once

`kubectl create configmap NAME-filter --from-file=NAME-filter.wasm=main.wasm`

#### Modify Application yml Deployment: only done once unless file is edited
* In the Deployment section of the App, under `spec.template.metadata` add these annotations:

- In the Deployment section of the App, under `spec.template.metadata` add these annotations:

- `sidecar.istio.io/userVolume: '[{"name":"NAME-filter","configMap":{"name":"NAME-filter"}}]'`
- `sidecar.istio.io/userVolumeMount: '[{"mountPath":"/var/local/wasm","name":"NAME-filter"}]'`
- `sidecar.istio.io/logLevel: "info"`

#### Redeploy Service

`kubectl apply -f <filename>.yml`

##### Can view annotations w/

`kubectl describe pod POD-NAME|grep -i NAME-filter`

**Testing:**
You can test Service Response now w/ wtv curl command you have. Just to ensure you have connectivity before we deploy the Plugin.

**Get Logs for Plugin**
`kubectl logs -f -l app=NAME-c istio-proxy`

**Deploy EnvoyFilter**
**Deploy EnvoyFilter**
`kubectl apply -f filter.yml`

**Verify deployment**
`kubectl get envoyfilter`

---

Here I have provided a manifest file for an HTTP as well as a TCP EnvoyFilter. Both of these were tested and worked when deploying using the strategy outlined in the article.

# HTTP Filter
## HTTP Filter

```yaml
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
Expand Down Expand Up @@ -86,7 +99,8 @@ spec:
runtime: envoy.wasm.runtime.v8
```

# TCP Filter
## TCP Filter

```yaml
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
Expand Down Expand Up @@ -118,4 +132,4 @@ spec:
local:
filename: /var/local/wasm/<FILE-NAME-IN-CONFIG-MAP>.wasm
allow_precompiled: true
```
```
9 changes: 5 additions & 4 deletions docs/Go&TinyGo.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
**Installing Go**
https://go.dev/doc/install
**Installing Go**
<https://go.dev/doc/install>

- Use `wget` to download tar file
- Use sudo into tar

**Installing TinyGo**
https://tinygo.org/getting-started/install/linux/
- Remember to add Path to .bashrc!
<https://tinygo.org/getting-started/install/linux/>

- Remember to add Path to .bashrc!
17 changes: 11 additions & 6 deletions docs/InstallingIstio.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,34 @@
# Installing Istio

## Docs:
https://istio.io/latest/docs/setup/getting-started/#download
## Docs

<https://istio.io/latest/docs/setup/getting-started/#download>

### Summary of the Above

`curl -L https://istio.io/downloadIstio | sh -`
`cd istio-1.16.1`
`export PATH=$PWD/bin:$PATH`

#### Adding permanently to path

`vim ~/.bashrc`
Paste at bottom:
`export PATH=/home/<user-name>/istio-1.16.1/bin:$PATH`

### Continuing Istio Install

`istioctl install --set profile=demo -y`

- Do Not Run this. Provided for reference: `istioctl uninstall -y --purge`


*Apply this at your discretion but it's more than likely it's what you want*
kubectl label namespace default istio-injection=enabled

- Do Not Run this. Provided for reference: `kubectl label namespace default istio-injection=disabled`

**If you already have pods installed/running, you'll likely need to restart the pods so that the Istio proxy can get injected properly**
`kubectl delete --all pods --namespace=default`

**Verify there are no Istio issues**
`istioctl analyze`
`istioctl analyze`
4 changes: 2 additions & 2 deletions docs/Istio&WASMBasics.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
## Tutorial

Simply Follow this for a basic Understanding of WASM Plugins in Istio:
https://tetrate.io/blog/istio-wasm-extensions-and-ecosystem/
<https://tetrate.io/blog/istio-wasm-extensions-and-ecosystem/>
Loading