Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for empty directory #226

Merged
merged 4 commits into from
Jun 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions api/v1alpha2/minicluster_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -250,6 +250,15 @@ type ContainerVolume struct {
// +default=false
// +optional
ReadOnly bool `json:"readOnly,omitempty"`

// Add an empty directory custom type
// +optional
EmptyDirMedium string `json:"emptyDirMedium,omitempty"`

// +kubebuilder:default=false
// +default=false
// +optional
EmptyDir bool `json:"emptyDir,omitempty"`
}

type FluxSpec struct {
Expand Down
8 changes: 8 additions & 0 deletions api/v1alpha2/swagger.json
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,14 @@
"description": "Config map name if the existing volume is a config map You should also define items if you are using this",
"type": "string"
},
"emptyDir": {
"type": "boolean",
"default": false
},
"emptyDirMedium": {
"description": "Add an empty directory custom type",
"type": "string"
},
"hostPath": {
"description": "An existing hostPath to bind to path",
"type": "string"
Expand Down
14 changes: 14 additions & 0 deletions api/v1alpha2/zz_generated.openapi.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 12 additions & 0 deletions chart/templates/minicluster-crd.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,12 @@ spec:
Config map name if the existing volume is a config map
You should also define items if you are using this
type: string
emptyDir:
default: false
type: boolean
emptyDirMedium:
description: Add an empty directory custom type
type: string
hostPath:
description: An existing hostPath to bind to path
type: string
Expand Down Expand Up @@ -720,6 +726,12 @@ spec:
Config map name if the existing volume is a config map
You should also define items if you are using this
type: string
emptyDir:
default: false
type: boolean
emptyDirMedium:
description: Add an empty directory custom type
type: string
hostPath:
description: An existing hostPath to bind to path
type: string
Expand Down
12 changes: 12 additions & 0 deletions config/crd/bases/flux-framework.org_miniclusters.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,12 @@ spec:
Config map name if the existing volume is a config map
You should also define items if you are using this
type: string
emptyDir:
default: false
type: boolean
emptyDirMedium:
description: Add an empty directory custom type
type: string
hostPath:
description: An existing hostPath to bind to path
type: string
Expand Down Expand Up @@ -723,6 +729,12 @@ spec:
Config map name if the existing volume is a config map
You should also define items if you are using this
type: string
emptyDir:
default: false
type: boolean
emptyDirMedium:
description: Add an empty directory custom type
type: string
hostPath:
description: An existing hostPath to bind to path
type: string
Expand Down
19 changes: 15 additions & 4 deletions controllers/flux/containers.go
Original file line number Diff line number Diff line change
Expand Up @@ -97,10 +97,21 @@ func getContainers(

// Add on existing volumes/claims
for volumeName, volume := range container.Volumes {
mount := corev1.VolumeMount{
Name: volumeName,
MountPath: volume.Path,
ReadOnly: volume.ReadOnly,

var mount corev1.VolumeMount
if volume.EmptyDir {

// We assume bindings to /dev/shm
mount = corev1.VolumeMount{
Name: volumeName,
MountPath: "/dev/shm",
}
} else {
mount = corev1.VolumeMount{
Name: volumeName,
MountPath: volume.Path,
ReadOnly: volume.ReadOnly,
}
}
mounts = append(mounts, mount)
}
Expand Down
18 changes: 17 additions & 1 deletion controllers/flux/volumes.go
Original file line number Diff line number Diff line change
Expand Up @@ -132,11 +132,27 @@ func getExistingVolumes(existing map[string]api.ContainerVolume) []corev1.Volume
},
}

} else if volumeMeta.EmptyDir {

// The Flux Operator supports default and memory
medium := corev1.StorageMediumDefault
if volumeMeta.EmptyDirMedium == "memory" {
medium = corev1.StorageMediumMemory
}
newVolume = corev1.Volume{
Name: volumeName,
VolumeSource: corev1.VolumeSource{
EmptyDir: &corev1.EmptyDirVolumeSource{
Medium: medium,
},
},
}

} else if volumeMeta.HostPath != "" {
newVolume = corev1.Volume{
Name: volumeName,
VolumeSource: corev1.VolumeSource{
// Empath path for type means no checks are done
// Empty path for type means no checks are done
HostPath: &corev1.HostPathVolumeSource{
Path: volumeMeta.Path,
},
Expand Down
61 changes: 61 additions & 0 deletions docs/getting_started/custom-resource-definition.md
Original file line number Diff line number Diff line change
Expand Up @@ -1115,6 +1115,7 @@ We did this because volumes are complex and it was challenging to support every
the volumes and persistent volume claims that the MiniCluster needs, and simply tell it about them. A volume (that must exist) can be:

- a hostpath (good for local development)
- an empty directory (and of a particular custom type, such as Memory)
- a persistent volume claim (PVC) and persistent volume (PV) that you've created
- a secret that you've created
- a config map that you've created
Expand Down Expand Up @@ -1171,6 +1172,66 @@ spec:

An example is provided in the [volumes test](https://github.com/flux-framework/flux-operator/tree/main/examples/tests/volumes).

#### emptyDir example

A standard empty directory might look like this:

```yaml
apiVersion: flux-framework.org/v1alpha2
kind: MiniCluster
metadata:
name: flux-sample
spec:
size: 2
containers:
- image: rockylinux:9
command: df -h /dev/shm
volumes:
# must be lowercase!
my-empty-dir:
emptyDir: true
```

And one for shared memory (to inherit the host) like this:


```yaml
apiVersion: flux-framework.org/v1alpha2
kind: MiniCluster
metadata:
name: flux-sample
spec:
size: 2
containers:
- image: rockylinux:9
command: ls /data
volumes:
# must be lowercase!
my-empty-dir:
emptyDir: true
emptyDirMedium: "memory"
```

The default binds to the path `/dev/shm` and is not customizable. This can be changed if needed. When you have the "memory" medium added,
you should see all the shared memory from the host, which is [calculated here](https://github.com/kubernetes/kubernetes/blob/e6616033cb844516b1e91b3ec7cd30f8c5d1ea50/pkg/volume/emptydir/empty_dir.go#L148-L157).
As an example, here is output from a local run with kind when shared memory is added:

```console
$ kubectl logs flux-sample-0-smflk
Defaulted container "flux-sample" out of: flux-sample, flux-view (init)
Filesystem Size Used Avail Use% Mounted on
tmpfs 32G 0 32G 0% /dev/shm
```

And here is the same MiniCluster with the volume removed (64M is the default):

```console
$ kubectl logs flux-sample-0-4bwjf -f
Defaulted container "flux-sample" out of: flux-sample, flux-view (init)
Filesystem Size Used Avail Use% Mounted on
shm 64M 0 64M 0% /dev/shm
```

#### persistent volume claim example

As an example, the IBM plugin we use to setup takes this approach, and then we define the existing volume on the level of the container:
Expand Down
12 changes: 12 additions & 0 deletions examples/dist/flux-operator-arm.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,12 @@ spec:
Config map name if the existing volume is a config map
You should also define items if you are using this
type: string
emptyDir:
default: false
type: boolean
emptyDirMedium:
description: Add an empty directory custom type
type: string
hostPath:
description: An existing hostPath to bind to path
type: string
Expand Down Expand Up @@ -729,6 +735,12 @@ spec:
Config map name if the existing volume is a config map
You should also define items if you are using this
type: string
emptyDir:
default: false
type: boolean
emptyDirMedium:
description: Add an empty directory custom type
type: string
hostPath:
description: An existing hostPath to bind to path
type: string
Expand Down
12 changes: 12 additions & 0 deletions examples/dist/flux-operator.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,12 @@ spec:
Config map name if the existing volume is a config map
You should also define items if you are using this
type: string
emptyDir:
default: false
type: boolean
emptyDirMedium:
description: Add an empty directory custom type
type: string
hostPath:
description: An existing hostPath to bind to path
type: string
Expand Down Expand Up @@ -729,6 +735,12 @@ spec:
Config map name if the existing volume is a config map
You should also define items if you are using this
type: string
emptyDir:
default: false
type: boolean
emptyDirMedium:
description: Add an empty directory custom type
type: string
hostPath:
description: An existing hostPath to bind to path
type: string
Expand Down
13 changes: 13 additions & 0 deletions examples/tests/emptydir/minicluster-without.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
apiVersion: flux-framework.org/v1alpha2
kind: MiniCluster
metadata:
name: flux-sample
spec:
size: 2
logging:
quiet: true

# This example shows the default shared memory (64M) without the empty dir
containers:
- image: rockylinux:9
command: df -h /dev/shm
18 changes: 18 additions & 0 deletions examples/tests/emptydir/minicluster.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
apiVersion: flux-framework.org/v1alpha2
kind: MiniCluster
metadata:
name: flux-sample
spec:
size: 2
logging:
quiet: true

# This example should show all the shared memory available on the node
containers:
- image: rockylinux:9
command: df -h /dev/shm
volumes:
# This must be all lowercase!
my-empty-dir:
emptyDir: true
emptyDirMedium: "memory"
2 changes: 2 additions & 0 deletions sdk/python/v1alpha2/docs/ContainerVolume.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ Name | Type | Description | Notes
------------ | ------------- | ------------- | -------------
**claim_name** | **str** | Claim name if the existing volume is a PVC | [optional]
**config_map_name** | **str** | Config map name if the existing volume is a config map You should also define items if you are using this | [optional]
**empty_dir** | **bool** | | [optional] [default to False]
**empty_dir_medium** | **str** | Add an empty directory custom type | [optional]
**host_path** | **str** | An existing hostPath to bind to path | [optional]
**items** | **dict[str, str]** | Items (key and paths) for the config map | [optional]
**path** | **str** | Path and claim name are always required if a secret isn't defined | [optional]
Expand Down
Loading
Loading