Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial work for clouds secrets and user-defined configmap #205

Closed
wants to merge 2 commits into from
+186 −25
Closed

Initial work for clouds secrets and user-defined configmap #205

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c8cdba9
Initial work for clouds secrets and user-defined configmap
alourie Jul 26, 2019
9f8b223
PR comments fixes
alourie Jul 31, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions deploy/crds/cassandraoperator_v1alpha1_cassandradatacenter_crd.yaml
View file
Original file line number Diff line number Diff line change
@@ -27,6 +27,8 @@ spec:
type: object
spec:
properties:
backupSecretVolume:
type: object
cassandraImage:
type: string
cluster:
@@ -35,6 +37,10 @@ spec:
type: string
dataVolumeClaimSpec:
type: object
env:
items:
type: object
type: array
imagePullPolicy:
type: string
imagePullSecrets:
@@ -44,12 +50,16 @@ spec:
nodes:
format: int32
type: integer
privilegedSupported:
type: boolean
prometheusSupport:
type: boolean
resources:
type: object
sidecarImage:
type: string
userConfigMapVolumeSource:
type: object
required:
- nodes
- cassandraImage
20 changes: 20 additions & 0 deletions examples/go/example-datacenter.yaml
View file
Original file line number Diff line number Diff line change
@@ -7,9 +7,29 @@ metadata:
spec:
cluster: "test-cluster"
nodes: 3
racks: 3
cassandraImage: "gcr.io/cassandra-operator/cassandra:3.11.3"
sidecarImage: "gcr.io/cassandra-operator/cassandra-sidecar:latest"
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: regcred
backupSecretVolume:
secretName: backup-secret
# type is a workaround for https://github.com/kubernetes/kubernetes/issues/68466
type: array
items:
- key: creds.json
path: creds.json
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: "/etc/google/creds.json"
userConfigMapVolumeSource:
name: concurrent-data
# type is a workaround for https://github.com/kubernetes/kubernetes/issues/68466
type: array
items:
- key: "concurrent.yaml"
path: "cassandra.yaml.d/100-user.yaml"
resources:
limits:
memory: 1Gi
25 changes: 13 additions & 12 deletions pkg/apis/cassandraoperator/v1alpha1/cassandradatacenter.go
View file
Original file line number Diff line number Diff line change
@@ -10,18 +10,19 @@ import (
type CassandraDataCenterSpec struct {
// Cluster is either a string or v1.LocalObjectReference
//Cluster interface{} `json:"cluster,omitempty"`
Cluster string `json:"cluster,omitempty"`
Nodes int32 `json:"nodes"`
CassandraImage string `json:"cassandraImage"`
SidecarImage string `json:"sidecarImage"`
ImagePullPolicy v1.PullPolicy `json:"imagePullPolicy"`
ImagePullSecrets []v1.LocalObjectReference `json:"imagePullSecrets,omitempty"`

Resources v1.ResourceRequirements `json:"resources"`

DataVolumeClaimSpec v1.PersistentVolumeClaimSpec `json:"dataVolumeClaimSpec"`

PrometheusSupport bool `json:"prometheusSupport"`
Cluster string `json:"cluster,omitempty"`
Nodes int32 `json:"nodes"`
CassandraImage string `json:"cassandraImage"`
SidecarImage string `json:"sidecarImage"`
ImagePullPolicy v1.PullPolicy `json:"imagePullPolicy"`
ImagePullSecrets []v1.LocalObjectReference `json:"imagePullSecrets,omitempty"`
BackupSecretVolume v1.SecretVolumeSource `json:"backupSecretVolume,omitempty"`
UserConfigMapVolumeSource v1.ConfigMapVolumeSource `json:"userConfigMapVolumeSource,omitempty"`
Resources v1.ResourceRequirements `json:"resources"`
DataVolumeClaimSpec v1.PersistentVolumeClaimSpec `json:"dataVolumeClaimSpec"`
PrivilegedSupported bool `json:"privilegedSupported,omitempty"`
PrometheusSupport bool `json:"prometheusSupport"`
Env []v1.EnvVar `json:"env,omitempty"`
}

// CassandraDataCenterStatus defines the observed state of CassandraDataCenter
32 changes: 30 additions & 2 deletions pkg/apis/cassandraoperator/v1alpha1/zz_generated.deepcopy.go
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

30 changes: 29 additions & 1 deletion pkg/apis/cassandraoperator/v1alpha1/zz_generated.openapi.go
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions pkg/controller/cassandradatacenter/configmap.go
View file
Original file line number Diff line number Diff line change
@@ -37,6 +37,7 @@ func createOrUpdateOperatorConfigMap(rctx *reconciliationRequestContext, seedNod

addPrometheusSupport(rctx.cdc, addFileFn)


if err := controllerutil.SetControllerReference(rctx.cdc, configMap, rctx.scheme); err != nil {
return err
}
93 changes: 83 additions & 10 deletions pkg/controller/cassandradatacenter/statefulset.go
View file
Original file line number Diff line number Diff line change
@@ -12,21 +12,32 @@ import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/runtime"
"path"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"strings"
"sync"
)

const DataVolumeMountPath = "/var/lib/cassandra"
const (
DataVolumeMountPath = "/var/lib/cassandra"
OperatorConfigVolumeMountPath = "/tmp/operator-config"
UserConfigVolumeMountPath = "/tmp/user-config"
)

const SidecarApiPort = 4567

const GOOGLE_APPLICATION_CREDENTIALS = "GOOGLE_APPLICATION_CREDENTIALS"

var sidecarClientOptions = sidecar.ClientOptions{
Port: SidecarApiPort,
Secure: false,
}

func boolRef(b bool) *bool {
return &b
}

func createOrUpdateStatefulSet(rctx *reconciliationRequestContext, configVolume *corev1.Volume) (*v1beta2.StatefulSet, error) {
statefulSet := &v1beta2.StatefulSet{ObjectMeta: DataCenterResourceMetadata(rctx.cdc)}

@@ -39,11 +50,12 @@ func createOrUpdateStatefulSet(rctx *reconciliationRequestContext, configVolume
}

dataVolumeClaim := newDataVolumeClaim(&rctx.cdc.Spec.DataVolumeClaimSpec)

podInfoVolume := newPodInfoVolume()
backupSecretVolume := newBackupSecretVolume(rctx)
userConfigVolume := newUserConfigVolume(rctx)

cassandraContainer := newCassandraContainer(rctx.cdc, dataVolumeClaim, configVolume)
sidecarContainer := newSidecarContainer(rctx.cdc, dataVolumeClaim, podInfoVolume)
cassandraContainer := newCassandraContainer(rctx.cdc, dataVolumeClaim, configVolume, userConfigVolume)
sidecarContainer := newSidecarContainer(rctx.cdc, dataVolumeClaim, podInfoVolume, backupSecretVolume)

sysctlLimitsContainer := newSysctlLimitsContainer(rctx.cdc)

@@ -52,6 +64,14 @@ func createOrUpdateStatefulSet(rctx *reconciliationRequestContext, configVolume
[]corev1.Container{*cassandraContainer, *sidecarContainer},
[]corev1.Container{*sysctlLimitsContainer})

if backupSecretVolume != nil {
podSpec.Volumes = append(podSpec.Volumes, *backupSecretVolume)
}

if userConfigVolume != nil {
podSpec.Volumes = append(podSpec.Volumes, *userConfigVolume)
}

statefulSetSpec := newStatefulSetSpec(rctx.cdc, podSpec, dataVolumeClaim)

if statefulSet.CreationTimestamp.IsZero() {
@@ -100,9 +120,7 @@ func newPodSpec(cdc *cassandraoperatorv1alpha1.CassandraDataCenter, volumes []co
return podSpec
}

func newCassandraContainer(cdc *cassandraoperatorv1alpha1.CassandraDataCenter, dataVolumeClaim *corev1.PersistentVolumeClaim, configVolume *corev1.Volume) *corev1.Container {
const OperatorConfigVolumeMountPath = "/tmp/operator-config"

func newCassandraContainer(cdc *cassandraoperatorv1alpha1.CassandraDataCenter, dataVolumeClaim *corev1.PersistentVolumeClaim, configVolume *corev1.Volume, userConfigVolume *corev1.Volume) *corev1.Container {
container := &corev1.Container{
Name: "cassandra",
Image: cdc.Spec.CassandraImage,
@@ -137,15 +155,20 @@ func newCassandraContainer(cdc *cassandraoperatorv1alpha1.CassandraDataCenter, d
},
}

if userConfigVolume != nil {
container.Args = append(container.Args, UserConfigVolumeMountPath)
container.VolumeMounts = append(container.VolumeMounts, corev1.VolumeMount{Name: userConfigVolume.Name, MountPath: UserConfigVolumeMountPath})
}

if cdc.Spec.PrometheusSupport == true {
container.Ports = append(container.Ports, corev1.ContainerPort{Name: "promql", ContainerPort: 9500})
}

return container
}

func newSidecarContainer(cdc *cassandraoperatorv1alpha1.CassandraDataCenter, dataVolumeClaim *corev1.PersistentVolumeClaim, podInfoVolume *corev1.Volume) *corev1.Container {
return &corev1.Container{
func newSidecarContainer(cdc *cassandraoperatorv1alpha1.CassandraDataCenter, dataVolumeClaim *corev1.PersistentVolumeClaim, podInfoVolume *corev1.Volume, backupSecretVolume *corev1.Volume) *corev1.Container {
container := &corev1.Container{
Name: "sidecar",
Image: cdc.Spec.SidecarImage,
ImagePullPolicy: cdc.Spec.ImagePullPolicy,
@@ -157,6 +180,30 @@ func newSidecarContainer(cdc *cassandraoperatorv1alpha1.CassandraDataCenter, dat
{Name: podInfoVolume.Name, MountPath: "/etc/pod-info"},
},
}

if len(cdc.Spec.Env) > 0 {
container.Env = cdc.Spec.Env
}

if backupSecretVolume != nil {
// find GOOGLE_APPLICATION_CREDENTIALS in env:
google_creds_path := "/etc/gcp"
for _, env := range cdc.Spec.Env {
if env.Name == GOOGLE_APPLICATION_CREDENTIALS {
google_creds_path = path.Dir(env.Value)
}
}

if google_creds_path == "/etc/gcp" {
// environment not set, log it, but it may be not an issue
log.Info("Warning: backupSecretVolume is set, but GOOGLE_APPLICATION_CREDENTIALS env is missing")
}

// Use it for the mount
container.VolumeMounts = append(container.VolumeMounts, corev1.VolumeMount{Name: backupSecretVolume.Name, MountPath: google_creds_path})
}

return container
}

func newSysctlLimitsContainer(cdc *cassandraoperatorv1alpha1.CassandraDataCenter) *corev1.Container {
@@ -165,7 +212,7 @@ func newSysctlLimitsContainer(cdc *cassandraoperatorv1alpha1.CassandraDataCenter
Image: cdc.Spec.CassandraImage,
ImagePullPolicy: cdc.Spec.ImagePullPolicy,
SecurityContext: &corev1.SecurityContext{
Privileged: func() *bool { b := true; return &b }(),
Privileged: boolRef(cdc.Spec.PrivilegedSupported),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a point in running the sysctl init container unprivileged? Shouldn't we completely drop the container if cdc.Spec.PrivilegedSupported == false?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a very good question. In Java version we did this, so I assumed we might want to do this here too. The cluster seems to be working fine in my local testing, but I'm not sure it actually tests this flow properly.

Copy link
Contributor

@johananl johananl Aug 5, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is what happens when the sysctl container runs unprivileged:

kubectl logs -f cassandra-test-dc-cassandra-0 -c sysctl-limits
+ sysctl -w vm.max_map_count=1048575
sysctl: setting key "vm.max_map_count": Read-only file system
+ true

The pod starts fine because we're doing || true, however it has no effect in this case as it cannot modify the setting on the host.

AFAICT we can drop this container when we can't run as privileged.

},
Command: []string{"bash", "-xuec"},
Args: []string{
@@ -174,6 +221,32 @@ func newSysctlLimitsContainer(cdc *cassandraoperatorv1alpha1.CassandraDataCenter
}
}

func newUserConfigVolume(rctx *reconciliationRequestContext) *corev1.Volume {

// check if set
if len(rctx.cdc.Spec.UserConfigMapVolumeSource.Name) == 0 {
return nil
}

return &corev1.Volume{
Name: rctx.cdc.Spec.UserConfigMapVolumeSource.Name,
VolumeSource: corev1.VolumeSource{ConfigMap: &rctx.cdc.Spec.UserConfigMapVolumeSource},
}
}

func newBackupSecretVolume(rctx *reconciliationRequestContext) *corev1.Volume {

// check if set
if len(rctx.cdc.Spec.BackupSecretVolume.SecretName) == 0 {
return nil
}

return &corev1.Volume{
Name: rctx.cdc.Spec.BackupSecretVolume.SecretName,
VolumeSource: corev1.VolumeSource{Secret: &rctx.cdc.Spec.BackupSecretVolume},
}
}

func newPodInfoVolume() *corev1.Volume {
return &corev1.Volume{
Name: "pod-info",