Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: updated STLS bootstrap integration with linux CSE #4386

Open
wants to merge 70 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
75e0a07
feat: STLS bootstrap integration with linux CSE
Apr 5, 2024
bc9d6fa
feat: impl
Apr 5, 2024
d78b2c1
Merge branch 'master' of https://github.com/Azure/AgentBaker into cam…
Apr 5, 2024
6fa54a2
chore: remove debug code
Apr 5, 2024
5f24b0f
Merge branch 'master' of https://github.com/Azure/AgentBaker into cam…
Apr 8, 2024
5c9ca22
refactor: download binary as separate unit, don't block on enable and…
Apr 8, 2024
2cfa54b
refactor: consolidate download into single unit
Apr 9, 2024
957efa5
chore: Before=kubelet.service
Apr 9, 2024
1f6c6d8
Merge branch 'master' of https://github.com/Azure/AgentBaker into cam…
Apr 9, 2024
607a090
chore: update version
Apr 9, 2024
a7d0c67
chore: remove dep on kubelet service
Apr 9, 2024
d5f1c31
chore: typo
Apr 9, 2024
782f779
fix: unbound var
Apr 10, 2024
319c54e
fix: bump client version for fixes
Apr 11, 2024
aaead67
fix: typos
Apr 11, 2024
80f990b
chore: generate
Apr 11, 2024
164ec41
fix: properly write SP file
Apr 11, 2024
475b644
chore: generate
Apr 11, 2024
e10dea1
chore: missing var
Apr 11, 2024
d114cd4
fix: apiserver fqdn, bump client version
Apr 11, 2024
18d9edb
fix: port
Apr 11, 2024
6359791
chore: resolve conflicts
Apr 12, 2024
97efab9
fix: only specify bootstrap-kubeconfig is valid kubeconfig does not e…
Apr 12, 2024
58cbac0
chore: bump client version and specify client cert/key file paths
Apr 12, 2024
0b91a35
chore: bump client version for bug
Apr 15, 2024
c568c8a
chore: conflicts
Apr 15, 2024
ce478fc
fix: guard client cert/key pair generation from RP
Apr 15, 2024
627b8dd
Merge branch 'master' of https://github.com/Azure/AgentBaker into cam…
Apr 18, 2024
b3a0cc5
chore: bump client version again for fixes
Apr 18, 2024
7b19852
chore: better logging
Apr 24, 2024
6f0b360
chore: log service status as event
Apr 24, 2024
5201760
feat: log bootstrap journal in events
Apr 24, 2024
ed1c29a
chore: fix event logging for errs
Apr 24, 2024
2ab182e
chore: cleanup
May 1, 2024
faf33f3
chore: tweak event logging
May 1, 2024
8ce2375
chore: resolve conflicts
May 6, 2024
3e5e7e1
Merge branch 'master' of https://github.com/Azure/AgentBaker into cam…
May 6, 2024
f18ed87
Merge branch 'master' of https://github.com/Azure/AgentBaker into cam…
May 8, 2024
09f5ef6
feat: initial impl for bootstrapping post-CSE
May 9, 2024
e9d75ce
chore: update kubelet service file
May 9, 2024
a0f361e
chore: more impl
May 9, 2024
2dc3d11
chore: remove kubelet-bootstrap.service
May 9, 2024
3f39345
chore: remove stls bootstrap service
May 9, 2024
5432003
Merge branch 'master' of https://github.com/Azure/AgentBaker into cam…
May 9, 2024
68ed294
chore: fix tests
May 9, 2024
04f9e74
Merge branch 'master' of https://github.com/Azure/AgentBaker into cam…
May 9, 2024
3908614
chore: update download url
May 9, 2024
074126d
chore: fix url
May 9, 2024
c01d45e
chore: remove stls bootstrap service
May 9, 2024
89cc014
chore: refactor ExecStartPre
May 9, 2024
3cf8573
fix: don't log bootstrap tokens
May 9, 2024
45209d4
fixes
May 9, 2024
ff3ed23
chore: more fixes
May 10, 2024
d81a2f4
chore: refactor
May 10, 2024
d4ce1c7
chore: more refactoring
May 10, 2024
ab7c029
chore: enable and start no wait
May 10, 2024
03c235c
tweak
May 10, 2024
f7888ca
chore: no block on systemctl start kubelet
May 11, 2024
7719d75
chore: update client version
May 13, 2024
1f1bf72
chore: conflicts
May 14, 2024
51dc3fc
chore: cleanup
May 14, 2024
05320cf
chore: more cleanup
May 14, 2024
279add8
chore: typo
May 14, 2024
b10b0f0
tweaks
May 14, 2024
337ba54
chore: conflicts
May 16, 2024
60d6453
chore: conflicts
May 17, 2024
f9da79a
Merge branch 'master' of https://github.com/Azure/AgentBaker into cam…
May 22, 2024
f7b3891
chore: add logs, remove hostname from event telemetry
May 24, 2024
a7e5064
chore: conflicts
May 24, 2024
030bacd
chore: generate
May 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion parts/linux/cloud-init/artifacts/cse_cmd.sh
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ NO_PROXY_URLS="{{GetNoProxy}}"
PROXY_VARS="{{GetProxyVariables}}"
ENABLE_TLS_BOOTSTRAPPING="{{EnableTLSBootstrapping}}"
ENABLE_SECURE_TLS_BOOTSTRAPPING="{{EnableSecureTLSBootstrapping}}"
CUSTOM_SECURE_TLS_BOOTSTRAP_AAD_SERVER_APP_ID="{{GetCustomSecureTLSBootstrapAADServerAppID}}"
CUSTOM_SECURE_TLS_BOOTSTRAP_AAD_RESOURCE="{{GetCustomSecureTLSBootstrapAADResource}}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cse_cmd.sh var naming change, want to loop in @Bryce-Soghigian @tallaxes here for viz, however, I dont think this variable is being used yet that the renaming would break them?

still tagging them.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, keep forgetting

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack, and appreciated. Not used yet, so should be fine - but only as long as not setting it at all is tolerated by the rest of the logic.

DHCPV6_SERVICE_FILEPATH="{{GetDHCPv6ServiceCSEScriptFilepath}}"
DHCPV6_CONFIG_FILEPATH="{{GetDHCPv6ConfigCSEScriptFilepath}}"
THP_ENABLED="{{GetTransparentHugePageEnabled}}"
Expand Down
92 changes: 42 additions & 50 deletions parts/linux/cloud-init/artifacts/cse_config.sh
Original file line number Diff line number Diff line change
Expand Up @@ -293,6 +293,27 @@ configureCNIIPTables() {
fi
}

configureKubeletSecureTLSBootstrap() {
# default AAD resource here so we can minimze bootstrap contract surface
AAD_RESOURCE="6dae42f8-4368-4678-94ff-3960e28e3630"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If user doesn't set CUSTOM_SECURE_TLS_BOOTSTRAP_AAD_RESOURCE, AAD_RESOURCE is defaulted to 6dae42f8-4368-4678-94ff-3960e28e3630. Does it mean this AAD_RESOURCE is always available in every user's subscription?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, as far as I know - this is the app ID of AKS's AAD server, I've spoken with Baichao and Weinong about this:

https://azure.github.io/kubelogin/concepts/aks.html

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

though will be a good idea to confirm this is correct gov environments as well

if [ -n "$CUSTOM_SECURE_TLS_BOOTSTRAP_AAD_RESOURCE" ]; then
AAD_RESOURCE="$CUSTOM_SECURE_TLS_BOOTSTRAP_AAD_RESOURCE"
fi
TIMEOUT_START_SECONDS=270 # 90s (default) + 180s for secure bootstrapping = 4.5 minutes

SECURE_TLS_BOOTSTRAP_KUBELET_DROP_IN=/etc/systemd/system/kubelet.service.d/10-securetlsbootstrap.conf
mkdir -p "$(dirname "${SECURE_TLS_BOOTSTRAP_KUBELET_DROP_IN}")"
touch "${SECURE_TLS_BOOTSTRAP_KUBELET_DROP_IN}"
chmod 0600 "${SECURE_TLS_BOOTSTRAP_KUBELET_DROP_IN}"
cat > "${SECURE_TLS_BOOTSTRAP_KUBELET_DROP_IN}" <<EOF
[Service]
TimeoutStartSec=${TIMEOUT_START_SECONDS}
ExecStartPre=-/opt/azure/tlsbootstrap/secure-tls-bootstrap.sh
Environment="SECURE_TLS_BOOTSTRAP_AAD_RESOURCE=${AAD_RESOURCE}"
Environment="API_SERVER_NAME=${API_SERVER_NAME}"
EOF
}

disableSystemdResolved() {
ls -ltr /etc/resolv.conf
cat /etc/resolv.conf
Expand Down Expand Up @@ -402,60 +423,15 @@ ensureKubelet() {
if [ -n "${AZURE_ENVIRONMENT_FILEPATH}" ]; then
echo "AZURE_ENVIRONMENT_FILEPATH=${AZURE_ENVIRONMENT_FILEPATH}" >> "${KUBELET_DEFAULT_FILE}"
fi

KUBE_CA_FILE="/etc/kubernetes/certs/ca.crt"
mkdir -p "$(dirname "${KUBE_CA_FILE}")"
echo "${KUBE_CA_CRT}" | base64 -d > "${KUBE_CA_FILE}"
chmod 0600 "${KUBE_CA_FILE}"

if [ "${ENABLE_SECURE_TLS_BOOTSTRAPPING}" == "true" ] || [ "${ENABLE_TLS_BOOTSTRAPPING}" == "true" ]; then
KUBELET_TLS_DROP_IN="/etc/systemd/system/kubelet.service.d/10-tlsbootstrap.conf"
mkdir -p "$(dirname "${KUBELET_TLS_DROP_IN}")"
touch "${KUBELET_TLS_DROP_IN}"
chmod 0600 "${KUBELET_TLS_DROP_IN}"
tee "${KUBELET_TLS_DROP_IN}" > /dev/null <<EOF
[Service]
Environment="KUBELET_TLS_BOOTSTRAP_FLAGS=--kubeconfig /var/lib/kubelet/kubeconfig --bootstrap-kubeconfig /var/lib/kubelet/bootstrap-kubeconfig"
EOF
fi

if [ "${ENABLE_SECURE_TLS_BOOTSTRAPPING}" == "true" ]; then
AAD_RESOURCE="6dae42f8-4368-4678-94ff-3960e28e3630"
if [ -n "$CUSTOM_SECURE_TLS_BOOTSTRAP_AAD_SERVER_APP_ID" ]; then
AAD_RESOURCE=$CUSTOM_SECURE_TLS_BOOTSTRAP_AAD_SERVER_APP_ID
fi
SECURE_BOOTSTRAP_KUBECONFIG_FILE=/var/lib/kubelet/bootstrap-kubeconfig
mkdir -p "$(dirname "${SECURE_BOOTSTRAP_KUBECONFIG_FILE}")"
touch "${SECURE_BOOTSTRAP_KUBECONFIG_FILE}"
chmod 0644 "${SECURE_BOOTSTRAP_KUBECONFIG_FILE}"
tee "${SECURE_BOOTSTRAP_KUBECONFIG_FILE}" > /dev/null <<EOF
apiVersion: v1
kind: Config
clusters:
- name: localcluster
cluster:
certificate-authority: /etc/kubernetes/certs/ca.crt
server: https://${API_SERVER_NAME}:443
users:
- name: kubelet-bootstrap
user:
exec:
apiVersion: client.authentication.k8s.io/v1
command: /opt/azure/tlsbootstrap/tls-bootstrap-client
args:
- bootstrap
- --next-proto=aks-tls-bootstrap
- --aad-resource=${AAD_RESOURCE}
interactiveMode: Never
provideClusterInfo: true
contexts:
- context:
cluster: localcluster
user: kubelet-bootstrap
name: bootstrap-context
current-context: bootstrap-context
EOF
elif [ "${ENABLE_TLS_BOOTSTRAPPING}" == "true" ]; then
if [ "${ENABLE_TLS_BOOTSTRAPPING}" == "true" ]; then
# used in vanilla TLS bootstrapping cases and when secure TLS bootstrapping has failed to generate a kubeconfig
# by the time we need to start kubelet
BOOTSTRAP_KUBECONFIG_FILE=/var/lib/kubelet/bootstrap-kubeconfig
mkdir -p "$(dirname "${BOOTSTRAP_KUBECONFIG_FILE}")"
touch "${BOOTSTRAP_KUBECONFIG_FILE}"
Expand All @@ -479,7 +455,13 @@ contexts:
name: bootstrap-context
current-context: bootstrap-context
EOF
else
fi

if [ "${ENABLE_SECURE_TLS_BOOTSTRAPPING}" == "true" ]; then
configureKubeletSecureTLSBootstrap
fi

if [ "${ENABLE_SECURE_TLS_BOOTSTRAPPING}" == "false" ] && [ "${ENABLE_TLS_BOOTSTRAPPING}" == "false" ]; then
KUBECONFIG_FILE=/var/lib/kubelet/kubeconfig
mkdir -p "$(dirname "${KUBECONFIG_FILE}")"
touch "${KUBECONFIG_FILE}"
Expand Down Expand Up @@ -528,6 +510,16 @@ EOF
logs_to_events "AKS.CSE.ensureKubelet.installCredentalProvider" installCredentalProvider
fi

if [ "${ENABLE_SECURE_TLS_BOOTSTRAPPING}" == "true" ]; then
# we continue with CSE without ensuring the kubelet startup completed
# due to user-assigned managed identities not being assigned by RP to agent nodes
# until after the initial PUT has been completed, thus we need to ensure to
# complete CSE as quickly as possible so secure bootstrapping has a chance to run
# in cases where the node is using a kubelet identity
systemctlEnableAndStartNoBlock kubelet || exit $ERR_KUBELET_START_FAIL
return 0
fi

systemctlEnableAndStart kubelet || exit $ERR_KUBELET_START_FAIL
}

Expand Down
30 changes: 29 additions & 1 deletion parts/linux/cloud-init/artifacts/cse_helpers.sh
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ ERR_ARTIFACT_STREAMING_INSTALL=153 # Error installing mirror proxy and overlaybd

ERR_HTTP_PROXY_CA_CONVERT=160 # Error converting http proxy ca cert from pem to crt format
ERR_UPDATE_CA_CERTS=161 # Error updating ca certs to include user-provided certificates
ERR_DOWNLOAD_SECURE_TLS_BOOTSTRAP_KUBELET_EXEC_PLUGIN_TIMEOUT=169 # Timeout waiting for secure TLS bootrstrap kubelet exec plugin download
ERR_DOWNLOAD_SECURE_TLS_BOOTSTRAP_CLIENT_BINARY=169 # Timeout waiting for secure TLS bootrstrap client binary download

ERR_DISBALE_IPTABLES=170 # Error disabling iptables service

Expand Down Expand Up @@ -231,6 +231,20 @@ systemctl_restart() {
fi
done
}
systemctl_restart_noblock() {
retries=$1; wait_sleep=$2; timeout=$3 svcname=$4
for i in $(seq 1 $retries); do
timeout $timeout systemctl daemon-reload
timeout $timeout systemctl restart $svcname --no-block && break || \
if [ $i -eq $retries ]; then
return 1
else
systemctl status $svcname --no-pager -l
journalctl -u $svcname
sleep $wait_sleep
fi
done
}
systemctl_stop() {
retries=$1; wait_sleep=$2; timeout=$3 svcname=$4
for i in $(seq 1 $retries); do
Expand Down Expand Up @@ -284,6 +298,20 @@ systemctlEnableAndStart() {
fi
}

systemctlEnableAndStartNoBlock() {
systemctl_restart_noblock 100 5 30 $1
RESTART_STATUS=$?
systemctl status $1 --no-pager -l > /var/log/azure/$1-status.log
if [ $RESTART_STATUS -ne 0 ]; then
echo "$1 could not be started"
return 1
fi
if ! retrycmd_if_failure 120 5 25 systemctl enable --no-block $1; then
echo "$1 could not be enabled by systemctl"
return 1
fi
}

systemctlDisableAndStop() {
if systemctl list-units --full --all | grep -q "$1.service"; then
systemctl_stop 20 5 25 $1 || echo "$1 could not be stopped"
Expand Down
29 changes: 13 additions & 16 deletions parts/linux/cloud-init/artifacts/cse_install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,6 @@ RUNC_DOWNLOADS_DIR="/opt/runc/downloads"
K8S_DOWNLOADS_DIR="/opt/kubernetes/downloads"
K8S_PRIVATE_PACKAGES_CACHE_DIR="/opt/kubernetes/downloads/private-packages"
UBUNTU_RELEASE=$(lsb_release -r -s)
SECURE_TLS_BOOTSTRAP_KUBELET_EXEC_PLUGIN_DOWNLOAD_DIR="/opt/azure/tlsbootstrap"
SECURE_TLS_BOOTSTRAP_KUBELET_EXEC_PLUGIN_VERSION="v0.1.0-alpha.2"
TELEPORTD_PLUGIN_DOWNLOAD_DIR="/opt/teleportd/downloads"
CREDENTIAL_PROVIDER_DOWNLOAD_DIR="/opt/credentialprovider/downloads"
CREDENTIAL_PROVIDER_BIN_DIR="/var/lib/kubelet/credential-provider"
Expand All @@ -22,6 +20,7 @@ CONTAINERD_WASM_VERSIONS="v0.3.0 v0.5.1 v0.8.0"
MANIFEST_FILEPATH="/opt/azure/manifest.json"
MAN_DB_AUTO_UPDATE_FLAG_FILEPATH="/var/lib/man-db/auto-update"
CURL_OUTPUT=/tmp/curl_verbose.out
SECURE_TLS_BOOTSTRAP_CLIENT_BINARY_VERSION="client-v0.1.0-alpha.4"

removeManDbAutoUpdateFlagFile() {
rm -f $MAN_DB_AUTO_UPDATE_FLAG_FILEPATH
Expand Down Expand Up @@ -92,19 +91,17 @@ installCredentalProvider() {
rm -rf ${CREDENTIAL_PROVIDER_DOWNLOAD_DIR}
}

downloadSecureTLSBootstrapKubeletExecPlugin() {
local plugin_url="https://k8sreleases.blob.core.windows.net/aks-tls-bootstrap-client/${SECURE_TLS_BOOTSTRAP_KUBELET_EXEC_PLUGIN_VERSION}/linux/amd64/tls-bootstrap-client"
if [[ $(isARM64) == 1 ]]; then
plugin_url="https://k8sreleases.blob.core.windows.net/aks-tls-bootstrap-client/${SECURE_TLS_BOOTSTRAP_KUBELET_EXEC_PLUGIN_VERSION}/linux/arm64/tls-bootstrap-client"
fi

mkdir -p $SECURE_TLS_BOOTSTRAP_KUBELET_EXEC_PLUGIN_DOWNLOAD_DIR
plugin_download_path="${SECURE_TLS_BOOTSTRAP_KUBELET_EXEC_PLUGIN_DOWNLOAD_DIR}/tls-bootstrap-client"

if [ ! -f "$plugin_download_path" ]; then
retrycmd_if_failure 30 5 60 curl -fSL -o "$plugin_download_path" "$plugin_url" || exit $ERR_DOWNLOAD_SECURE_TLS_BOOTSTRAP_KUBELET_EXEC_PLUGIN_TIMEOUT
chown -R root:root "$SECURE_TLS_BOOTSTRAP_KUBELET_EXEC_PLUGIN_DOWNLOAD_DIR"
chmod -R 755 "$SECURE_TLS_BOOTSTRAP_KUBELET_EXEC_PLUGIN_DOWNLOAD_DIR"
downloadSecureTLSBootstrapClient() {
CPU_ARCH=$(getCPUArch)
CLIENT_BINARY_DOWNLOAD_URL="https://kubernetesreleases.blob.core.windows.net/aks-tls-bootstrap-client/${SECURE_TLS_BOOTSTRAP_CLIENT_BINARY_VERSION}/linux/${CPU_ARCH}/tls-bootstrap-client"
CLIENT_BINARY_DIR="/opt/azure/tlsbootstrap"
CLIENT_BINARY_PATH="${CLIENT_BINARY_DIR}/tls-bootstrap-client"

mkdir -p $CLIENT_BINARY_DIR
if [ ! -f "$CLIENT_BINARY_PATH" ]; then
retrycmd_if_failure 30 5 60 curl -fSL -o "$CLIENT_BINARY_PATH" "$CLIENT_BINARY_DOWNLOAD_URL" || exit $ERR_DOWNLOAD_SECURE_TLS_BOOTSTRAP_CLIENT_BINARY
chown -R root:root "$CLIENT_BINARY_DIR"
chmod -R 755 "$CLIENT_BINARY_DIR"
fi
}

Expand Down Expand Up @@ -486,4 +483,4 @@ datasource:
apply_network_config: false
EOF
}
#EOF
#EOF
2 changes: 1 addition & 1 deletion parts/linux/cloud-init/artifacts/cse_main.sh
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ if [ "${IS_KRUSTLET}" == "true" ]; then
fi

if [ "${ENABLE_SECURE_TLS_BOOTSTRAPPING}" == "true" ]; then
logs_to_events "AKS.CSE.downloadSecureTLSBootstrapKubeletExecPlugin" downloadSecureTLSBootstrapKubeletExecPlugin
logs_to_events "AKS.CSE.downloadSecureTLSBootstrapClient" downloadSecureTLSBootstrapClient
fi

# By default, never reboot new nodes.
Expand Down
13 changes: 2 additions & 11 deletions parts/linux/cloud-init/artifacts/kubelet.service
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
[Unit]
Description=Kubelet
ConditionPathExists=/usr/local/bin/kubelet
ConditionPathExists=/opt/azure/containers/start-kubelet.sh
Wants=network-online.target containerd.service
After=network-online.target containerd.service

Expand All @@ -19,17 +20,7 @@ ExecStartPre=/bin/mount --make-shared /var/lib/kubelet
ExecStartPre=-/sbin/ebtables -t nat --list
ExecStartPre=-/sbin/iptables -t nat --numeric --list

ExecStart=/usr/local/bin/kubelet \
--enable-server \
--node-labels="${KUBELET_NODE_LABELS}" \
--v=2 \
--volume-plugin-dir=/etc/kubernetes/volumeplugins \
$KUBELET_TLS_BOOTSTRAP_FLAGS \
$KUBELET_CONFIG_FILE_FLAGS \
$KUBELET_CONTAINERD_FLAGS \
$KUBELET_CONTAINER_RUNTIME_FLAG \
$KUBELET_CGROUP_FLAGS \
$KUBELET_FLAGS
ExecStart=/opt/azure/containers/start-kubelet.sh

[Install]
WantedBy=multi-user.target
93 changes: 93 additions & 0 deletions parts/linux/cloud-init/artifacts/secure-tls-bootstrap.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
#!/bin/bash

set -euxo pipefail

EVENTS_LOGGING_DIR="/var/log/azure/Microsoft.Azure.Extensions.CustomScript/events"
NEXT_PROTO_VALUE="aks-tls-bootstrap"

RETRY_PERIOD_SECONDS=180 # 3 minutes
RETRY_WAIT_SECONDS=3

AAD_RESOURCE="${SECURE_TLS_BOOTSTRAP_AAD_RESOURCE:-""}"
API_SERVER_NAME="${API_SERVER_NAME:-""}"

CLIENT_BINARY_PATH="${SECURE_TLS_BOOTSTRAP_CLIENT_BINARY_PATH:-/opt/azure/tlsbootstrap/tls-bootstrap-client}"
KUBECONFIG_PATH="${SECURE_TLS_BOOTSTRAP_KUBECONFIG_PATH:-/var/lib/kubelet/kubeconfig}"
CLIENT_CERT_PATH="${SECURE_TLS_BOOTSTRAP_CLIENT_CERT_PATH:-/etc/kubernetes/certs/client.crt}"
CLIENT_KEY_PATH="${SECURE_TLS_BOOTSTRAP_CLIENT_KEY_PATH:-/etc/kubernetes/certs/client.key}"
AZURE_CONFIG_PATH="${SECURE_TLS_BOOTSTRAP_ZURE_CONFIG_PATH:-/etc/kubernetes/azure.json}"
CLUSTER_CA_FILE_PATH="${SECURE_TLS_BOOTSTRAP_CLUSTER_CA_FILE_PATH:-/etc/kubernetes/certs/ca.crt}"
LOG_FILE_PATH="${SECURE_TLS_BOOTSTRAP_LOG_FILE_PATH:-/var/log/azure/aks/secure-tls-bootstrap.log}"

logs_to_events() {
local task=$1; shift
local eventsFileName=$(date +%s%3N)

local startTime=$(date +"%F %T.%3N")
${@}
ret=$?
local endTime=$(date +"%F %T.%3N")

msg_string=$(jq -n --arg Status "Succeeded" '{Status: $Status}')
if [ "$ret" != "0" ]; then
msg_string=$(jq -n --arg Status "Failed" --arg LogTail "$(tail -n 10 $LOG_FILE_PATH)" '{Status: $Status, LogTail: $LogTail}')
fi

json_string=$( jq -n \
--arg Timestamp "${startTime}" \
--arg OperationId "${endTime}" \
--arg Version "1.23" \
--arg TaskName "${task}" \
--arg EventLevel "Informational" \
--arg Message "${msg_string}" \
--arg EventPid "0" \
--arg EventTid "0" \
'{Timestamp: $Timestamp, OperationId: $OperationId, Version: $Version, TaskName: $TaskName, EventLevel: $EventLevel, Message: $Message, EventPid: $EventPid, EventTid: $EventTid}'
)
echo ${json_string} > "${EVENTS_LOGGING_DIR}/${eventsFileName}.json"

if [ "$ret" != "0" ]; then
return $ret
fi
}

bootstrap() {
if [ -z "$API_SERVER_NAME" ]; then
echo "ERROR: missing apiserver FQDN, cannot continue bootstrapping"
return 1
fi
if [ ! -f "$CLIENT_BINARY_PATH" ]; then
echo "ERROR: bootstrap client binary does not exist at path $CLIENT_BINARY_PATH"
return 1
fi

chmod +x $CLIENT_BINARY_PATH

deadline=$(($(date +%s) + RETRY_PERIOD_SECONDS))
while true; do
now=$(date +%s)
if [ $((now - deadline)) -ge 0 ]; then
echo "ERROR: bootstrapping deadline exceeded"
return 1
fi

$CLIENT_BINARY_PATH bootstrap \
--aad-resource="$AAD_RESOURCE" \
--apiserver-fqdn="$API_SERVER_NAME" \
--cluster-ca-file="$CLUSTER_CA_FILE_PATH" \
--azure-config="$AZURE_CONFIG_PATH" \
--cert-file="$CLIENT_CERT_PATH" \
--key-file="$CLIENT_KEY_PATH" \
--next-proto="$NEXT_PROTO_VALUE" \
--kubeconfig="$KUBECONFIG_PATH" \
--log-file="$LOG_FILE_PATH"

[ $? -eq 0 ] && echo "secure TLS bootstrapping succeeded, generated kubeconfig is at ${KUBECONFIG_PATH}" && break

sleep $RETRY_WAIT_SECONDS
done
}

logs_to_events "AKS.performSecureTLSBootstrapping" bootstrap || exit $?

#EOF
Loading
Loading