Skip to content

Commit

Permalink
Merge branch 'develop' into fix/helm-charts
Browse files Browse the repository at this point in the history
  • Loading branch information
lbeckman314 authored Oct 22, 2024
2 parents efd9def + 2531a4b commit 08cfeb0
Show file tree
Hide file tree
Showing 22 changed files with 402 additions and 60 deletions.
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Godoc](https://img.shields.io/badge/godoc-ref-blue.svg)](http://godoc.org/github.com/ohsu-comp-bio/funnel)


Funnel
======

https://ohsu-comp-bio.github.io/funnel/
<a title="Funnel Homepage" href="https://ohsu-comp-bio.github.io/funnel">
<img title="Funnel Logo" src="https://github.com/user-attachments/assets/f51cf06b-d802-4e20-bde1-bcd1fc5657e6" />
</a>

Funnel is a toolkit for distributed, batch task execution, including a server, worker, and a set of compute, storage, and database backends. Given a task description, Funnel will find a worker to execute the task, download inputs, run a series of (Docker) containers, upload outputs, capture logs, and track the whole process.

Funnel is an implementation of the [GA4GH Task Execution Schemas](https://github.com/ga4gh/task-execution-schemas), an effort to standardize the APIs used for task execution across many platforms.

Funnel provides an API server, multiple storage backends (local FS, S3, Google Bucket, etc.), multiple compute backends (local, HTCondor, Google Cloud, etc.), and a web dashboard.

https://ohsu-comp-bio.github.io/funnel/
3 changes: 2 additions & 1 deletion config/kubernetes-executor-template.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ spec:
serviceAccountName: funnel-sa
containers:
- name: funnel-worker-{{.TaskId}}
image: {{.Image}}
# TODO: Should this be configurable?
image: development
imagePullPolicy: Always
command: ["/bin/sh", "-c"]
args: {{.Command}}
Expand Down
21 changes: 5 additions & 16 deletions deployments/kubernetes/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
> [!WARNING]
> Funnel's Kubernetes support is in active development and may involve frequent updates 🚧
# Overview
Expand All @@ -16,7 +15,7 @@ Kuberenetes Resources:

## 1. Create a Service:

> *[funnel-service.yml](./funnel-service.yml)*
> *[funnel-service.yml](https://github.com/ohsu-comp-bio/funnel/blob/develop/deployments/kubernetes/funnel-service.yml)*
```sh
kubectl apply -f funnel-service.yml
Expand All @@ -38,16 +37,6 @@ sed -i "s|\${HOSTNAME}|${HOSTNAME}|g" funnel-worker.yaml

Use this value to configure the server hostname of the worker config.

## 2. Create Funnel config files

> *[funnel-server-config.yml](./funnel-server-config.yml)*
We recommend setting `DisableJobCleanup` to `true` for debugging - otherwise failed jobs will be cleanup up.

> *[funnel-worker-config.yml](./funnel-worker-config.yml)*
***Remember to modify the file to have the actual server hostname.***

## 3. Create a ConfigMap

```sh
Expand All @@ -58,9 +47,9 @@ kubectl create configmap funnel-config --from-file=funnel-server.yaml --from-fil

Define a Role and RoleBinding:

> *[role.yml](./role.yml)*
> *[role.yml](https://github.com/ohsu-comp-bio/funnel/blob/develop/deployments/kubernetes/role.yml)*
> *[role_binding.yml](./role_binding.yml)*
> *[role_binding.yml](https://github.com/ohsu-comp-bio/funnel/blob/develop/deployments/kubernetes/role_binding.yml)*
```sh
kubectl create serviceaccount funnel-sa --namespace default
Expand All @@ -72,15 +61,15 @@ kubectl apply -f role_binding.yml

Define a PVC for storage:

> *[funnel-storage-pvc.yml](./funnel-storage-pvc.yml)*
> *[funnel-storage-pvc.yml](https://github.com/ohsu-comp-bio/funnel/blob/develop/deployments/kubernetes/funnel-storage-pvc.yml)*
```sh
kubectl apply -f funnel-storage-pvc.yml
```

## 6. Create a Deployment

> *[funnel-deployment.yml](./funnel-deployment.yml)*
> *[funnel-deployment.yml](https://github.com/ohsu-comp-bio/funnel/blob/develop/deployments/kubernetes/funnel-deployment.yml)*
```sh
kubectl apply -f funnel-deployment.yml
Expand Down
2 changes: 1 addition & 1 deletion deployments/kubernetes/funnel-worker-config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,5 +21,5 @@ EventWriters:
- log

Server:
HostName: < funnel service clusterIP >
HostName: "${HOSTNAME}"
RPCPort: 9090
23 changes: 23 additions & 0 deletions deployments/kubernetes/helm/.helmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*.orig
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/
24 changes: 24 additions & 0 deletions deployments/kubernetes/helm/Chart.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
apiVersion: v2
name: funnel
description: A Helm chart for Kubernetes

# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
# to be deployed.
#
# Library charts provide useful utilities or functions for the chart developer. They're included as
# a dependency of application charts to inject those utilities and functions into the rendering
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
type: application

# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.1.0

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
# It is recommended to use it with quotes.
appVersion: "1.16.0"
5 changes: 5 additions & 0 deletions deployments/kubernetes/helm/templates/NOTES.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
1. To access the Funnel application, use the following instructions:

To access the service locally, use:
kubectl --namespace {{ .Release.Namespace }} port-forward svc/{{ include "funnel.fullname" . }} 8080:8000
echo "Visit http://127.0.0.1:8080"
62 changes: 62 additions & 0 deletions deployments/kubernetes/helm/templates/_helpers.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "funnel.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "funnel.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}

{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "funnel.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Common labels
*/}}
{{- define "funnel.labels" -}}
helm.sh/chart: {{ include "funnel.chart" . }}
{{ include "funnel.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

{{/*
Selector labels
*/}}
{{- define "funnel.selectorLabels" -}}
app.kubernetes.io/name: {{ include "funnel.name" . }}
app.kubernetes.io/instance: funnel
{{- end }}

{{/*
Create the name of the service account to use
*/}}
{{- define "funnel.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "funnel.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}
84 changes: 84 additions & 0 deletions deployments/kubernetes/helm/templates/configmap.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: funnel-config
data:
funnel-server-config.yaml: |-
Database: boltdb
Compute: kubernetes
Logger:
Level: debug
Kubernetes:
# The executor used to execute tasks. Available executors: docker, kubernetes
Executor: "kubernetes"
DisableJobCleanup: false
DisableReconciler: false
ReconcileRate: 5m
Namespace: default
Template: |
apiVersion: batch/v1
kind: Job
metadata:
## DO NOT CHANGE NAME
name: {{.TaskId}}
namespace: {{.Namespace}}
spec:
backoffLimit: 0
completions: 1
template:
spec:
restartPolicy: Never
serviceAccountName: funnel-sa
containers:
- name: {{printf "funnel-worker-%s" .TaskId}}
image: quay.io/ohsu-comp-bio/funnel:latest
imagePullPolicy: IfNotPresent
args:
- "worker"
- "run"
- "--config"
- "/etc/config/funnel-worker-config.yaml"
- "--taskID"
- {{.TaskId}}
resources:
requests:
cpu: {{if ne .Cpus 0 -}}{{.Cpus}}{{ else }}{{"100m"}}{{end}}
memory: {{if ne .RamGb 0.0 -}}{{printf "%.0fG" .RamGb}}{{else}}{{"16M"}}{{end}}
ephemeral-storage: {{if ne .DiskGb 0.0 -}}{{printf "%.0fG" .DiskGb}}{{else}}{{"100M"}}{{end}}
volumeMounts:
- name: {{printf "funnel-storage-%s" .TaskId}}
mountPath: {{printf "/opt/funnel/funnel-work-dir/%s" .TaskId}}
- name: config-volume
mountPath: /etc/config
securityContext:
privileged: true
volumes:
- name: {{printf "funnel-storage-%s" .TaskId}}
emptyDir: {}
- name: config-volume
configMap:
name: funnel-config
funnel-worker-config.yaml: |-
Database: boltdb
BoltDB:
Path: /opt/funnel/funnel-work-dir/funnel.bolt.db
Compute: kubernetes
Kubernetes:
Executor: "kubernetes"
Logger:
Level: debug
RPCClient:
MaxRetries: 3
Timeout: 30s
EventWriters:
- rpc
- log
Server:
HostName: ""
RPCPort: 9090
51 changes: 51 additions & 0 deletions deployments/kubernetes/helm/templates/deployment.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "funnel.fullname" . }}
labels:
{{- include "funnel.labels" . | nindent 4 }}
spec:
replicas: 1
selector:
matchLabels:
{{- include "funnel.selectorLabels" . | nindent 6 }}
template:
metadata:
labels:
{{- include "funnel.labels" . | nindent 8 }}
spec:
serviceAccountName: {{ include "funnel.serviceAccountName" . }}
containers:
- name: funnel
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: Always
command:
- 'funnel'
- 'server'
- 'run'
- '--config'
- '/etc/config/funnel-server-config.yaml'
resources:
requests:
cpu: {{ .Values.resources.requests.cpu }}
memory: {{ .Values.resources.requests.memory }}
limits:
cpu: {{ .Values.resources.limits.cpu }}
memory: {{ .Values.resources.limits.memory }}
ports:
- name: http
containerPort: 8000
- name: rpc
containerPort: 9090
volumeMounts:
- name: funnel-deployment-storage
mountPath: /opt/funnel/funnel-work-dir
- name: config-volume
mountPath: /etc/config
volumes:
- name: funnel-deployment-storage
persistentVolumeClaim:
claimName: funnel-pvc
- name: config-volume
configMap:
name: funnel-config
16 changes: 16 additions & 0 deletions deployments/kubernetes/helm/templates/role.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{{- if .Values.rbac.create }}
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ include "funnel.fullname" . }}-role
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list", "watch"]
- apiGroups: ["batch", "extensions"]
resources: ["jobs"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["extensions", "apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
{{- end }}
13 changes: 13 additions & 0 deletions deployments/kubernetes/helm/templates/rolebinding.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{{- if .Values.rbac.create }}
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: funnel-rolebinding
subjects:
- kind: ServiceAccount
name: funnel-sa
roleRef:
kind: Role
name: funnel-role
apiGroup: rbac.authorization.k8s.io
{{- end }}
Loading

0 comments on commit 08cfeb0

Please sign in to comment.