Deployment

To deploy Knot you need a typical Kubernetes installation - one that includes a storage controller for handling requests (claims) for persistent volumes and a network controller that assigns IP addresses to "LoadBalancer" type services.

You also need Helm, the Helm diff plugin, and Helmfile installed.

Apply the the latest Knot helmfile.yaml with:

export KNOT_HOST=example.com
helmfile -f git::https://github.com/CARV-ICS-FORTH/knot.git@helmfile.yaml sync

You can apply a specific version by adding the ref option at the end of the URL. As an example:

helmfile -f git::https://github.com/CARV-ICS-FORTH/knot.git@helmfile.yaml?ref=v4.0.0 sync

SSL certificate

By default, the installation issues a self-signed, wildcard certificate for the given KNOT_HOST. You need to make sure that at the DNS level, both the domain name and its wildcard point to your server (i.e., both example.com and *.example.com). If you already know your external IP address, you can use a nip.io name (i.e., set KNOT_HOST to <your IP address>.nip.io).

If you already have a certificate, place it in a secret in the ingress-nginx namespace with:

kubectl create namespace ingress-nginx
kubectl create secret tls -n ingress-nginx ssl-certificate --key <key file> --cert <crt file>

And then skip the self-signing process at installation by specifying --state-values-set ingress.createSelfsignedCertificate="false" to helmfile.

Storage

For storage, Knot uses two persistent volume claims: one for internal state (shared by all services) and one for user files.

To setup Knot on top of existing PVCs, use --state-values-set storage.stateVolume.existingClaim=<claim name> and --state-values-set storage.filesVolume.existingClaim=<claim name> at the helmfile command line.

Another option is to directly use local storage via --state-values-set storage.stateVolume.hostPath=<state path> and --state-values-set storage.filesVolume.hostPath=<files path> at the helmfile command line. This is useful for development, single-server setups, or compute clusters, where shared mountpoints are already setup among nodes. Note that direct access to storage may require setting up the appropriate permissions. The respective commands can be found in the Knot's Makefile.

Bare metal setup

The following script installs Kubernetes and then Knot on a fresh Ubuntu 22.04 machine.

#!/bin/bash

# Disable swap
sed -e '/swap/ s/^#*/#/' -i /etc/fstab
swapoff -a

# Install Docker (https://docs.docker.com/engine/install/ubuntu/)
apt-get update
apt-get install -y ca-certificates curl gnupg
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io
apt-mark hold docker-ce docker-ce-cli containerd.io

# Configure Docker
mkdir -p /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF
systemctl enable docker
systemctl daemon-reload
systemctl restart docker

# Install cri-dockerd (https://github.com/Mirantis/cri-dockerd)
CRI_DOCKERD_VERSION="0.3.4"
curl -LO https://github.com/Mirantis/cri-dockerd/releases/download/v${CRI_DOCKERD_VERSION}/cri-dockerd-${CRI_DOCKERD_VERSION}.$(dpkg --print-architecture).tgz
tar -zxvf cri-dockerd-${CRI_DOCKERD_VERSION}.$(dpkg --print-architecture).tgz
cp cri-dockerd/cri-dockerd /usr/local/bin/
rm -rf cri-dockerd-${CRI_DOCKERD_VERSION}.$(dpkg --print-architecture).tgz cri-dockerd
curl -Lo /etc/systemd/system/cri-docker.service https://raw.githubusercontent.com/Mirantis/cri-dockerd/v${CRI_DOCKERD_VERSION}/packaging/systemd/cri-docker.service
curl -Lo /etc/systemd/system/cri-docker.socket https://raw.githubusercontent.com/Mirantis/cri-dockerd/v${CRI_DOCKERD_VERSION}/packaging/systemd/cri-docker.socket
sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
sed -i -e '/^ExecStart=/ s/$/ --pod-infra-container-image registry.k8s.io\/pause:3.9/' /etc/systemd/system/cri-docker.service
systemctl daemon-reload
systemctl enable --now cri-docker.socket

# Install kubeadm (https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/)
KUBERNETES_VERSION="1.28"
apt-get update
apt-get install -y apt-transport-https ca-certificates curl
curl -fsSL https://pkgs.k8s.io/core:/stable:/v${KUBERNETES_VERSION}/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v${KUBERNETES_VERSION}/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

# Initialize Kubernetes (https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/)
kubeadm init --pod-network-cidr=10.244.0.0/16 --cri-socket unix:///var/run/cri-dockerd.sock
mkdir -p $HOME/.kube
cp /etc/kubernetes/admin.conf $HOME/.kube/config
kubectl taint nodes --all node-role.kubernetes.io/control-plane-

# Install a Pod network add-on
FLANNEL_VERSION="0.22.3"
kubectl apply -f https://github.com/flannel-io/flannel/releases/download/v${FLANNEL_VERSION}/kube-flannel.yml

# Download and install Helm (https://helm.sh)
HELM_VERSION="3.12.3"
curl -LO https://get.helm.sh/helm-v${HELM_VERSION}-linux-$(dpkg --print-architecture).tar.gz
tar -zxvf helm-v${HELM_VERSION}-linux-$(dpkg --print-architecture).tar.gz
cp linux-$(dpkg --print-architecture)/helm /usr/local/bin/
rm -rf helm-v${HELM_VERSION}-linux-$(dpkg --print-architecture).tar.gz linux-$(dpkg --print-architecture)
helm plugin install https://github.com/databus23/helm-diff

# Download and install Helmfile (https://github.com/roboll/helmfile)
HELMFILE_VERSION="0.144.0"
curl -LO https://github.com/roboll/helmfile/releases/download/v${HELMFILE_VERSION}/helmfile_linux_$(dpkg --print-architecture)
cp helmfile_linux_$(dpkg --print-architecture) /usr/local/bin/helmfile
chmod +x /usr/local/bin/helmfile
rm -f helmfile_linux_$(dpkg --print-architecture)

# Install Knot
IP_ADDRESS=`ip -o route get 1 | sed -n 's/.*src \([0-9.]\+\).*/\1/p'`
export KNOT_HOST=${IP_ADDRESS}.nip.io
mkdir -p /mnt/state /mnt/state/jupyterhub /mnt/state/harbor/{database,redis,registry,chartmuseum,jobservice}
chown 1000:1000 /mnt/state/jupyterhub
chown 999:999 /mnt/state/harbor/{database,redis}
chown 10000:10000 /mnt/state/harbor/{registry,chartmuseum,jobservice}
mkdir -p /mnt/files
helmfile -f git::https://github.com/CARV-ICS-FORTH/knot.git@helmfile.yaml \
  --state-values-set ingress.service.type=NodePort \
  --state-values-set ingress.service.externalIPs\[0\]=${IP_ADDRESS} \
  --state-values-set storage.stateVolume.hostPath=/mnt/state \
  --state-values-set storage.filesVolume.hostPath=/mnt/files \
  sync