In the last part of this blog, we discussed why innovative technology is very important for organizations. We demonstrated how to start an innovation project in a real-world scenario with immudb. Now we have to get on the next level and find a proper production environment for immudb.
The lab is a great place to present and test technology but it is isolated and doesn’t fulfill production requirements like scalability, high availability, or a backup concept.
How do we find that place in a large company? There are many platforms to choose from. Innovative projects are always very risky, that’s why you should look out for positive cross effects. Which platform is strategic for the company or is becoming big in the future? Which skills are valuable for your position? Doing something new is always a great opportunity for upskilling. Just do it in the right direction.
Kubernetes is a good example of a fast-growing platform that will sooner or later be used by most organizations.
In this blog, we will deploy immudb as a StatefulSet in Kubernetes with replication. For that, we start with a minikube tutorial. Of course, we will also go into the real-world challenges of applying for the same work in a large organization. The real-world examples are from an undisclosed IT service provider hosting over 2000 servers and 16000 virtual machines.
In this tutorial shows how to run the immudb database as a replicated stateful application using a StatefulSet controller. Before that, we need a Kubernetes cluster and the kubectl command-line tool. This tutorial is using minikube to create and run the cluster. The example immudb deployment consists of YAML files that are creating a ConfigMap, Services and the StatefulSet itself:
Note: this tutorial is focusing on running immudb as a stateful application in Kubernetes. It is not a production configuration as Kubernetes policies are not part of the tutorial and immudb settings remain defaults. Also, you have to be familiar with the concepts of StatefulSets, ConfigMaps, Services , Volumes, StorageClasses, and PersistentVolumes/Claims.
The ConfigMap is being used for providing configuration data to the application containers. With the ConfigMap, we can use customized configurations for the immudb database servers.
apiVersion: v1
kind: ConfigMap
metadata:
name: immudb
labels:
app: immudb
data:
primary.cnf: |
dir = "./data/master"
network = "tcp"
address = "0.0.0.0"
port = 3322
dbname = "immudb"
pgsql-server = true # enable or disable pgsql server
pgsql-server-port = 5432
web-server-port = 8080
replica.cnf: |
replication-enabled = true
replication-master-address="immudb-0.immudb"
replication-master-port=3322
replication-follower-username="immudb"
replication-follower-password="immudb"
port=3334
pgsql-server-port=5434
dir="./data/replica"
pgsql-server = true # enable or disable pgsql server
web-server-port = 9090The ConfigMap with the name “immudb” provides individual configuration files for the master and replica servers. The primary.cnf is almost equivalent to the standard config of immudb. The working directory for the databases has been changed to “./data/master”. That directory will later be mounted in the StatefulSet. The replica.cnf config however is different. immudb will start up as a replica server when using the replica.cnf. It will be read-only and replicate databases in the master database server (at the moment systemdb and defaultdb). That’s why the ports are different. It also contains the replication parameters. The replication-master-address is a static DNS name for the immudb master server. That means it won’t change even if the pod is restarting. Apply, show, and delete the ConfigMap like this:
# Apply the ConfigMap
kubectl apply -f configmap.yaml
# Show the content of the ConfigMap immudb
kubectl describe configmap immudb
# Clean-up
kubectl delete configmap immudbServices in Kubernetes are a way to expose an app running on a set of Pods.
# Headless service for stable DNS entries of StatefulSet members.
apiVersion: v1
kind: Service
metadata:
name: immudb
labels:
app: immudb
spec:
ports:
- name: immudb
port: 3322
clusterIP: None
selector:
app: immudb
---
# Client service for connecting to any immudb instance for reads.
# For writes, you must instead connect to the primary: immudb-0.immudb.
apiVersion: v1
kind: Service
metadata:
name: immudb-read
labels:
app: immudb
spec:
ports:
- name: immudb
port: 3334
selector:
app: immudb
---
# immudb master webserver (read/write)
apiVersion: v1
kind: Service
metadata:
name: immudb-webserver
labels:
app: immudb
spec:
ports:
- name: immudb
port: 8080
selector:
statefulset.kubernetes.io/pod-name: immudb-0
---
# immudb replica webserver (read-only)
apiVersion: v1
kind: Service
metadata:
labels:
app: immudb
name: immudb-ext
spec:
ports:
- port: 9090
selector:
statefulset.kubernetes.io/pod-name: immudb-2bThe Headless Service “immudb” is beeing defined with the attribute “clusterIP: None”. That means it uses the DNS entries from the StatefulSet controller. Pods are accessible by their pod-name.headless-service-name so for our immudb master pod that would be “immudb-0.immudb”. The Service “immudb-read” is a Client Serivce. It is a standard Service with its own cluster IP and is connecting to all immudb Pods including the master. The Service “immudb-webserver” is connecting to the immudb master webserver “immudb-0” and providing read/write capabilities. The Service “immudb-ext” is conneting to the webinterface of a replica server. These Services have been assign to later test our StatefulSet App comfortably from webbrowser.
# Apply the ConfigMap
kubectl apply -f service.yaml
# Show Services
kubectl get services
# Clean-up
kubectl delete service <service-name>A StatefulSet manages the deployment and scaling of a set of Pods, and provides guarantees about the ordering and uniqueness of these Pods.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: immudb
spec:
selector:
matchLabels:
app: immudb
serviceName: immudb
replicas: 3
template:
metadata:
labels:
app: immudb
spec:
initContainers:
- name: init-immudb
image: codenotary/immudb:latest-bullseye-slim
command:
- bash
- "-c"
- |
set -ex
# Generate immudb server-id from pod ordinal index.
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1
ordinal=${BASH_REMATCH[1]}
if [[ $ordinal -eq 0 ]]; then
cp /mnt/config-map/primary.cnf /mnt/configs/immudb.toml
else
cp /mnt/config-map/replica.cnf /mnt/configs/immudb.toml
fi
volumeMounts:
- name: conf
mountPath: /mnt/configs
subPath: immudb.toml
- name: config-map
mountPath: /mnt/config-map
containers:
- name: immudb
image: codenotary/immudb:latest-bullseye-slim
command: ["immudb","--config","/mnt/configs/immudb.toml"]
env:
- name: IMMU_ADMIN_PASSWORD
value: "secret"
ports:
- name: immudbgprc
containerPort: 3322
- name: immudbweb
containerPort: 8080
- name: immudbrepbweb
containerPort: 9090
volumeMounts:
- name: data
mountPath: /data/master
- name: data
mountPath: /data/replica
- name: conf
mountPath: /mnt/configs
subPath: immudb.toml
volumes:
- name: conf
emptyDir: {}
- name: scripts
emptyDir: {}
- name: config-map
configMap:
name: immudb
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 2GiThe “initContainer” is copying the configs of the ConfigMap based on the Hostname into the mounted volumes for the Container. The Hostname of the immudb master database server is “immudb-0” and the replicas are called “immudb-1”, “immudb-2”, and so on. The “primary.cnf” is only copied into “immudb.toml” when the pod name ends with a zero, every pod name greater than that will get the replica config. The image from DockerHub for the containers is the “codenotary/immudb:latest-bullseye-slim” image with bash and hostname commands available. The immudb is then being started by using the command “immudb –config=”/mnt/configs/immudb.toml” which translates into “command: [“immudb”,”–config”,”/mnt/configs/immudb.toml”]” in the immudb-statefulset.yaml file.
# Apply the statefulset
kubectl apply -f immudb-statefulset.yaml
# Clean-up
kubectl delete statefulset immudbThere are several ways to check if your application is running. Look up the status of the pods:
kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
immudb-0 1/1 Running 0 40m 172.17.0.5 minikube <none> <none>
immudb-1 1/1 Running 0 40m 172.17.0.6 minikube <none> <none>
immudb-2 1/1 Running 0 40m 172.17.0.7 minikube <none> <none> Get the logs of a pod:
# Logs of pod-name immudb-1 in container (-c) immudb
kubectl logs immudb-1 -c immudb
immudb 2021/09/25 16:46:20 INFO: Replication from 'systemdb@immudb-0.immudb:3322' to 'systemdb' succesfully initialized
immudb 2021/09/25 16:46:20 INFO: Creating database 'defaultdb' {replica = true}...
immudb 2021/09/25 16:46:20 INFO: Connecting to 'immudb-0.immudb':'3322' for database 'systemdb'...Jump into the pod’s commandline to figure out problems like “no file or directory”:
kubectl exec --stdin --tty immudb-o -c immudb -- bashThe most comfortable way is to start up the Kubernetes Dashboard. This is how it works with Minikube:
minikube dashboardThe dashboard is showing an overview about your apps and helps troubleshooting aswell as managing the cluster.
The Apps can be accessed by running a minikube tunnel. Through the tunnel it is possible to connect to the defined Services
minikube tunnel
Status:
machine: minikube
pid: 66484
route: 10.96.0.0/12 -> 192.168.99.101
minikube: Running
services: [immudb-ext]
errors:
minikube: no errorsThe immudb webinterface is now available the ip and port displayed in the Services information.
kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
immudb ClusterIP None <none> 3322/TCP 3d14h
immudb-ext LoadBalancer 10.104.160.179 10.104.160.179 9090:30184/TCP 57m
immudb-read ClusterIP 10.107.246.97 <none> 3322/TCP 3d14h
immudb-webserver ClusterIP 10.108.213.230 <none> 8080/TCP 57m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3d17hThat means the immudb master webserver is found on 10.108.213.230:8080 and the replica webservers use 10.104.160.179:9090. Each server is individually taking care of their users. Logon to test the replication feature with the default immudb credentials (user: immudb, password immudb). Add demo data to the default database in the master webserver. The data should now be available in the replica too. Refresh the replica and query the database table.
It is one thing to set something up on your own computer with minikube with all the authorizations and without any policies. Unfortunately, when working in a managed environment, most challenges are quite different from what you would expect. First of all many colleagues don’t like to share privileges or a whole lot of knowledge. You are basically given access to a black box. In this case a Kubernetes cluster. The internal documentation is often incomplete, outdated, or even straight-up misleading. Questions that would take 30 seconds to be answered are being declined by saying that there is no time and they will look at it if a ticket or issue is being created. The main hiccups are due to customized policies and custom storage configurations. The lesson is that there is no way around understanding the stack yourself and that you have to look for people with the same goals as you. You can find those in the immudb community. Join the wonderful immudb community!
In the next blog, we will integrate immudb into an enterprise process and enhance our Kubernetes Application. We will also talk about how to present an innovative project because work can only be valued if it’s visible. It is the perceived performance that is most important.