Pushing new technology the hard way (Part2)

In the last part of this blog, we discussed why innovative technology is very important for organizations. We demonstrated how to start an innovation project in a real-world scenario with immudb. Now we have to get on the next level and find a proper production environment for immudb.

The lab is a great place to present and test technology but it is isolated and doesn’t fulfill production requirements like scalability, high availability, or a backup concept.

How do we find that place in a large company? There are many platforms to choose from. Innovative projects are always very risky, that’s why you should look out for positive cross effects. Which platform is strategic for the company or is becoming big in the future? Which skills are valuable for your position? Doing something new is always a great opportunity for upskilling. Just do it in the right direction.

Kubernetes is a good example of a fast-growing platform that will sooner or later be used by most organizations.

In this blog, we will deploy immudb as a StatefulSet in Kubernetes with replication. For that, we start with a minikube tutorial. Of course, we will also go into the real-world challenges of applying for the same work in a large organization. The real-world examples are from an undisclosed IT service provider hosting over 2000 servers and 16000 virtual machines.

Run immudb as Replicated Stateful Application

In this tutorial shows how to run the immudb database as a replicated stateful application using a StatefulSet controller. Before that, we need a Kubernetes cluster and the kubectl command-line tool. This tutorial is using minikube to create and run the cluster. The example immudb deployment consists of YAML files that are creating a ConfigMap, Services and the StatefulSet itself:

• configmap.yaml
• service.yaml
• immudb-statefulset.yaml

Note: this tutorial is focusing on running immudb as a stateful application in Kubernetes. It is not a production configuration as Kubernetes policies are not part of the tutorial and immudb settings remain defaults. Also, you have to be familiar with the concepts of StatefulSets, ConfigMaps, Services , Volumes, StorageClasses, and PersistentVolumes/Claims.

ConfigMap

The ConfigMap is being used for providing configuration data to the application containers. With the ConfigMap, we can use customized configurations for the immudb database servers.

apiVersion: v1
kind: ConfigMap
metadata:
  name: immudb
  labels:
    app: immudb
data:
  primary.cnf: |
    dir = "./data/master"
    network = "tcp"
    address = "0.0.0.0"
    port = 3322
    dbname = "immudb"
    pgsql-server = true # enable or disable pgsql server
    pgsql-server-port = 5432  
    web-server-port = 8080   
  replica.cnf: |
    replication-enabled = true
    replication-master-address="immudb-0.immudb"
    replication-master-port=3322 
    replication-follower-username="immudb"
    replication-follower-password="immudb"
    port=3334
    pgsql-server-port=5434
    dir="./data/replica"
    pgsql-server = true # enable or disable pgsql server
    web-server-port = 9090

The ConfigMap with the name “immudb” provides individual configuration files for the master and replica servers. The primary.cnf is almost equivalent to the standard config of immudb. The working directory for the databases has been changed to “./data/master”. That directory will later be mounted in the StatefulSet. The replica.cnf config however is different. immudb will start up as a replica server when using the replica.cnf. It will be read-only and replicate databases in the master database server (at the moment systemdb and defaultdb). That’s why the ports are different. It also contains the replication parameters. The replication-master-address is a static DNS name for the immudb master server. That means it won’t change even if the pod is restarting. Apply, show, and delete the ConfigMap like this:

    # Apply the ConfigMap
    kubectl apply -f configmap.yaml
    # Show the content of the ConfigMap immudb
    kubectl describe configmap immudb
    # Clean-up
    kubectl delete configmap immudb

Services

Services in Kubernetes are a way to expose an app running on a set of Pods.

# Headless service for stable DNS entries of StatefulSet members.
apiVersion: v1
kind: Service
metadata:
  name: immudb
  labels:
    app: immudb
spec:
  ports:
  - name: immudb
    port: 3322
  clusterIP: None
  selector:
    app: immudb
---
# Client service for connecting to any immudb instance for reads.
# For writes, you must instead connect to the primary: immudb-0.immudb.
apiVersion: v1
kind: Service
metadata:
  name: immudb-read
  labels:
    app: immudb
spec:
  ports:
  - name: immudb
    port: 3334
  selector:
    app: immudb
---
# immudb master webserver (read/write)
apiVersion: v1
kind: Service
metadata:
  name: immudb-webserver
  labels:
    app: immudb
spec:
  ports:
  - name: immudb
    port: 8080
  selector:
    statefulset.kubernetes.io/pod-name: immudb-0
---
# immudb replica webserver (read-only)
apiVersion: v1
kind: Service
metadata:
  labels:
    app: immudb
  name: immudb-ext
spec:
  ports:
    - port: 9090
  selector:
    statefulset.kubernetes.io/pod-name: immudb-2b

The Headless Service “immudb” is beeing defined with the attribute “clusterIP: None”. That means it uses the DNS entries from the StatefulSet controller. Pods are accessible by their pod-name.headless-service-name so for our immudb master pod that would be “immudb-0.immudb”. The Service “immudb-read” is a Client Serivce. It is a standard Service with its own cluster IP and is connecting to all immudb Pods including the master. The Service “immudb-webserver” is connecting to the immudb master webserver “immudb-0” and providing read/write capabilities. The Service “immudb-ext” is conneting to the webinterface of a replica server. These Services have been assign to later test our StatefulSet App comfortably from webbrowser.

# Apply the ConfigMap
kubectl apply -f service.yaml
# Show Services 
kubectl get services 
# Clean-up 
kubectl delete service <service-name>

StatefulSet

A StatefulSet manages the deployment and scaling of a set of Pods, and provides guarantees about the ordering and uniqueness of these Pods.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: immudb
spec:
  selector:
    matchLabels:
      app: immudb
  serviceName: immudb
  replicas: 3
  template:
    metadata:
      labels:
        app: immudb
    spec:    
      initContainers:
      - name: init-immudb
        image: codenotary/immudb:latest-bullseye-slim
        command:
        - bash
        - "-c"
        - |
          set -ex
          # Generate immudb server-id from pod ordinal index.
          [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
          ordinal=${BASH_REMATCH[1]}
          if [[ $ordinal -eq 0 ]]; then
            cp /mnt/config-map/primary.cnf /mnt/configs/immudb.toml
          else
            cp /mnt/config-map/replica.cnf /mnt/configs/immudb.toml
          fi          
        volumeMounts:
        - name: conf
          mountPath: /mnt/configs
          subPath: immudb.toml
        - name: config-map
          mountPath: /mnt/config-map
      containers:
      - name: immudb
        image: codenotary/immudb:latest-bullseye-slim
        command: ["immudb","--config","/mnt/configs/immudb.toml"]
        env:
        - name: IMMU_ADMIN_PASSWORD
          value: "secret"
        ports:
        - name: immudbgprc
          containerPort: 3322
        - name: immudbweb
          containerPort: 8080
        - name: immudbrepbweb
          containerPort: 9090
        volumeMounts:
        - name: data
          mountPath: /data/master
        - name: data
          mountPath: /data/replica
        - name: conf
          mountPath: /mnt/configs
          subPath: immudb.toml
      volumes:
      - name: conf
        emptyDir: {} 
      - name: scripts
        emptyDir: {} 
      - name: config-map
        configMap:
          name: immudb
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 2Gi

The “initContainer” is copying the configs of the ConfigMap based on the Hostname into the mounted volumes for the Container. The Hostname of the immudb master database server is “immudb-0” and the replicas are called “immudb-1”, “immudb-2”, and so on. The “primary.cnf” is only copied into “immudb.toml” when the pod name ends with a zero, every pod name greater than that will get the replica config. The image from DockerHub for the containers is the “codenotary/immudb:latest-bullseye-slim” image with bash and hostname commands available. The immudb is then being started by using the command “immudb –config=”/mnt/configs/immudb.toml” which translates into “command: [“immudb”,”–config”,”/mnt/configs/immudb.toml”]” in the immudb-statefulset.yaml file.

# Apply the statefulset
kubectl apply -f immudb-statefulset.yaml
# Clean-up
kubectl delete statefulset immudb

Did it work?

There are several ways to check if your application is running. Look up the status of the pods:

kubectl get pods -o wide

NAME       READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
immudb-0   1/1     Running   0          40m   172.17.0.5   minikube   <none>           <none>
immudb-1   1/1     Running   0          40m   172.17.0.6   minikube   <none>           <none>
immudb-2   1/1     Running   0          40m   172.17.0.7   minikube   <none>           <none>    

Get the logs of a pod:

# Logs of pod-name immudb-1 in container (-c) immudb 
kubectl logs immudb-1 -c immudb
immudb  2021/09/25 16:46:20 INFO: Replication from 'systemdb@immudb-0.immudb:3322' to 'systemdb' succesfully initialized
immudb  2021/09/25 16:46:20 INFO: Creating database 'defaultdb' {replica = true}...
immudb  2021/09/25 16:46:20 INFO: Connecting to 'immudb-0.immudb':'3322' for database 'systemdb'...

Jump into the pod’s commandline to figure out problems like “no file or directory”:

kubectl exec --stdin --tty immudb-o -c immudb -- bash

The most comfortable way is to start up the Kubernetes Dashboard. This is how it works with Minikube:

minikube dashboard

The dashboard is showing an overview about your apps and helps troubleshooting aswell as managing the cluster.

Expose the Services to the LocalHost

The Apps can be accessed by running a minikube tunnel. Through the tunnel it is possible to connect to the defined Services

minikube tunnel
Status:
    machine: minikube
    pid: 66484
    route: 10.96.0.0/12 -> 192.168.99.101
    minikube: Running
    services: [immudb-ext]
    errors:
        minikube: no errors

The immudb webinterface is now available the ip and port displayed in the Services information.

kubectl get services
NAME               TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)          AGE
immudb             ClusterIP      None             <none>           3322/TCP         3d14h
immudb-ext         LoadBalancer   10.104.160.179   10.104.160.179   9090:30184/TCP   57m
immudb-read        ClusterIP      10.107.246.97    <none>           3322/TCP         3d14h
immudb-webserver   ClusterIP      10.108.213.230   <none>           8080/TCP         57m
kubernetes         ClusterIP      10.96.0.1        <none>           443/TCP          3d17h

That means the immudb master webserver is found on 10.108.213.230:8080 and the replica webservers use 10.104.160.179:9090. Each server is individually taking care of their users. Logon to test the replication feature with the default immudb credentials (user: immudb, password immudb). Add demo data to the default database in the master webserver. The data should now be available in the replica too. Refresh the replica and query the database table.

Real-world challenges when applying the work in a large organization

It is one thing to set something up on your own computer with minikube with all the authorizations and without any policies. Unfortunately, when working in a managed environment, most challenges are quite different from what you would expect. First of all many colleagues don’t like to share privileges or a whole lot of knowledge. You are basically given access to a black box. In this case a Kubernetes cluster. The internal documentation is often incomplete, outdated, or even straight-up misleading. Questions that would take 30 seconds to be answered are being declined by saying that there is no time and they will look at it if a ticket or issue is being created. The main hiccups are due to customized policies and custom storage configurations. The lesson is that there is no way around understanding the stack yourself and that you have to look for people with the same goals as you. You can find those in the immudb community. Join the wonderful immudb community!

Coming up next

In the next blog, we will integrate immudb into an enterprise process and enhance our Kubernetes Application. We will also talk about how to present an innovative project because work can only be valued if it’s visible. It is the perceived performance that is most important.