How to Build A Kubernetes Operator

This is the second part of our series focusing on Kubernetes Operators, and it shows how you can build a Kubernetes Operator based on the Bitnami Apache Helm chart. Note that you can refer to the steps in this tutorial to build an operator for your own applications.

Prerequisites

1. We assume you followed the first part of the series. Thus, you should have a Kubernetes cluster (v1.7 or newer) with a control plane and two workers running on your computer. Also, the Operator Lifecycle Manager should be installed on your system. You can enter the following command to verify that everything is set up:

kubectl get all --namespace olm

NAME                                    READY   STATUS    RESTARTS   AGE
pod/catalog-operator-64b6b59c4f-brck9   1/1     Running   0          3m28s
pod/olm-operator-844fb69f58-fn57f       1/1     Running   0          3m28s
pod/operatorhubio-catalog-5s8k2         1/1     Running   0          3m4s
pod/packageserver-65df5d5cc9-nz26h      1/1     Running   0          3m2s
pod/packageserver-65df5d5cc9-x7hwc      1/1     Running   0          3m2s

NAME                                       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
service/operatorhubio-catalog              ClusterIP   10.103.1.120   <none>        50051/TCP   3m3s
service/v1-packages-operators-coreos-com   ClusterIP   10.99.75.171   <none>        443/TCP     3m3s

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/catalog-operator   1/1     1            1           3m28s
deployment.apps/olm-operator       1/1     1            1           3m28s
deployment.apps/packageserver      2/2     2            2           3m2s

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/catalog-operator-64b6b59c4f   1         1         1       3m28s
replicaset.apps/olm-operator-844fb69f58       1         1         1       3m28s
replicaset.apps/packageserver-65df5d5cc9      2         2         2       3m2s

2. The Operator SDK is installed on your machine. For details about installing the Operator SDK, refer to the Install the Operator SDK CLI page.
3. Helm CLI is installed on your computer. To install Helm CLI, follow the instructions from the Installing Helm page.
4. Docker. For details about installing Docker, refer to the Install Docker page.
5. You need a quay.io account.

Set Up The Apache Operator

In this section, we’ll walk you through the process of setting up the Apache Operator using Bitnami’s Helm chart.

1. Add the Bitnami Helm repository to your Helm client by running the following command:

helm repo add bitnami https://charts.bitnami.com

"bitnami" has been added to your repositories

2. Create a Helm-based Operator by running the operator-sdk new command, and passing it the following arguments:

• The name of your operator (apache-operator).
• The --api-version flag with the Kubernetes apiVersion. The format is $GROUP_NAME/$VERSION. In this tutorial, we’ll be using appfleet.com/v1alpha1.
• The --kind flag with the name of the Kubernetes CRD (Apache)
• The --type flag with the type of operator. We’ll be using helm. Other valid types are Go and Ansible
• The helm-chart flag with the name of the Helm chart (bitnami/apache)

operator-sdk new apache-operator --api-version=appfleet.com/v1alpha1 --kind=Apache --type=helm --helm-chart=bitnami/apache

INFO[0000] Creating new Helm operator 'apache-operator'.
INFO[0001] Created helm-charts/apache
INFO[0001] Generating RBAC rules
WARN[0001] The RBAC rules generated in deploy/role.yaml are based on the chart's default manifest. Some rules may be missing for resources that are only enabled with custom values, and some existing rules may be overly broad. Double check the rules generated in deploy/role.yaml to ensure they meet the operator's permission requirements.
INFO[0001] Created build/Dockerfile
INFO[0001] Created watches.yaml
INFO[0001] Created deploy/service_account.yaml
INFO[0001] Created deploy/role.yaml
INFO[0001] Created deploy/role_binding.yaml
INFO[0001] Created deploy/operator.yaml
INFO[0001] Created deploy/crds/appfleet.com_v1alpha1_apache_cr.yaml
INFO[0001] Generated CustomResourceDefinition manifests.
INFO[0001] Project creation complete.

This command creates the following directory structure:

tree apache-operator -L 2

apache-operator
├── build
│   └── Dockerfile
├── deploy
│   ├── crds
│   ├── operator.yaml
│   ├── role.yaml
│   ├── role_binding.yaml
│   └── service_account.yaml
├── helm-charts
│   └── apache
└── watches.yaml

Things to note:

• Kubernetes compares the actual state of the cluster with the desired state. Then, it takes action to match these states.  The Operators extend this pattern by watching a specific custom resource type and taking actions to match the spec in that resource. In this example, the Operator watches the Apache resource as defined in the watches.yaml file:

cat apache-operator/watches.yaml

---
- version: v1alpha1
  group: appfleet.com
  kind: Apache
  chart: helm-charts/apache

• The Dockerfile uses the quay.io/operator-framework/helm-operator:v0.15.1 image as the base, and then it copies watches.yaml file and the Helm charts:

cat apache-operator/build/Dockerfile

FROM quay.io/operator-framework/helm-operator:v0.15.1

COPY watches.yaml ${HOME}/watches.yaml
COPY helm-charts/ ${HOME}/helm-charts/

3. Now you can build the Apache Operator by moving into the apache-operator directory, and then entering the following operator-sdk build command:

operator-sdk build apache-operator:v0.1

INFO[0000] Building OCI image apache-operator:v0.1
Sending build context to Docker daemon  64.51kB
Step 1/3 : FROM quay.io/operator-framework/helm-operator:v0.15.1
 ---> 450a3ca2d02d
Step 2/3 : COPY watches.yaml ${HOME}/watches.yaml
 ---> Using cache
 ---> db5c285c02fb
Step 3/3 : COPY helm-charts/ ${HOME}/helm-charts/
 ---> 50255ede17de
Successfully built 50255ede17de
Successfully tagged apache-operator:v0.1
INFO[0003] Operator build complete.

4. Verify that the Docker image was created with:

docker images | grep apache

apache-operator                            v0.1                50255ede17de        38 seconds ago      174MB

Set Up quay.io with Docker

You can think of Quay as something similar to GitHub, but for Docker images. It’s a registry where you can host images and share them. There are a couple of ways to set up quay.io with Docker. For the sake of simplicity, you’ll use the docker login command. It’s important to note that the docker login command stores the password you enter as plain-text. Thus, you should first generate an encrypted password.

1. Point your browser to https://quay.io/, and then navigate to Account Settings:

2. From the Account Settings page select Generate Encrypted Password:

3. You will be prompted to enter your quay.io password:

4. Select Docker Login and then copy the command containing your Docker encrypted password:

5. In a terminal window, log in to quay.io by entering the following command:

docker login -u="<YOUR_USERNAME>" -p="<YOUR_ENCRYPTED_PASSWORD>" quay.io

WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded

Push the Apache Operator Image to quay.io

To do this, you first need to properly tag the image with the hostname of the registry, and the name of your repository.  Then, you can push the image.

1. Enter the following command to tag the local image named apache-operator into the quay.io/andreipope repository:

docker tag apache-operator:v0.1 quay.io/andreipope/apache-operator:v0.1

Note that the image name is comprised by a slash-separated list of the:

• Registry hostname (quay.io)
• Repository (andreipope)
• Operator name (apache-version).

☞ The name of our repository is andreipope, but yours will be different.

2. To push the image that we created in the previous section, run the following command docker push command:

docker push quay.io/andreipope/apache-operator:v0.1

b8325e5fabd7: Pushed
2f4354fc6a73: Pushed
b496b494f6f9: Pushed
9fd48ecc1227: Pushed
0141daa77f22: Pushed
27cd2023d60a: Pushed
4b52dfd1f9d9: Pushed
v0.1: digest: sha256:7230926984fc3d688e02764748441a74907eeb772e3b51eb06b1bac225ba9f98 size: 1778

3. Point your browser to https://quay.io/, navigate to the apache-operator repository, and make repository public:

Deploy the Apache Operator

You are now ready to deploy the Apache Operator. Before that, you must customize the specs.

1. Open the deploy/operator.yaml file in a plain-text editor and update the placeholder image: REPLACE_IMAGE with the location of your image (quay.io/andreipope/apache-operator:v0.1)

Your deploy.operator.yaml file should look similar to the following:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apache-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: apache-operator
  template:
    metadata:
      labels:
        name: apache-operator
    spec:
      serviceAccountName: apache-operator
      containers:
        - name: apache-operator
          # Replace this with the built image name
          image: quay.io/andreipope/apache-operator:v0.1
          imagePullPolicy: Always
          env:
            - name: WATCH_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: "apache-operator"

2. Enter these kubectl create commands to deploy the Apache Operator:

kubectl create -f deploy/service_account.yaml
kubectl create -f deploy/role.yaml
kubectl create -f deploy/role_binding.yaml
kubectl create -f deploy/operator.yaml

serviceaccount/expressjs-operator created
role.rbac.authorization.k8s.io/expressjs-operator created
rolebinding.rbac.authorization.k8s.io/expressjs-operator created
deployment.apps/expressjs-operator created

3. Check the status of the deployment:

kubectl get deployment
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
apache-operator   1/1     1            1           24s

☞ Note that the deployment doesn’t define the spec for the Apache cluster. You’ll describe the Apache cluster in the next section, once the Operator is running.

4. The Operator is a pod running in this deployment. To see it, type the following command:

kubectl get pods

kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
apache-operator-6d5795f879-np6pr   1/1     Running   1          38s

Deploy the Apache Cluster

In the Set Up the Apache Operator section, you created a CRD  defining a new kind of resource, an Apache cluster. Now, you will apply that spec so that the Operator starts watching the Apache resources. Then, you will deploy the Apache cluster itself.

1. First, let’s deploy the CRD that defines the resources the Operator will monitor:

kubectl apply -f deploy/crds/appfleet.com_apaches_crd.yaml
customresourcedefinition.apiextensions.k8s.io/apaches.appfleet.com created

customresourcedefinition.apiextensions.k8s.io/apaches.appfleet.com created

2. At this point, you are ready to deploy the Apache cluster:

kubectl apply -f deploy/crds/appfleet.com_v1alpha1_apache_cr.yaml

apache.appfleet.com/example-apache created

3. The deployment takes a bit of time to complete. Once everything is ready, you should see a new pod running Apache:

kubectl get pods

NAME                               READY   STATUS    RESTARTS   AGE
apache-operator-6d5795f879-np6pr   1/1     Running   4          2m53s
example-apache-7cf789fc98-462dr    1/1     Running   0          39s

4. You can also check that the deployment was created by entering the following command:

kubectl get deployment

NAME              READY   UP-TO-DATE   AVAILABLE   AGE
apache-operator   1/1     1            1           5m43s
example-apache    2/2     2            2           3m29s

Scaling Up

At this point, you have a running Apache cluster. To add another instance, you must modify the replicaCount field  in the deploy/crds/appfleet.com_v1alpha1_apache_cr.yamlapache.appfleet.com/example-apache  file. Then, you need to apply the new spec.

1. Open the deploy/crds/appfleet.com_v1alpha1_apache_cr.yamlapache.appfleet.com/example-apache file in a plain-text editor, and specify spec.replicaCount: 2.
   The updated file should look similar to the following:

apiVersion: appfleet.com/v1alpha1
kind: Apache
metadata:
  name: example-apache
spec:
  # Default values copied from <project_dir>/helm-charts/apache/values.yaml

  affinity: {}
  cloneHtdocsFromGit:
    enabled: false
    interval: 60
  git:
    pullPolicy: IfNotPresent
    registry: docker.io
    repository: bitnami/git
    tag: 2.25.0-debian-10-r0
  image:
    debug: false
    pullPolicy: IfNotPresent
    registry: docker.io
    repository: bitnami/apache
    tag: 2.4.41-debian-10-r0
  ingress:
    annotations: {}
    certManager: false
    enabled: false
    hostname: example.local
    secrets: null
    tls:
    - hosts:
      - example.local
      secretName: example.local-tls
  livenessProbe:
    enabled: true
    failureThreshold: 6
    initialDelaySeconds: 180
    path: /
    periodSeconds: 20
    port: http
    successThreshold: 1
    timeoutSeconds: 5
  metrics:
    enabled: false
    image:
      pullPolicy: IfNotPresent
      registry: docker.io
      repository: bitnami/apache-exporter
      tag: 0.7.0-debian-10-r0
    podAnnotations:
      prometheus.io/port: "9117"
      prometheus.io/scrape: "true"
    resources:
      limits: {}
      requests: {}
  nodeSelector: {}
  podAnnotations: {}
  readinessProbe:
    enabled: true
    failureThreshold: 6
    initialDelaySeconds: 30
    path: /
    periodSeconds: 10
    port: http
    successThreshold: 1
    timeoutSeconds: 5
  replicaCount: 2
  resources:
    limits: {}
    requests: {}
  service:
    annotations: {}
    externalTrafficPolicy: Cluster
    httpsPort: 443
    nodePorts:
      http: ""
      https: ""
    port: 80
    type: LoadBalancer
  tolerations: {}

2. You can apply the updated spec with:

kubectl apply -f deploy/crds/appfleet.com_v1alpha1_apache_cr.yaml

apache.appfleet.com/example-apache configured

3. After applying the updated spec, the state of the cluster differs from the desired state. The Operator starts a new instance of the Apache webserver to reconcile the two, scaling up the cluster:

kubectl get pods

NAME                               READY   STATUS              RESTARTS   AGE
apache-operator-6d5795f879-np6pr   1/1     Running             4          4m10s
example-apache-7cf789fc98-462dr    1/1     Running             0          116s
example-apache-7cf789fc98-fvttm    0/1     ContainerCreating   0          1s

Wait a bit until the second container is created:

kubectl get pods

NAME                               READY   STATUS    RESTARTS   AGE
apache-operator-6d5795f879-np6pr   1/1     Running   0          5m2s
example-apache-7cf789fc98-462dr    1/1     Running   0          2m48s
example-apache-7cf789fc98-fvttm    1/1     Running   0          53s

In the above output, note that two example-apache pods are running.

Verify Your Installation

The Apache Operator creates a Kubernetes service which is basically an endpoint where clients can obtain access to the group of pods running Apache. You will use this service to access your cluster using a web browser.

1. Make sure that the service is up with:

kubectl get service

NAME                      TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
apache-operator-metrics   ClusterIP      10.104.41.83   <none>        8686/TCP,8383/TCP            5m46s
example-apache            LoadBalancer   10.99.54.115   <pending>     80:31058/TCP,443:30362/TCP   3m51s
kubernetes                ClusterIP      10.96.0.1      <none>        443/TCP

2. Forward the connections made to http://localhost:80 to the pod running the service/example-apache service:

kubectl port-forward service/example-apache 80:80

Forwarding from 127.0.0.1:80 -> 8080
Forwarding from [::1]:80 -> 8080

3. Point your browser to http://localhost:80. If everything worked well, you should see something like the following:

Now you should have a good understanding of how Kubernetes Operators work. Furthermore, by completing this tutorial you learned how create an operator for your application using a Helm chart.

Thanks for reading!