How to Build A Kubernetes Operator

This is the second part of our series focusing on Kubernetes Operators, and it shows how you can build a Kubernetes Operator based on the Bitnami Apache Helm chart. Note that you can refer to the steps in this tutorial to build an operator for your own applications.

Prerequisites

1. We assume you followed the first part of the series. Thus, you should have a Kubernetes cluster (v1.7 or newer) with a control plane and two workers running on your computer. Also, the Operator Lifecycle Manager should be installed on your system. You can enter the following command to verify that everything is set up:

kubectl get all --namespace olm

NAME                                    READY   STATUS    RESTARTS   AGEpod/catalog-operator-64b6b59c4f-brck9   1/1     Running   0          3m28spod/olm-operator-844fb69f58-fn57f       1/1     Running   0          3m28spod/operatorhubio-catalog-5s8k2         1/1     Running   0          3m4spod/packageserver-65df5d5cc9-nz26h      1/1     Running   0          3m2spod/packageserver-65df5d5cc9-x7hwc      1/1     Running   0          3m2sNAME                                       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGEservice/operatorhubio-catalog              ClusterIP   10.103.1.120   <none>        50051/TCP   3m3sservice/v1-packages-operators-coreos-com   ClusterIP   10.99.75.171   <none>        443/TCP     3m3sNAME                               READY   UP-TO-DATE   AVAILABLE   AGEdeployment.apps/catalog-operator   1/1     1            1           3m28sdeployment.apps/olm-operator       1/1     1            1           3m28sdeployment.apps/packageserver      2/2     2            2           3m2sNAME                                          DESIRED   CURRENT   READY   AGEreplicaset.apps/catalog-operator-64b6b59c4f   1         1         1       3m28sreplicaset.apps/olm-operator-844fb69f58       1         1         1       3m28sreplicaset.apps/packageserver-65df5d5cc9      2         2         2       3m2s

2. The Operator SDK is installed on your machine. For details about installing the Operator SDK, refer to the Install the Operator SDK CLI page.
3. Helm CLI is installed on your computer. To install Helm CLI, follow the instructions from the Installing Helm page.
4. Docker. For details about installing Docker, refer to the Install Docker page.
5. You need a quay.io account.

Set Up The Apache Operator

In this section, we’ll walk you through the process of setting up the Apache Operator using Bitnami’s Helm chart.

1. Add the Bitnami Helm repository to your Helm client by running the following command:

helm repo add bitnami https://charts.bitnami.com

"bitnami" has been added to your repositories

2. Create a Helm-based Operator by running the operator-sdk new command, and passing it the following arguments:

• The name of your operator (apache-operator).
• The --api-version flag with the Kubernetes apiVersion. The format is $GROUP_NAME/$VERSION. In this tutorial, we’ll be using appfleet.com/v1alpha1.
• The --kind flag with the name of the Kubernetes CRD (Apache)
• The --type flag with the type of operator. We’ll be using helm. Other valid types are Go and Ansible
• The helm-chart flag with the name of the Helm chart (bitnami/apache)

operator-sdk new apache-operator --api-version=appfleet.com/v1alpha1 --kind=Apache --type=helm --helm-chart=bitnami/apache

INFO[0000] Creating new Helm operator 'apache-operator'.INFO[0001] Created helm-charts/apacheINFO[0001] Generating RBAC rulesWARN[0001] The RBAC rules generated in deploy/role.yaml are based on the chart's default manifest. Some rules may be missing for resources that are only enabled with custom values, and some existing rules may be overly broad. Double check the rules generated in deploy/role.yaml to ensure they meet the operator's permission requirements.INFO[0001] Created build/DockerfileINFO[0001] Created watches.yamlINFO[0001] Created deploy/service_account.yamlINFO[0001] Created deploy/role.yamlINFO[0001] Created deploy/role_binding.yamlINFO[0001] Created deploy/operator.yamlINFO[0001] Created deploy/crds/appfleet.com_v1alpha1_apache_cr.yamlINFO[0001] Generated CustomResourceDefinition manifests.INFO[0001] Project creation complete.

This command creates the following directory structure:

tree apache-operator -L 2

apache-operator├── build│   └── Dockerfile├── deploy│   ├── crds│   ├── operator.yaml│   ├── role.yaml│   ├── role_binding.yaml│   └── service_account.yaml├── helm-charts│   └── apache└── watches.yaml

Things to note:

• Kubernetes compares the actual state of the cluster with the desired state. Then, it takes action to match these states.  The Operators extend this pattern by watching a specific custom resource type and taking actions to match the spec in that resource. In this example, the Operator watches the Apache resource as defined in the watches.yaml file:

cat apache-operator/watches.yaml

---- version: v1alpha1  group: appfleet.com  kind: Apache  chart: helm-charts/apache

• The Dockerfile uses the quay.io/operator-framework/helm-operator:v0.15.1 image as the base, and then it copies watches.yaml file and the Helm charts:

cat apache-operator/build/Dockerfile

FROM quay.io/operator-framework/helm-operator:v0.15.1COPY watches.yaml ${HOME}/watches.yamlCOPY helm-charts/ ${HOME}/helm-charts/

3. Now you can build the Apache Operator by moving into the apache-operator directory, and then entering the following operator-sdk build command:

operator-sdk build apache-operator:v0.1

INFO[0000] Building OCI image apache-operator:v0.1Sending build context to Docker daemon  64.51kBStep 1/3 : FROM quay.io/operator-framework/helm-operator:v0.15.1 ---> 450a3ca2d02dStep 2/3 : COPY watches.yaml ${HOME}/watches.yaml ---> Using cache ---> db5c285c02fbStep 3/3 : COPY helm-charts/ ${HOME}/helm-charts/ ---> 50255ede17deSuccessfully built 50255ede17deSuccessfully tagged apache-operator:v0.1INFO[0003] Operator build complete.

4. Verify that the Docker image was created with:

docker images | grep apache

apache-operator                            v0.1                50255ede17de        38 seconds ago      174MB

Set Up quay.io with Docker

You can think of Quay as something similar to GitHub, but for Docker images. It’s a registry where you can host images and share them. There are a couple of ways to set up quay.io with Docker. For the sake of simplicity, you’ll use the docker login command. It’s important to note that the docker login command stores the password you enter as plain-text. Thus, you should first generate an encrypted password.

1. Point your browser to https://quay.io/, and then navigate to Account Settings:

2. From the Account Settings page select Generate Encrypted Password:

3. You will be prompted to enter your quay.io password:

4. Select Docker Login and then copy the command containing your Docker encrypted password:

5. In a terminal window, log in to quay.io by entering the following command:

docker login -u="<YOUR_USERNAME>" -p="<YOUR_ENCRYPTED_PASSWORD>" quay.io

WARNING! Using --password via the CLI is insecure. Use --password-stdin.Login Succeeded

Push the Apache Operator Image to quay.io

To do this, you first need to properly tag the image with the hostname of the registry, and the name of your repository.  Then, you can push the image.

1. Enter the following command to tag the local image named apache-operator into the quay.io/andreipope repository:

docker tag apache-operator:v0.1 quay.io/andreipope/apache-operator:v0.1

Note that the image name is comprised by a slash-separated list of the:

• Registry hostname (quay.io)
• Repository (andreipope)
• Operator name (apache-version).

☞ The name of our repository is andreipope, but yours will be different.

2. To push the image that we created in the previous section, run the following command docker push command:

docker push quay.io/andreipope/apache-operator:v0.1

b8325e5fabd7: Pushed2f4354fc6a73: Pushedb496b494f6f9: Pushed9fd48ecc1227: Pushed0141daa77f22: Pushed27cd2023d60a: Pushed4b52dfd1f9d9: Pushedv0.1: digest: sha256:7230926984fc3d688e02764748441a74907eeb772e3b51eb06b1bac225ba9f98 size: 1778

3. Point your browser to https://quay.io/, navigate to the apache-operator repository, and make repository public:

Deploy the Apache Operator

You are now ready to deploy the Apache Operator. Before that, you must customize the specs.

1. Open the deploy/operator.yaml file in a plain-text editor and update the placeholder image: REPLACE_IMAGE with the location of your image (quay.io/andreipope/apache-operator:v0.1)

Your deploy.operator.yaml file should look similar to the following:

apiVersion: apps/v1kind: Deploymentmetadata:  name: apache-operatorspec:  replicas: 1  selector:    matchLabels:      name: apache-operator  template:    metadata:      labels:        name: apache-operator    spec:      serviceAccountName: apache-operator      containers:        - name: apache-operator          # Replace this with the built image name          image: quay.io/andreipope/apache-operator:v0.1          imagePullPolicy: Always          env:            - name: WATCH_NAMESPACE              valueFrom:                fieldRef:                  fieldPath: metadata.namespace            - name: POD_NAME              valueFrom:                fieldRef:                  fieldPath: metadata.name            - name: OPERATOR_NAME              value: "apache-operator"

2. Enter these kubectl create commands to deploy the Apache Operator:

kubectl create -f deploy/service_account.yamlkubectl create -f deploy/role.yamlkubectl create -f deploy/role_binding.yamlkubectl create -f deploy/operator.yaml

serviceaccount/expressjs-operator createdrole.rbac.authorization.k8s.io/expressjs-operator createdrolebinding.rbac.authorization.k8s.io/expressjs-operator createddeployment.apps/expressjs-operator created

3. Check the status of the deployment:

kubectl get deploymentNAME              READY   UP-TO-DATE   AVAILABLE   AGEapache-operator   1/1     1            1           24s

☞ Note that the deployment doesn’t define the spec for the Apache cluster. You’ll describe the Apache cluster in the next section, once the Operator is running.

4. The Operator is a pod running in this deployment. To see it, type the following command:

kubectl get pods

kubectl get podsNAME                               READY   STATUS    RESTARTS   AGEapache-operator-6d5795f879-np6pr   1/1     Running   1          38s

Deploy the Apache Cluster

In the Set Up the Apache Operator section, you created a CRD  defining a new kind of resource, an Apache cluster. Now, you will apply that spec so that the Operator starts watching the Apache resources. Then, you will deploy the Apache cluster itself.

1. First, let’s deploy the CRD that defines the resources the Operator will monitor:

kubectl apply -f deploy/crds/appfleet.com_apaches_crd.yamlcustomresourcedefinition.apiextensions.k8s.io/apaches.appfleet.com created

customresourcedefinition.apiextensions.k8s.io/apaches.appfleet.com created

2. At this point, you are ready to deploy the Apache cluster:

kubectl apply -f deploy/crds/appfleet.com_v1alpha1_apache_cr.yaml

apache.appfleet.com/example-apache created

3. The deployment takes a bit of time to complete. Once everything is ready, you should see a new pod running Apache:

kubectl get pods

NAME                               READY   STATUS    RESTARTS   AGEapache-operator-6d5795f879-np6pr   1/1     Running   4          2m53sexample-apache-7cf789fc98-462dr    1/1     Running   0          39s

4. You can also check that the deployment was created by entering the following command:

kubectl get deployment

NAME              READY   UP-TO-DATE   AVAILABLE   AGEapache-operator   1/1     1            1           5m43sexample-apache    2/2     2            2           3m29s

Scaling Up

At this point, you have a running Apache cluster. To add another instance, you must modify the replicaCount field  in the deploy/crds/appfleet.com_v1alpha1_apache_cr.yamlapache.appfleet.com/example-apache  file. Then, you need to apply the new spec.

1. Open the deploy/crds/appfleet.com_v1alpha1_apache_cr.yamlapache.appfleet.com/example-apache file in a plain-text editor, and specify spec.replicaCount: 2.
   The updated file should look similar to the following:

apiVersion: appfleet.com/v1alpha1kind: Apachemetadata:  name: example-apachespec:  # Default values copied from <project_dir>/helm-charts/apache/values.yaml  affinity: {}  cloneHtdocsFromGit:    enabled: false    interval: 60  git:    pullPolicy: IfNotPresent    registry: docker.io    repository: bitnami/git    tag: 2.25.0-debian-10-r0  image:    debug: false    pullPolicy: IfNotPresent    registry: docker.io    repository: bitnami/apache    tag: 2.4.41-debian-10-r0  ingress:    annotations: {}    certManager: false    enabled: false    hostname: example.local    secrets: null    tls:    - hosts:      - example.local      secretName: example.local-tls  livenessProbe:    enabled: true    failureThreshold: 6    initialDelaySeconds: 180    path: /    periodSeconds: 20    port: http    successThreshold: 1    timeoutSeconds: 5  metrics:    enabled: false    image:      pullPolicy: IfNotPresent      registry: docker.io      repository: bitnami/apache-exporter      tag: 0.7.0-debian-10-r0    podAnnotations:      prometheus.io/port: "9117"      prometheus.io/scrape: "true"    resources:      limits: {}      requests: {}  nodeSelector: {}  podAnnotations: {}  readinessProbe:    enabled: true    failureThreshold: 6    initialDelaySeconds: 30    path: /    periodSeconds: 10    port: http    successThreshold: 1    timeoutSeconds: 5  replicaCount: 2  resources:    limits: {}    requests: {}  service:    annotations: {}    externalTrafficPolicy: Cluster    httpsPort: 443    nodePorts:      http: ""      https: ""    port: 80    type: LoadBalancer  tolerations: {}

2. You can apply the updated spec with:

kubectl apply -f deploy/crds/appfleet.com_v1alpha1_apache_cr.yaml

apache.appfleet.com/example-apache configured

3. After applying the updated spec, the state of the cluster differs from the desired state. The Operator starts a new instance of the Apache webserver to reconcile the two, scaling up the cluster:

kubectl get pods

NAME                               READY   STATUS              RESTARTS   AGEapache-operator-6d5795f879-np6pr   1/1     Running             4          4m10sexample-apache-7cf789fc98-462dr    1/1     Running             0          116sexample-apache-7cf789fc98-fvttm    0/1     ContainerCreating   0          1s

Wait a bit until the second container is created:

kubectl get pods

NAME                               READY   STATUS    RESTARTS   AGEapache-operator-6d5795f879-np6pr   1/1     Running   0          5m2sexample-apache-7cf789fc98-462dr    1/1     Running   0          2m48sexample-apache-7cf789fc98-fvttm    1/1     Running   0          53s

In the above output, note that two example-apache pods are running.

Verify Your Installation

The Apache Operator creates a Kubernetes service which is basically an endpoint where clients can obtain access to the group of pods running Apache. You will use this service to access your cluster using a web browser.

1. Make sure that the service is up with:

kubectl get service

NAME                      TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGEapache-operator-metrics   ClusterIP      10.104.41.83   <none>        8686/TCP,8383/TCP            5m46sexample-apache            LoadBalancer   10.99.54.115   <pending>     80:31058/TCP,443:30362/TCP   3m51skubernetes                ClusterIP      10.96.0.1      <none>        443/TCP

2. Forward the connections made to http://localhost:80 to the pod running the service/example-apache service:

kubectl port-forward service/example-apache 80:80

Forwarding from 127.0.0.1:80 -> 8080Forwarding from [::1]:80 -> 8080

3. Point your browser to http://localhost:80. If everything worked well, you should see something like the following:

Now you should have a good understanding of how Kubernetes Operators work. Furthermore, by completing this tutorial you learned how create an operator for your application using a Helm chart.

Discover more with Gcore Managed Kubernetes