Day 2 Operations

After you have successfully rolled out your application by following the instructions in the integration guide, Keptn also assists you with day 2 operations for your application.

Tasks that fall under this category include:

• Updating the version of one or more workloads that are part of the same application
• Adding a new workload to an existing application
• Monitoring the health of your application using KeptnMetrics, as described here
• Optimizing the resource usage of your applications by integrating KeptnMetrics into a HorizontalPodAutoscaler (HPA), as described here

Updating Workload Versions

After a first successful deployment of your application with Keptn, you will eventually need to update the version of one or more workloads that are part of the application. This usually involves updating the image of a deployment and changing the configuration of a deployment. For example, using a different service account name for a pod could be seen as an update. Regardless of that, however, it is the user who decides what Keptn sees as a version bump in the application by setting the value of the keptn.sh/version or app.kubernetes.io/version labels/annotations in their workloads.

When this changes, Keptn interprets a change as a new version and thus re-runs the pre- and post-tasks and evaluations for the application as well as promotion tasks if the previous phases succeeded.

If the version label/annotation does not change, Keptn does not consider a change of a workload configuration to be an update, and therefore no pre- and post-tasks/evaluations are executed because they have already been completed for the version set in the labels/annotations.

To illustrate the update of a workload, let's assume the following example, including a workload called podtato-head-frontend that includes a pre-deployment task.

apiVersion: v1
kind: Namespace
metadata:
  name: podtato-kubectl
  annotations:
    keptn.sh/lifecycle-toolkit: "enabled"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: podtato-head-frontend
  namespace: podtato-kubectl
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: podtato-head-frontend
  template:
    metadata:
      labels:
        app.kubernetes.io/name: podtato-head-frontend
        app.kubernetes.io/part-of: podtato-head
        app.kubernetes.io/version: 0.3.0
        keptn.sh/pre-deployment-tasks: pre-deployment-check
    spec:
      containers:
        - name: podtato-head-frontend
          image: ghcr.io/podtato-head/podtato-server:v0.3.0
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: podtato-head-hat
  namespace: podtato-kubectl
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: podtato-head-hat
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: podtato-head-hat
        app.kubernetes.io/part-of: podtato-head
        app.kubernetes.io/version: 0.3.0
    spec:
      containers:
        - name: podtato-head-hat
          image: ghcr.io/podtato-head/podtato-server:v0.3.0
---
apiVersion: lifecycle.keptn.sh/v1
kind: KeptnTaskDefinition
metadata:
  name: pre-deployment-check
  namespace: podtato-kubectl
spec:
  deno:
    inline:
      code: |
        console.log("Success")
---
apiVersion: lifecycle.keptn.sh/v1
kind: KeptnAppContext
metadata:
  name: podtato-head
  namespace: podtato-kubectl
spec:
  preDeploymentTasks:
    - pre-deployment-check

Now, let's assume that the configuration of that workload needs to be changed. In this example we assume that the image of that workload should be updated, but a configuration change is not limited to that. From here, you essentially have two options:

• Only update the configuration without updating the app.kubernetes.io/version label: This can be useful when the change in the configuration should happen regardless of the result of any task or evaluation, e.g., when the previously used image has a critical vulnerability and the image must be updated as quickly as possible. To do that, change podtato-head-frontend as follows:

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: podtato-head-frontend
    namespace: podtato-kubectl
  spec:
    selector:
      matchLabels:
        app.kubernetes.io/name: podtato-head-frontend
    template:
      metadata:
        labels:
          app.kubernetes.io/name: podtato-head-frontend
          app.kubernetes.io/part-of: podtato-head
          app.kubernetes.io/version: 0.3.0 # the version number stays the same
          keptn.sh/pre-deployment-tasks: pre-deployment-check
      spec:
        containers:
          - name: podtato-head-frontend
            image: ghcr.io/podtato-head/podtato-server:v0.3.1 # the image tag has been updated to v0.3.1
  

• Update the configuration and the version label: Doing so causes the KeptnWorkload that is associated with the podtato-head-frontend deployment to be updated, and therefore the pre-task my-task and pre-evaluation my-evaluation are executed before the updated pods are scheduled. In this case, the deployment should be changed as follows:

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: podtato-head-frontend
    namespace: podtato-kubectl
  spec:
    selector:
      matchLabels:
        app.kubernetes.io/name: podtato-head-frontend
    template:
      metadata:
        labels:
          app.kubernetes.io/name: podtato-head-frontend
          app.kubernetes.io/part-of: podtato-head
          app.kubernetes.io/version: 0.3.1 # The version label is now updated as well
          keptn.sh/pre-deployment-tasks: pre-deployment-check
      spec:
        containers:
          - name: podtato-head-frontend
            image: ghcr.io/podtato-head/podtato-server:v0.3.1 # The image tag has been updated from 'a' to 'b'
  

Applying this causes the KeptnApp resource to be updated with a new version, and a new KeptnAppVersion resource to be created. Due to this, all checks defined in the KeptnAppContext resource as well as those defined in the deployment's keptn.sh/pre-deployment-tasks label are executed again.

After applying the updated manifests, you can monitor the status of the application and related workloads using the following commands:

$ kubectl get keptnworkloadversion -n podtato-kubectl

NAMESPACE   NAME                                             APPNAME         WORKLOADNAME                         WORKLOADVERSION      PHASE
podtato-kubectl   podtato-head-podtato-head-frontend-0.3.0   podtato-head    podtato-head-podtato-head-frontend   0.3.0                Completed
podtato-kubectl   podtato-head-podtato-head-hat-0.3.0        podtato-head    podtato-head-podtato-head-hat        0.3.0                Completed
podtato-kubectl   podtato-head-podtato-head-frontend-0.3.1   podtato-head    podtato-head-podtato-head-frontend   0.3.1                Completed

As can be seen in the output of the command, the KeptnWorkloadVersion resources from the previous deployment are still here, but a new KeptnWorkloadVersion for the updated workload has been added. For the workload that remained unchanged (podtato-head-hat), no new KeptnWorkloadVersion needed to be created.

Similarly, retrieving the list of KeptnAppVersions will reflect the update by returning a newly created KeptnAppVersion.

$ kubectl get keptnappversion -n podtato-kubectl

NAMESPACE         NAME                               APPNAME        VERSION   PHASE
podtato-kubectl   podtato-head-f13dcb00ea-6b86b273   podtato-head   0.1.0     Completed
podtato-kubectl   podtato-head-1c40c739cf-d4735e3a   podtato-head   0.1.0     Completed

Adding a new Workload to an Application

To add a new workload (e.g. a new deployment) to an existing app, you must:

• Make sure the keptn.sh/app/app.kubernetes.io/part-of label/annotation is present on the new workload
• Add the new workload to the KeptnApp, if you have previously defined the KeptnApp resource manually. If the application has been discovered automatically, this step is not needed.

For example, to add the deployment podtato-head-left-leg to the podtato-head application, the configuration for that new deployment would look like this, with the required label being set:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: podtato-head-left-leg
  namespace: podtato-kubectl
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: podtato-head-left-leg
  template:
    metadata:
      labels:
        app.kubernetes.io/name: podtato-head-left-leg
        app.kubernetes.io/part-of: podtato-head
        app.kubernetes.io/version: 0.3.0
    spec:
      containers:
        - name: podtato-head-left-leg
          image: ghcr.io/podtato-head/podtato-server:v0.3.1

After applying the updated manifests, you can monitor the status of the application and related workloads using the following commands:

$ kubectl get keptnworkloadversion -n podtato-kubectl

NAMESPACE   NAME                                             APPNAME         WORKLOADNAME                         WORKLOADVERSION      PHASE
podtato-kubectl   podtato-head-podtato-head-frontend-0.3.0   podtato-head    podtato-head-podtato-head-frontend   0.3.0                Completed
podtato-kubectl   podtato-head-podtato-head-frontend-0.3.1   podtato-head    podtato-head-podtato-head-frontend   0.3.1                Completed
podtato-kubectl   podtato-head-podtato-head-hat-0.3.0        podtato-head    podtato-head-podtato-head-hat        0.3.0                Completed
podtato-kubectl   podtato-head-podtato-head-left-leg-0.3.0   podtato-head    podtato-head-podtato-head-left-leg   0.3.0                Completed

As can be seen in the output of the command, in addition to the previous KeptnWorkloadVersions, the newly created KeptnWorkloadVersion, podtato-head-podtato-head-left-leg-0.3.0 has been added to the results.

Comments