Dynatrace

In order to evaluate the quality gates and allow self-healing in production, we have to set up monitoring to get the needed data.

Prerequisites

Please make sure to have the following tool(s) installed:

  • jq - a lightweight and flexible command-line JSON processor.

Open for installation instructions

  sudo apt-get update
  sudo apt install jq -y

Setup Dynatrace

  1. Bring your Dynatrace SaaS or Dynatrace-managed tenant

    If you don’t have a Dynatrace tenant, sign up for a free trial or a developer account.

  2. Create a Dynatrace API Token

    Log in to your Dynatrace tenant, and go to Settings > Integration > Dynatrace API. Then, create a new API token with the following permissions:

    • Access problem and event feed, metrics and topology
    • Access logs
    • Configure maintenance windows
    • Read configuration
    • Write configuration
    • Capture request data
    • Real user monitoring JavaScript tag management
    Dynatrace API Token
  3. Create a Dynatrace PaaS Token

    In your Dynatrace tenant, go to Settings > Integration > Platform as a Service, and create a new PaaS Token.

  4. Clone the install repository and setup your credentials by executing the following steps:

    git clone --branch 0.5.0 https://github.com/keptn-contrib/dynatrace-service --single-branch
    
    cd dynatrace-service/deploy/scripts
    
    ./defineDynatraceCredentials.sh
    

    When the script asks for your Dynatrace tenant, please enter your tenant according to the appropriate pattern:

    • Dynatrace SaaS tenant: {your-environment-id}.live.dynatrace.com
    • Dynatrace-managed tenant: {your-domain}/e/{your-environment-id}
  5. Execute the installation script for your platform:

    • If you are on Azure AKS, please execute
    ./deployDynatraceOnAKS.sh
    
    • If you are on AWS EKS, please execute
    ./deployDynatraceOnEKS.sh
    
    • If you are on Google GKE, please execute
    ./deployDynatraceOnGKE.sh
    

    Please read Note 2 after this section in case you are running GKE Container-optimized os.

    • If you are on Pivotal PKS, please execute
    ./deployDynatraceOnPKS.sh
    
    • If you are on OpenShift, please execute
    ./deployDynatraceOnOpenshift.sh
    

When this script is finished, the Dynatrace OneAgent and the dynatrace-service are deployed in your cluster. Execute the following commands to verify the deployment of the OneAgent as well as of the dynatrace-service.

kubectl get svc dynatrace-service -n keptn
NAME                TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
dynatrace-service   ClusterIP   10.0.44.191   <none>        8080/TCP   2m48s
kubectl get pods -n dynatrace
NAME                                           READY   STATUS    RESTARTS   AGE
dynatrace-oneagent-operator-7f477bf78d-dgwb6   1/1     Running   0          12m
oneagent-5lcqh                                 0/1     Running   0          3s
oneagent-ps6t4                                 0/1     Running   0          3s

Note 1: To monitor the services that are already onboarded in the dev, staging, and production namespace, make sure to restart the pods. If you defined different environments in your shipyard file, please adjust the parameters accordingly.

kubectl delete pods --all --namespace=sockshop-dev
kubectl delete pods --all --namespace=sockshop-staging
kubectl delete pods --all --namespace=sockshop-production

Note 2: If the nodes in your cluster run on Container-Optimized OS (cos) (default for GKE), the Dynatrace OneAgent might not work properly, and another step is necessary. To verify that the OneAgent does not work properly, the output of kubectl get pods -n dynatrace might look as follows:

NAME                                           READY   STATUS             RESTARTS   AGE
dynatrace-oneagent-operator-7f477bf78d-dgwb6   1/1     Running            0          8m21s
oneagent-b22m4                                 0/1     Error              6          8m15s
oneagent-k7jn6                                 0/1     CrashLoopBackOff   6          8m15s

This means that after the initial setup with deployDynatrace.sh, which is a step below, the cr.yml has to be edited and applied again. You can do that by editing the already downloaded cr.yml in ../manifests/dynatrace/gen and set the environemnt variable as follows:

  env:
  - name: ONEAGENT_ENABLE_VOLUME_STORAGE
    value: "true"

Then apply the file using:

kubectl apply -f cr.yml

Finally, don’t forget to restart the pods as described in Note 1 above.

Verify setup in Dynatrace

When you navigate to Settings > Tags > Automatically applied tags in your Dynatrace tenant, you will find following tagging rules:

  • keptn_deployment
  • keptn_project
  • keptn_service
  • keptn_stage

This means that Dynatrace will automatically apply tags to your onboarded services.

In addition, a Problem Notification has automatically been set up to inform Keptn of any problems with your services to allow auto-remediation. This will be described in more detail in the Runbook Automation tutorial. You can check the problem notification by navigating to Settings > Integration > Problem notifications and you will find a keptn remediation problem notification.

Setup Dynatrace SLI provider

During the evaluation of a quality gate, the Dynatrace SLI provider is required that is implemented by an internal Keptn service, the dynatrace-sli-service. This servcie will fetch the values for the SLIs that are referenced in a SLO configuration.

  • To install the dynatrace-sli-service, complete the following tasks:
  git clone --branch 0.2.0 https://github.com/keptn-contrib/dynatrace-sli-service --single-branch
  cd dynatrace-sli-service/deploy
  kubectl apply -f distributor.yaml
  kubectl apply -f service.yaml
  • To verify the deplyoment, execute:
  kubectl get pods -n keptn | grep dynatrace-sli
  dynatrace-sli-service
  dynatrace-sli-service-monitoring-configure-distributor

Note: If you don’t monitor your Kubernetes cluster with Dynatrace (i.e., you have not completed the steps from Setup Dynatrace), the dynatrace-sli-service needs a secret containing the Tenant ID and API token in a yaml file as shown below.

  • Provide a the file your_credential_file.yaml with following content:
  DT_TENANT: YOUR_TENANT_ID.live.dynatracelabs.com
  DT_API_TOKEN: XYZ123456789
  • Before executing the next command, please adapt the instruction to match the name of your project. Then, create the secret in the keptn namespace using:
  kubectl create secret generic dynatrace-credentials-PROJECTNAME -n "keptn" --from-file=dynatrace-credentials=your_credential_file.yaml

Set DT_CUSTOM_PROP before onboarding a service

The created tagging rules in Dynatrace expect the environment variable DT_CUSTOM_PROP for your onboarded service. Consequently, make sure to specify the environment variable for deployment in the Helm chart of the service you are going to onboard with the following value:

  env:
  - name: DT_CUSTOM_PROP
    value: "keptn_project={{ .Values.keptn.project }} keptn_service={{ .Values.keptn.service }} keptn_stage={{ .Values.keptn.stage }} keptn_deployment={{ .Values.keptn.deployment }}"
        

See Keptn events in Dynatrace

The dynatrace-service in Keptn will take care of pushing events of the Keptn workflow to the artifacts that have been onboarded. For example, the deployment as well as custom infos like starting and finishing of tests will appear in the details screen of your services in your Dynatrace tenant.

Keptn events

Create process group naming rule in Dynatrace

While it is not a technical requirement, we encourage you setting up a process group naming rule within Dynatrace.

  1. Go to Settings, Process and containers, and click on Process group naming.
  2. Create a new process group naming rule with Add new rule.
  3. Edit that rule:
    • Rule name: Container.Namespace
    • Process group name format: {ProcessGroup:KubernetesContainerName}.{ProcessGroup:KubernetesNamespace}
    • Condition: Kubernetes namespace > exits
  4. Click on Preview and Save.

    Screenshot shows this rule definition.

    Dynatrace naming rule

Uninstall Dynatrace

If you want to uninstall Dynatrace, there are scripts provided to do so. Uninstalling Keptn will not automatically uninstall Dynatrace.

  1. (optional) If you do not have the dynatrace-service repository, clone the latest release using:
  git clone --branch 0.5.0 https://github.com/keptn-contrib/dynatrace-service --single-branch
  1. Go to correct folder and execute the uninstallDynatrace.sh script:
  cd dynatrace-service/deploy/scripts
  ./uninstallDynatrace.sh