Application Performance Monitoring on Kubernetes

In this quick demo, we are going to setup Application Performance Monitoring on Kubernetes using Elastic APM server and Elasticsearch and Kibana.

What is Application Performance Monitoring(APM)?

APM focuses on tracking the performance of an application. APM collects errors, traces, requests rates, latency times from applications to identify the source of the performance issues/bottlenecks.

How to setup Elastic APM on Kubernetes?

Prerequisites:

• A Kubernetes cluster(For information on how to deploy a GKE cluster, see this post.)
• kubectl client library to connect to Kubernetes Cluster
• Admin privileges on Kubernetes Cluster
• Up & Running Elasticsearch Cluster with Kibana frontend(For information on how to deploy Elasticsearch and Kibana on GKE, see this post)

Connect to GKE cluster

gcloud container clusters get-credentials demo-k8s-cluster

Create a Namespace for Elastic APM Server

By running following command, we are going to create a namespace called “apm” for creating APM server Pods, Service.

cloudshell:~/apm$ kubectl create namespace apm
namespace/apm created

Deploy Elastic APM Server

Following yaml configuration creates a ConfigMap for APM server configuration, a service for exposign APM server pods, a deployment for managing APM server pods. It pushes data to Elasticsearch endpoint “elasticsearch.elk.svc.cluster.local:9200“.

apiVersion: v1
kind: ConfigMap
metadata:
  name: apm-server-config
  namespace: apm
  labels:
    k8s-app: apm-server
data:
  apm-server.yml: |-
    apm-server:
      host: "0.0.0.0:8200"
      frontend:
        enabled: true
    output.elasticsearch:
      hosts: ["elasticsearch.elk.svc.cluster.local:9200"]
---
apiVersion: v1
kind: Service
metadata:
  name: apm-server
  namespace: apm
  labels:
    k8s-app: apm-server
spec:
  ports:
  - port: 8200
    targetPort: 8200
    name: http

  selector:
    k8s-app: apm-server
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: apm-server
  namespace: apm
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: apm-server
  template:
    metadata:
      labels:
        k8s-app: apm-server
    spec:
      containers:
      - name: apm-server
        image: docker.elastic.co/apm/apm-server:7.17.5
        ports:
        - containerPort: 8200
          name: apm-port
        volumeMounts:
        - name: apm-server-config
          mountPath: /usr/share/apm-server/apm-server.yml
          readOnly: true
          subPath: apm-server.yml
      volumes:
      - name: apm-server-config
        configMap:
          name: apm-server-config

Copy above yaml configuration to a file called apm-server.yaml and apply it using kubectl command.

cloudshell:~/apm$ k apply -f apm.yaml
configmap/apm-server-config created
service/apm-server created
deployment.apps/apm-server created

Deploy Sample application with APM agent as a side car container

The following yaml configuration, when applied, creates a deployment with 2 containers, Elastic APM agent container and a sample petclinic application container for generating performance monitoring data. It also a external load balancer to access petclinic application.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: petclinic
  namespace: apm
  labels:
    app: petclinic
spec:
  replicas: 1
  selector:
    matchLabels:
      app: petclinic
  template:
    metadata:
      labels:
        app: petclinic
    spec:
      dnsPolicy: ClusterFirstWithHostNet
      volumes:
      - name: elastic-apm-agent
        emptyDir: {}
      initContainers:
      - name: elastic-java-agent
        image: docker.elastic.co/observability/apm-agent-java:1.12.0
        volumeMounts:
        - mountPath: /elastic/apm/agent
          name: elastic-apm-agent
        command: ['cp', '-v', '/usr/agent/elastic-apm-agent.jar', '/elastic/apm/agent']
      containers:
      - name: petclinic
        image: eyalkoren/pet-clinic:without-agent
        volumeMounts:
        - mountPath: /elastic/apm/agent
          name: elastic-apm-agent
        env:
        - name: ELASTIC_APM_SERVER_URL
          value: "http://apm-server.apm.svc.cluster.local:8200"
        - name: ELASTIC_APM_SERVICE_NAME
          value: "petclinic"
        - name: ELASTIC_APM_APPLICATION_PACKAGES
          value: "org.springframework.samples.petclinic"
        - name: ELASTIC_APM_LOG_LEVEL
          value: DEBUG
        - name: JAVA_TOOL_OPTIONS
          value: -javaagent:/elastic/apm/agent/elastic-apm-agent.jar

---
apiVersion: v1
kind: Service
metadata:
  name: petclinic
  namespace: apm
  labels:
    app: petclinic
spec:
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector:
    app: petclinic

Copy above yaml configuration to a file called apm-agent-with-sample-app.yaml and apply it using kubectl command.

cloudshell:~/apm$ k apply -f apm-agent-with-sample-app.yaml
deployment.apps/petclinic created
service/petclinic created

Observe APM metrics on Kibana

Now let’s go to GKE services console to access petclinic sample application and browse the application to generate some performance monitoring data.

Click on external load balancer links to open Kibana and petclinic appliation.

Browse petclinic application by clicking on all tabs of the navigation menu.

Now let’s navigate to Kibana’s APM server console. Click on burger menu and then click on “APM” option under “Observability” section.

It will take to APM server console. Click on “petclinic”.

Under petclinic application you can see performance monitoring information and dashboards.

Under “Errors” tab you can see application errors captured by APM server

Under “Transactions” tab you can see trace sample for each call.

Under “JVMs” you can see JVM’s CPU and memory usage.

Conclusion

In this quick start demo we have deployed Elastic APM server on a Kubernetes cluster to monitor application performance and ship monitoring data to Elasticsearch cluster. You can find more information about APM in official documentation.

You may also like:

Elasticsearch & Kibana Setup on Kubernetes Cluster

Fluentd: Kubernetes Log Collection with Fluentd, Elasticsearch