What is Elasticsearch?

Elasticsearch is a distributed search engine based on Apache Lucene library.

It is multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.

Elasticsearch is the central component of the ELK/EFK(Elasticsearch, Logstash/Fluentd, Kibana) Stack. It is very useful for managing logs of IT systems and applications.

What is Kibana?

Kibana is a frontend application that provides search and data visualization capabilities for data indexed in Elasticsearch.

Commonly known as the charting tool for the ELK/EFK Stack. Kibana also acts as the user interface for monitoring, managing, and securing an Elastic Stack cluster.

In this quick start demo, we are going to setup an Elasticsearch cluster on a Google Kubernetes Engine cluster.

Elasticsearch Cluster Setup

Prerequisites:

• A Kubernetes cluster with 3 nodes(For information on how to deploy a GKE cluster, see this post.)
• kubectl client library to connect to Kubernetes Cluster
• Admin privileges on Kubernetes Cluster

Connect to GKE cluster

gcloud container clusters get-credentials demo-k8s-cluster

Make sure that the cluster has 3 nodes since we are going to deploy a Elasticsearch cluster with 3 pods.

cloudshell:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-demo-cluster-demo-gke-node-pool-ce0ec65d-4zk7 Ready 4d23h v1.22.8-gke.202
gke-demo-cluster-demo-gke-node-pool-ce0ec65d-hr8h Ready 29h v1.22.8-gke.202
gke-demo-cluster-demo-gke-node-pool-ce0ec65d-sqfg Ready 3h8m v1.22.8-gke.202

Create a Namespace for Elasticsearch Cluster

By running following command, we are going to create a namespace called “elk” for creating Elasticsearch Pods, Services, Storage Class and PVs.

cloudshell:~/elk$ kubectl create namespace elk
namespace/elk created

Create a Storage Class for Persistence Storage

Since Elasticsearch needs storage persistency, we are going to use Kubernetes Statefulsets to manage pods.

Here you can find more information about Kubernetes Statefulsets.

Copy following configuration to a file called es-sc.yaml and apply it using kubectl command.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: elastic-sc
  namespace: elk
provisioner: kubernetes.io/gce-pd
volumeBindingMode: Immediate
allowVolumeExpansion: true
reclaimPolicy: Delete
parameters:
  type: pd-standard
  fstype: ext4
  replication-type: none

cloudshell:~/elk$ kubectl apply -f es-sc.yaml
storageclass.storage.k8s.io/elastic-sc created

StatefulSet Deployment

Kubernetes StatefulSet requires a headless service to provide network identity to the pods it creates.

Copy following configuration to a file called es-service.yaml and apply it using kubectl command.

kind: Service
apiVersion: v1
metadata:
  name: elasticsearch
  namespace: elk
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  clusterIP: None
  ports:
    - port: 9200
      name: rest
    - port: 9300
      name: inter-node

cloudshell:~/elk$ kubectl apply -f es-service.yaml
service/elasticsearch created

Now we are going to create Elasticsearch StatefulSets.

Copy following configuration to a file called es-statefulsets.yaml.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es-cluster
  namespace: elk
spec:
  serviceName: elasticsearch
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: elasticsearch:7.17.5
        imagePullPolicy: Always
        resources:
            limits:
              cpu: 1000m
            requests:
              cpu: 100m
        ports:
        - containerPort: 9200
          name: rest
          protocol: TCP
        - containerPort: 9300
          name: inter-node
          protocol: TCP
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
        env:
          - name: cluster.name
            value: k8s-logs
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: discovery.seed_hosts
            value: "es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch"
          - name: cluster.initial_master_nodes
            value: "es-cluster-0"
          - name: ES_JAVA_OPTS
            value: "-Xms512m -Xmx512m"
          - name: network.host
            value: "_site_"
      initContainers:
      - name: fix-permissions
        image: busybox
        command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
        securityContext:
          privileged: true
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
      - name: increase-vm-max-map
        image: busybox
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      - name: increase-fd-ulimit
        image: busybox
        command: ["sh", "-c", "ulimit -n 65536"]
        securityContext:
          privileged: true
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        app: elasticsearch
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: elastic-sc
      resources:
        requests:
          storage: 100Gi

Apply it using kubectl command.

When applied 3 init containers will run first to change permissions and kernel parameters. Each pod will have 100G persistence storage.

cloudshell:~/elk$ kubectl apply -f es-statefulsets.yaml
statefulset.apps/es-cluster created

List running pods, services, volumes using kubectl commands.

cloudshell:~/elk$ kubectl get pods -n elk
NAME READY STATUS RESTARTS AGE
es-cluster-0 1/1 Running 0 11m
es-cluster-1 1/1 Running 0 11m
es-cluster-2 1/1 Running 0 10m
cloudshell:~/elk$
cloudshell:~/elk$ kubectl get endpoints -n elk -o wide
NAME ENDPOINTS AGE
elasticsearch 172.22.1.5:9200,172.22.2.3:9200,172.22.2.4:9200 + 3 more… 21m
cloudshell:~/elk$
cloudshell:~/elk$ kubectl get pvc -n elk
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-es-cluster-0 Bound pvc-62de1f22-6d52-47c3-a538-7199496fee37 100Gi RWO elastic-sc 12m
data-es-cluster-1 Bound pvc-942287cc-0cfa-4c1b-9479-ae7638232b2e 100Gi RWO elastic-sc 11m
data-es-cluster-2 Bound pvc-a54edd05-e7fe-4cd0-ab23-82306209efe9 100Gi RWO elastic-sc 11m

Deploying Kibana

Now let’s deploy Kibana on GKE cluster to search Elasticsearch cluster using Web interface.

Copy following configuration to a file called kibana.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kb-deployment
  namespace: elk
  labels:
    app: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: kibana:7.17.5
        imagePullPolicy: Always
        resources:
          limits:
            cpu: 2000m
            memory: "2048Mi"
          requests:
            cpu: 500m
            memory: "1024Mi"
        env:
          - name: ELASTICSEARCH_URL
            value: http://elasticsearch.elk.svc.cluster.local:9200
        ports:
        - containerPort: 5601
---
apiVersion: v1
kind: Service
metadata:
  name: kb-svc
  namespace: elk
  labels:
    app: kibana
spec:
  ports:
  - port: 80
    targetPort: 5601
  selector:
    app: kibana
  type: LoadBalancer

Apply above configuration using kubectl command.

This will create a Kibana deployment and a service with external load balancer to access from internet.

This configuration will point Kibana to elasticsearch.elk.svc.cluster.local:9200 which we created in earlier steps.

cloudshell:~/elk$ kubectl apply -f kibana.yaml
deployment.apps/kb-deployment created
service/kb-svc created

After couple of minutes you can access Kibana web UI using external load balancer IP address from GKE’s ‘Services & Ingress’ console.

We can check Elasticsearch cluster health status from Kibana console. Go to ‘Management’ -> ‘Dev Tools’ section.

Run GET request against ‘/_cluster/health’ endpoint to get Elasticsearch cluster health status. In the following screen shot you can see our ‘k8s-logs‘ status is green.

After sending some log messages using Fluentd(refer this post) we can create an index pattern and access them from ‘Analytics’ -> ‘Discover’ console.

To create index pattern go to ‘Stack Mangement’ -> ‘index pattern’ from Kibana main console, then click on ‘Create index pattern’

Since we are using Fluentd for log shipping it will create index with “logstash” prefix by default.

Give a name to index pattern, since our log indexes have “logstash” prefix to match with them we need to use “logstatsh-*” as index pattern name.

Then select “@timestamp” as Timestamp field, then click on “Create index pattern” button.

Now go to ‘Analytics’ -> ‘Discovr’ console to see and search log messages stored in Elasticsearch cluster.

Conclusion

In this quick start demo we have deployed Elasticsearch cluster and Kibana on a GKE cluster and accessed Kibana web UI and verified Elasticsearch cluster health status. We have also created a index pattern and access logs stored in Elasticsearch cluster. You can find more information about Elastic stack in official documentation.