In this quick start demo, we are going to do Prometheus Blackbox Exporter setup on Kubernetes cluster to probe HTTP or HTTPS endpoints.

What is Blackbox Exporter?

Blackbox exporter allows blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP, ICMP and gRPC. We can use Prometheus to scrap blackbox exporter metrics and create dashboards in Grafana.

Exporter Setup on Kubernetes

Prerequisites:

• A Kubernetes cluster(For information on how to deploy a GKE cluster, see this post.)
• kubectl client library to connect to Kubernetes Cluster
• Admin privileges on Kubernetes Cluster.
• up & and running Prometheus Server (For information on how to deploy a Prometheus Server, see this post.)
• up & and running Grafana Server (For information on how to deploy a Grafana instance, see this post.)

Connect to GKE cluster

gcloud container clusters get-credentials demo-k8s-cluster

To deploy Blackbox exporter and service on Kubernetes cluster, copy below yaml configuration to blackbox.yaml file.

apiVersion: v1
kind: ConfigMap
metadata:
  name: blackbox-exporter-config
  namespace: monitoring
  labels:
    k8s-app: blackbox-exporter
data:
  blackbox.yaml: |
    modules:
      http_2xx:
        http:
          no_follow_redirects: true
          preferred_ip_protocol: ip4
          valid_http_versions:
          - HTTP/1.1
          - HTTP/2.0
          valid_status_codes: []
        prober: http
        timeout: 5s
---
kind: Service
apiVersion: v1
metadata:
  name: blackbox-exporter
  namespace: monitoring
  labels:
    k8s-app: blackbox-exporter
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 9115
      protocol: TCP
  selector:
    k8s-app: blackbox-exporter

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: blackbox-exporter
  namespace: monitoring
  labels:
    k8s-app: blackbox-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: blackbox-exporter
  template:
    metadata:
      labels:
        k8s-app: blackbox-exporter
    spec:
      restartPolicy: Always
      containers:
        - name: blackbox-exporter
          image: "prom/blackbox-exporter"
          imagePullPolicy: IfNotPresent
          securityContext:
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1000
          args:
            - "--config.file=/config/blackbox.yaml"
          resources:
            {}
          ports:
            - containerPort: 9115
              name: http
          livenessProbe:
            httpGet:
              path: /health
              port: http
          readinessProbe:
            httpGet:
              path: /health
              port: http
          volumeMounts:
            - mountPath: /config
              name: config
        - name: configmap-reload
          image: "jimmidyson/configmap-reload:v0.2.2"
          imagePullPolicy: "IfNotPresent"
          securityContext:
            runAsNonRoot: true
            runAsUser: 65534
          args:
            - --volume-dir=/etc/config
            - --webhook-url=http://localhost:9115/-/reload
          resources:
            {}
          volumeMounts:
            - mountPath: /etc/config
              name: config
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: blackbox-exporter-config

The above configuration monitors HTTP, HTTPS endpoints and SSL certificate expiration dates. Apply above configuration using kubectl command. When applied it will create a config map for Blackbox configuration and service for exposing Blackbox pods and a deployment for Blackbox pod management.

cloudshell:~/prometheus$ kubectl apply -f blockbox.yaml                                                                                                
configmap/blackbox-exporter-config created
service/blackbox-exporter created
deployment.apps/blackbox-exporter created

Now we need to add below Blackbox scrape configuration to Prometheus server configuration and restart the Prometheus service.

- job_name: 'kube-api-blackbox'
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
   - targets:
      - https://www.google.com
      - http://www.example.com
      - https://prometheus.io
  relabel_configs:
  - source_labels: [__address__]
    target_label: __param_target
  - source_labels: [__param_target]
    target_label: instance
  - target_label: __address__
    replacement: blackbox-exporter.elk.svc.cluster.local:9115

Let’s add dashboards for Blackbox exporter metrics Grafana. Go to Grafana dash boards web site. and search for Blackbox dashboards.

Click on the dashboard that we want to import and copy dashboard id.

Open Grafana console and navigate to Dashbaords section and click on “Import” button to import a dashboard.

Then enter the Blackbox dashboard id that we copied and click on load.

Then select source, since we are using Prometheus to scrape Blackbox endpoint source will be Prometheus. we already created this in Grafana setup.

Then Click on “Import” button. Once dashboards are imported you can see Blackbox metric dashboards on Grafana.

Since we are use Kubernetes deployment for Grafana Pods, whenever Pod gets restarted these dashboards will disappear. Workaround for this issue configuring Grafana to use Kubernetes StatefulSets or add dashbaord configuration json to Grafana dashboard config map, refer this post.

Conclusion

In this quick start demo we have configured Blackbox exporter for Prometheus monitoring on a GKE cluster to probe HTTP, HTTPS endpoints and imported Blackbox dashboards from Grafana dashboards website.

For more on Prometheus monitoring on Kubernetes:

Monitoring Kubernetes Cluster with Prometheus

Prometheus Node Exporter Setup on Kubernetes

Grafana Setup for Prometheus Server on Kubernetes

Alertmanager Setup on Kubernetes for Prometheus Monitoring