Prometheus Node Exporter Setup on Kubernetes

What is Node Exporter?

A Prometheus Exporter is a small application that can fetch monitoring metrics from a target system and expose those metrics through a web URL to allow a Prometheus server to scrape those metrics.

Node exporter exports hardware and OS metrics for *NIX kernels. By default, Kubernetes does not expose node level monitoring metrics. To collect, monitoring metrics for Kubernetes nodes, we need Prometheus node exporter running on all nodes. It collects all the system monitoring metrics and expose them via /metrics web path on port 9100. It needs to be deployed as a Daemon set so that Kubernetes runs it on each and every cluster node.

Deploying Node Exporter on Kubernetes Cluster

Prerequisites:

• A Kubernetes cluster(For information on how to deploy a GKE cluster, see this post.)
• kubectl client library to connect to Kubernetes Cluster
• Admin privileges on Kubernetes Cluster.
• up & and running Prometheus Server (For information on how to deploy a Prometheus Server, see this post.)

Connect to GKE cluster

Copy the following content into a file called node-exporter.yaml. Here we are using monitoring namespace that we have already created while deploying Prometheus Server

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: node-exporter
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      containers:
      - args:
        - --path.sysfs=/host/sys
        - --path.rootfs=/host/root
        - --no-collector.wifi
        - --no-collector.hwmon
        - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/pods/.+)($|/)
        - --collector.netclass.ignored-devices=^(veth.*)$
        name: node-exporter
        image: prom/node-exporter
        ports:
          - containerPort: 9100
            protocol: TCP
        resources:
          limits:
            cpu: 250m
            memory: 180Mi
          requests:
            cpu: 102m
            memory: 180Mi
        volumeMounts:
        - mountPath: /host/sys
          mountPropagation: HostToContainer
          name: sys
          readOnly: true
        - mountPath: /host/root
          mountPropagation: HostToContainer
          name: root
          readOnly: true
      volumes:
      - hostPath:
          path: /sys
        name: sys
      - hostPath:
          path: /
        name: root

Deploy the daemonset using “kubctl apply” command.

List daemonsets that are running monitoring namespace and make sure that their status is Running. There should be a daemon set for each every node. In this demo, we have only one node so there is only one daemonset running.

Now we are going to deploy a Kubernetes service for node-exporter daemon sets. Copy the following yaml content into a file called node-exporter-service.yaml.

kind: Service
apiVersion: v1
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
      app: node-exporter
  ports:
  - name: node-exporter
    protocol: TCP
    port: 9100
    targetPort: 9100

Create a node-exporter service with below command.

List services for node-exporter service.

Now we need to make sure that the node exporter scrape configuration is added to Prometheus server config file. we already added this while deploying Prometheus server in the earlier post. Add the following configuration under scrape_configs sections, if it is not added already.

When the above scrape config is added to Prometheus server config file and reloaded the configuration then from that point onwards Promethes start scraping endpoints names matching with ‘node-exporter’.

Now let’s connect to Prometheus web console using kubeproxy and navigate to Status -> targets section to see node-exporter target.

Connect to Prometheus console using kubectl port-forwarding, see below.

from browser you can access Prometheus console by using url: http://localhost:8080 

See the targets section screen shot below for node-exporter target.

You can query node exporter metrics from Prometheus search console using “node_” prefix.

Conclusion

In this quick start demo we have configured Node Exporter on a GKE cluster and allowed Promethes to scrape node exporter metrics. You can find more information about Node Exporter here.