Grafana Setup for Prometheus Server on Kubernetes

What is Grafana?

Grafana open source software enable users to query, visualize, alert on, and explore metrics, logs, and traces wherever they are stored. Grafana provides tools to turn time-series database (TSDB) data collected by tools like Prometheus, into insightful graphs and visualizations.

In this quick start demo, we are going to deploy Grafana application on a GKE cluster and access Kubernetes monitoring time series data in visual form from Prometheus server.

Grafana Setup on Kubernetes Cluster for Prometheus

Prerequisites:

A Kubernetes cluster(For information on how to deploy a GKE cluster, see this post.)
kubectl client library to connect to Kubernetes Cluster
Admin privileges on Kubernetes Cluster
up & and running Prometheus Server (For information on how to deploy a Prometheus Server, see this post.)

Connect to GKE cluster

Shell

gcloud container clusters get-credentials demo-k8s-cluster

Copy the following content into a file called grafana-dashboards.yaml. Here we are using monitoring namespace that we have already created while deploying Prometheus Server

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboards
  labels:
    name: grafana-dashboards
  namespace: monitoring
data:
  dashboards.yaml: |
    apiVersion: 1
    providers:
    - name: 'Prometheus'
      orgId: 1
      folder: ''
      type: file
      disableDeletion: false
      editable: true
      options:
        path: /etc/grafana/provisioning/dashboards
  podmetrics.json: |-
    {"annotations":{"list":[{"builtIn":1,"datasource":"-- Grafana --","enable":true,"hide":true,"iconColor":"rgba(0, 211, 255, 1)","name":"Annotations & Alerts","type":"dashboard"}]},"description":"Monitors Kubernetes cluster using Prometheus. Shows overall cluster CPU / Memory / Filesystem usage as well as individual pod, containers, systemd services statistics. Uses cAdvisor metrics only.","editable":true,"gnetId":747,"graphTooltip":0,"iteration":1598046210166,"links":[],"panels":[{"collapsed":false,"datasource":null,"gridPos":{"h":1,"w":24,"x":0,"y":0},"id":57,"panels":[],"repeat":null,"title":"Container Restarts","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":6,"w":16,"x":0,"y":1},"hiddenSeries":false,"id":42,"legend":{"alignAsTable":true,"avg":false,"current":false,"max":false,"min":false,"rightSide":true,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pluginVersion":"7.1.4","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"increase(kube_pod_container_status_restarts_total{cluster=\"$Cluster\",namespace=\"$namespace\",pod_template_hash=\"\",pod=\"$Pod\",container=\"$container\",container!=\"\"}[1h])  ","format":"time_series","intervalFactor":2,"legendFormat":"{{ container }}","refId":"A","step":3600}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Container Restarts Increment in 1 hour for $Pod","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"DS_Prometheus","fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":6,"w":4,"x":16,"y":1},"id":44,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"kube_pod_container_status_restarts_total{cluster=\"doug-ts-200-management-prometheus\", container=\"elasticsearch\", instance=\"10.24.0.20:8080\", job=\"kubernetes-service-endpoints\", k8s_app=\"kube-state-metrics\", kubernetes_name=\"kube-state-metrics\", kubernetes_namespace=\"monitoring\", namespace=\"tigera-elasticsearch\", pod=\"tigera-secure-es-9bd8b7baa502d70d-0\"}","targets":[{"expr":"kube_pod_container_status_restarts_total{pod_template_hash=\"\",pod=\"$Pod\", container=\"$container\",container!=\"\",cluster=\"$Cluster\"} ","format":"time_series","intervalFactor":1,"refId":"A","step":21600}],"thresholds":"","title":"Total Container ( $container ) restarts","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"DS_Prometheus","fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":6,"w":4,"x":20,"y":1},"id":55,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"kube_pod_container_status_running{cluster=\"doug-ts-200-management-prometheus\", container=\"elasticsearch\", instance=\"10.24.0.20:8080\", job=\"kubernetes-service-endpoints\", k8s_app=\"kube-state-metrics\", kubernetes_name=\"kube-state-metrics\", kubernetes_namespace=\"monitoring\", namespace=\"tigera-elasticsearch\", pod=\"tigera-secure-es-9bd8b7baa502d70d-0\"}","targets":[{"expr":"kube_pod_container_status_running{pod_template_hash=\"\",namespace=\"$namespace\", pod=\"$Pod\",container=\"$container\",container!=\"\",cluster=\"$Cluster\"}","format":"time_series","intervalFactor":2,"refId":"A","step":43200}],"thresholds":"","title":"Container ( $container ) Status","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"Running","value":"1"},{"op":"=","text":"Not Running","value":"0"}],"valueName":"current"},{"collapsed":false,"datasource":null,"gridPos":{"h":1,"w":24,"x":0,"y":7},"id":58,"panels":[],"repeat":null,"title":"Container Errors","type":"row"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":5,"w":6,"x":0,"y":8},"id":41,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum(kube_pod_container_status_last_terminated_reason{namespace=\"$namespace\",container=\"$container\",container!=\"\",cluster=\"$Cluster\",pod_template_hash=\"\",reason=\"Error\"})","format":"time_series","intervalFactor":2,"refId":"A","step":43200}],"thresholds":"","title":"Container ( $container ) Errors","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":5,"w":6,"x":6,"y":8},"id":43,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum(kube_pod_container_status_last_terminated_reason{namespace=\"$namespace\",container=\"$container\",container!=\"\",cluster=\"$Cluster\",pod_template_hash=\"\",reason=\"OOMKilled\"})","format":"time_series","intervalFactor":1,"refId":"A","step":21600}],"thresholds":"","title":"Container ( $container ) OOM Kills","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"DS_Prometheus","fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":5,"w":6,"x":12,"y":8},"id":45,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum(kube_pod_container_status_last_terminated_reason{namespace=\"$namespace\",container=\"$container\",container!=\"\",cluster=\"$Cluster\",pod_template_hash=\"\",reason=\"ContainerCannotRun\"})","format":"time_series","intervalFactor":2,"refId":"A","step":43200}],"thresholds":"","title":"Container ( $container ) ContainerCannotRun","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"DS_Prometheus","fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":5,"w":6,"x":18,"y":8},"id":46,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"kube_pod_container_status_last_terminated_reason{reason!=\"\", namespace=\"$namespace\",container=\"$container\",container!=\"\",cluster=\"$Cluster\",pod_template_hash=\"\",pod=\"$Pod\"} > 0","format":"time_series","intervalFactor":2,"legendFormat":"{{ reason }}","refId":"A","step":43200}],"thresholds":"","title":"Last Termination Reason for $container","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"name"},{"collapsed":false,"datasource":null,"gridPos":{"h":1,"w":24,"x":0,"y":13},"id":59,"panels":[],"repeat":null,"title":"Dashboard Row","type":"row"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"DS_Prometheus","fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":24,"x":0,"y":14},"height":"1px","id":56,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"20%","prefix":"","prefixFontSize":"20%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"kube_pod_container_status_waiting_reason{reason!=\"\", namespace=\"$namespace\",pod_template_hash=\"\",pod=\"$Pod\",cluster=\"$Cluster\"} > 0","format":"time_series","intervalFactor":2,"legendFormat":" {{ container }}  {{ reason }}","refId":"A","step":43200}],"thresholds":"","title":"Pod ($Pod ) Waiting Reason","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"Not Waiting","value":"null"}],"valueName":"name"},{"collapsed":false,"datasource":null,"gridPos":{"h":1,"w":24,"x":0,"y":17},"id":60,"panels":[],"repeat":null,"title":"Total usage","type":"row"},{"cacheTimeout":null,"colorBackground":false,"colorValue":true,"colors":["rgba(50, 172, 45, 0.97)","rgba(237, 129, 40, 0.89)","rgba(245, 54, 54, 0.9)"],"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"percent","gauge":{"maxValue":100,"minValue":0,"show":true,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":5,"w":12,"x":0,"y":18},"height":"180px","id":4,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (container_memory_working_set_bytes{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\"}) / sum (machine_memory_bytes{kubernetes_io_hostname=~\"^$Node$\"}) * 100","format":"time_series","interval":"10s","intervalFactor":1,"refId":"A","step":162}],"thresholds":"65, 90","title":"Pod memory usage","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"cacheTimeout":null,"colorBackground":false,"colorValue":true,"colors":["rgba(50, 172, 45, 0.97)","rgba(237, 129, 40, 0.89)","rgba(245, 54, 54, 0.9)"],"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"percent","gauge":{"maxValue":100,"minValue":0,"show":true,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":5,"w":12,"x":12,"y":18},"height":"180px","id":6,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (rate (container_cpu_usage_seconds_total{pod=~\"^$Pod$\",cluster=\"$Cluster\"}[1m])) / sum (machine_cpu_cores{kubernetes_io_hostname=~\"^$Node$\",cluster=\"$Cluster\"}) * 100","format":"time_series","interval":"10s","intervalFactor":1,"refId":"A","step":162}],"thresholds":"65, 90","title":"Pod CPU usage","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"bytes","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":6,"x":0,"y":23},"height":"100px","id":36,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum(container_memory_working_set_bytes{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\"})","format":"time_series","intervalFactor":2,"refId":"A","step":43200}],"thresholds":"","title":"Used","type":"singlestat","valueFontSize":"50%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"bytes","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":6,"x":6,"y":23},"height":"100px","id":37,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (machine_memory_bytes{kubernetes_io_hostname=~\"^$Node$\",cluster=\"$Cluster\"})","format":"time_series","intervalFactor":2,"refId":"A","step":43200}],"thresholds":"","title":"Total","type":"singlestat","valueFontSize":"50%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":6,"x":12,"y":23},"height":"100px","id":38,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (rate (container_cpu_usage_seconds_total{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"}[1m]))","format":"time_series","intervalFactor":2,"refId":"A","step":43200}],"thresholds":"","title":"Used","type":"singlestat","valueFontSize":"50%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":6,"x":18,"y":23},"height":"100px","id":39,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (rate (container_cpu_usage_seconds_total{kubernetes_io_hostname=\"$Node\",cluster=\"$Cluster\"}[1m]))","format":"time_series","intervalFactor":2,"refId":"A","step":43200}],"thresholds":"","title":"Total","type":"singlestat","valueFontSize":"50%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(50, 172, 45, 0.97)","rgba(237, 129, 40, 0.89)","rgba(245, 54, 54, 0.9)"],"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"bytes","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":24,"x":0,"y":26},"height":"1px","id":9,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"20%","prefix":"","prefixFontSize":"20%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (container_memory_working_set_bytes{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\"})","interval":"10s","intervalFactor":1,"refId":"A","step":162}],"thresholds":"","title":"Pod memory working bytes","type":"singlestat","valueFontSize":"50%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":6,"w":8,"x":0,"y":29},"hiddenSeries":false,"id":73,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"nullPointMode":"null","percentage":false,"pluginVersion":"7.1.4","pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"kubelet_volume_stats_used_bytes{cluster=\"$Cluster\",namespace=\"$namespace\"} / (1024*1024*1024)","interval":"","legendFormat":"{{ persistentvolumeclaim }}","refId":"A"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Persistent Volume Usage (GB)","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"$$hashKey":"object:521","format":"decgbytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"$$hashKey":"object:522","format":"decgbytes","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":6,"w":8,"x":8,"y":29},"hiddenSeries":false,"id":75,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"nullPointMode":"null","percentage":false,"pluginVersion":"7.1.4","pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"kubelet_volume_stats_used_bytes{cluster=\"$Cluster\",namespace=\"$namespace\"} / kubelet_volume_stats_capacity_bytes{cluster=\"$Cluster\",namespace=\"$namespace\"} * 100","interval":"","legendFormat":"{{ persistentvolumeclaim }}","refId":"A"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Persistent Volume Usage %","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"$$hashKey":"object:521","format":"percent","label":null,"logBase":1,"max":null,"min":null,"show":true},{"$$hashKey":"object:522","format":"decgbytes","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":6,"w":8,"x":16,"y":29},"hiddenSeries":false,"id":74,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"nullPointMode":"null","percentage":false,"pluginVersion":"7.1.4","pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"kubelet_volume_stats_capacity_bytes{cluster=\"$Cluster\",namespace=\"$namespace\"} / (1024*1024*1024)","interval":"","legendFormat":"{{ persistentvolumeclaim }}","refId":"A"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Persistent Volume Capacity","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"$$hashKey":"object:521","format":"decgbytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"$$hashKey":"object:522","format":"decgbytes","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"datasource":null,"gridPos":{"h":1,"w":24,"x":0,"y":35},"id":61,"panels":[],"repeat":null,"title":"Pods CPU usage","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":3,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":0,"fillGradient":0,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":36},"height":"","hiddenSeries":false,"id":17,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":true,"show":true,"sort":null,"sortDesc":null,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","percentage":false,"pluginVersion":"7.1.4","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum (rate (container_cpu_usage_seconds_total{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"}[1m])) by (pod)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"{{ $Pod }}","metric":"container_cpu","refId":"A","step":162}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Pods CPU usage","tooltip":{"msResolution":true,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"none","label":"cores","logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"datasource":null,"gridPos":{"h":1,"w":24,"x":0,"y":43},"id":62,"panels":[],"repeat":null,"title":"All processes CPU usage","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":3,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":0,"fillGradient":0,"grid":{},"gridPos":{"h":10,"w":24,"x":0,"y":44},"hiddenSeries":false,"id":20,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":true,"show":true,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","percentage":false,"pluginVersion":"7.1.4","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum (rate (container_cpu_usage_seconds_total{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"}[1m])) by (id)","format":"time_series","hide":false,"interval":"10s","intervalFactor":1,"legendFormat":"{{ id }}","metric":"container_cpu","refId":"A","step":162}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"All processes CPU usage","tooltip":{"msResolution":true,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"percentunit","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"datasource":null,"gridPos":{"h":1,"w":24,"x":0,"y":54},"id":63,"panels":[],"repeat":null,"title":"Pods memory usage","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":0,"fillGradient":0,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":55},"hiddenSeries":false,"id":25,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":true,"show":true,"sideWidth":200,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","percentage":false,"pluginVersion":"7.1.4","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum (container_memory_working_set_bytes{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\"}) by (pod)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"{{ pod }}","metric":"container_memory_usage:sort_desc","refId":"A","step":162}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Pods memory usage","tooltip":{"msResolution":false,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"datasource":null,"gridPos":{"h":1,"w":24,"x":0,"y":62},"id":64,"panels":[],"repeat":null,"title":"All processes memory usage","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":0,"fillGradient":0,"grid":{},"gridPos":{"h":9,"w":24,"x":0,"y":63},"hiddenSeries":false,"id":28,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":true,"show":true,"sideWidth":200,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","percentage":false,"pluginVersion":"7.1.4","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum (container_memory_working_set_bytes{id!=\"/\",pod=~\"^$Pod$\",cluster=\"$Cluster\"}) by (id)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"{{ id }}","metric":"container_memory_usage:sort_desc","refId":"A","step":162}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"All processes memory usage","tooltip":{"msResolution":false,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"datasource":null,"gridPos":{"h":1,"w":24,"x":0,"y":72},"id":65,"panels":[],"repeat":null,"title":"Dashboard Row","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":7,"w":12,"x":0,"y":73},"hiddenSeries":false,"id":47,"legend":{"alignAsTable":true,"avg":true,"current":false,"max":true,"min":false,"rightSide":true,"show":true,"total":false,"values":true},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pluginVersion":"7.1.4","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(irate(container_cpu_usage_seconds_total{container=\"$container\",container!=\"\",cluster=\"$Cluster\", namespace=\"$namespace\"}[30s])) by (namespace,container) / sum(container_spec_cpu_quota{container=\"$container\",container!=\"\",cluster=\"$Cluster\", namespace=\"$namespace\"} / container_spec_cpu_period{container=\"$container\",container!=\"\",cluster=\"$Cluster\", namespace=\"$namespace\"}) by (namespace,container)","format":"time_series","intervalFactor":1,"legendFormat":"actual","refId":"A","step":1800}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Cpu usage (relative to limit)","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"percentunit","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":7,"w":12,"x":12,"y":73},"hiddenSeries":false,"id":48,"legend":{"alignAsTable":true,"avg":true,"current":false,"max":true,"min":false,"rightSide":true,"show":true,"total":false,"values":true},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pluginVersion":"7.1.4","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(container_memory_working_set_bytes{container=\"$container\",image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\"}) by (container) / sum(container_spec_memory_limit_bytes{container=\"$container\",image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\"}) by (container)","format":"time_series","intervalFactor":2,"legendFormat":"actual","refId":"A","step":3600}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Memory usage (relative to limit)","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"percentunit","label":null,"logBase":1,"max":null,"min":"0","show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"datasource":null,"gridPos":{"h":1,"w":24,"x":0,"y":80},"id":66,"panels":[],"repeat":null,"title":"Dashboard Row","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":7,"w":12,"x":0,"y":81},"hiddenSeries":false,"id":49,"legend":{"alignAsTable":true,"avg":true,"current":false,"max":true,"min":false,"rightSide":true,"show":true,"total":false,"values":true},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pluginVersion":"7.1.4","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(irate(container_cpu_usage_seconds_total{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"}[30s])) by (id,pod)","format":"time_series","intervalFactor":2,"legendFormat":"{{ pod }}","refId":"A","step":3600},{"expr":"sum(container_spec_cpu_quota{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"} / container_spec_cpu_period{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"}) by (namespace,container) / count(container_memory_usage_bytes{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"}) by (namespace,container) ","format":"time_series","intervalFactor":2,"legendFormat":"limit","refId":"B","step":3600},{"expr":"sum(container_spec_cpu_shares{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"} / 1024) by (namespace,container) / count(container_spec_cpu_shares{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"}) by (namespace,container) ","format":"time_series","intervalFactor":2,"legendFormat":"request","refId":"C","step":3600}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Cpu usage (per pod)","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":7,"w":12,"x":12,"y":81},"hiddenSeries":false,"id":50,"legend":{"alignAsTable":true,"avg":true,"current":false,"max":true,"min":false,"rightSide":true,"show":true,"total":false,"values":true},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pluginVersion":"7.1.4","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(container_memory_working_set_bytes{container=\"$container\",image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\"}) by (pod)","format":"time_series","intervalFactor":2,"legendFormat":"{{pod}}","refId":"A","step":3600},{"expr":"sum(container_spec_memory_limit_bytes{container=\"$container\",image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\"}) by (container) / count(container_memory_usage_bytes{container=\"$container\",image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\"}) by (container)","format":"time_series","intervalFactor":2,"legendFormat":"limit","refId":"B","step":3600},{"expr":"sum(kube_pod_container_resource_requests_memory_bytes{container=\"$container\",container!=\"\",cluster=\"$Cluster\", namespace=\"$namespace\"}) by (namespace,container) / count(kube_pod_container_resource_requests_memory_bytes{container=\"$container\",container!=\"\",cluster=\"$Cluster\", namespace=\"$namespace\"}) by (namespace,container)","format":"time_series","intervalFactor":1,"legendFormat":"request","refId":"C"},{"refId":"D"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Memory usage (per pod)","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"decbytes","label":null,"logBase":1,"max":null,"min":"0","show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"datasource":null,"gridPos":{"h":1,"w":24,"x":0,"y":88},"id":67,"panels":[],"repeat":null,"title":"Dashboard Row","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":7,"w":12,"x":0,"y":89},"hiddenSeries":false,"id":51,"legend":{"alignAsTable":true,"avg":true,"current":false,"max":true,"min":false,"rightSide":true,"show":true,"total":false,"values":true},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pluginVersion":"7.1.4","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(irate(container_cpu_usage_seconds_total{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"}[30s])) by (namespace,container) / count(container_memory_usage_bytes{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"}) by (namespace,container) ","format":"time_series","intervalFactor":2,"legendFormat":"actual","refId":"A","step":3600},{"expr":"sum(container_spec_cpu_quota{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"} / container_spec_cpu_period{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"}) by (namespace,container) / count(container_memory_usage_bytes{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"}) by (namespace,container) ","format":"time_series","intervalFactor":2,"legendFormat":"limit","refId":"B","step":3600},{"expr":"sum(container_spec_cpu_shares{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"} / 1024) by (namespace,container) / count(container_spec_cpu_shares{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"}) by (namespace,container) ","format":"time_series","intervalFactor":2,"legendFormat":"request","refId":"C","step":3600}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Cpu usage (avg per pod)","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":"0","show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":7,"w":12,"x":12,"y":89},"hiddenSeries":false,"id":52,"legend":{"alignAsTable":true,"avg":true,"current":false,"max":true,"min":false,"rightSide":true,"show":true,"total":false,"values":true},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pluginVersion":"7.1.4","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(container_memory_working_set_bytes{container=\"$container\",image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\"}) by (namespace,container) / count(container_memory_working_set_bytes{container=\"$container\",image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\"})by (namespace,container) ","format":"time_series","intervalFactor":2,"legendFormat":"actual","refId":"A","step":3600},{"expr":"sum(container_spec_memory_limit_bytes{container=\"$container\",container!=\"\",cluster=\"$Cluster\", namespace=\"$namespace\"}) by (namespace,container) / count(container_memory_usage_bytes{container=\"$container\",container!=\"\",cluster=\"$Cluster\", namespace=\"$namespace\"}) by (namespace,container) ","format":"time_series","intervalFactor":2,"legendFormat":"limit","refId":"B","step":3600},{"expr":"sum(kube_pod_container_resource_requests_memory_bytes{container=\"$container\",container!=\"\",cluster=\"$Cluster\", namespace=\"$namespace\"}) by (namespace,container) / count(kube_pod_container_resource_requests_memory_bytes{container=\"$container\",container!=\"\",cluster=\"$Cluster\", namespace=\"$namespace\"}) by (namespace,container)","format":"time_series","intervalFactor":1,"legendFormat":"request","refId":"C"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Memory usage (avg per pod)","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":null,"logBase":1,"max":null,"min":"0","show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"datasource":null,"gridPos":{"h":1,"w":24,"x":0,"y":96},"id":68,"panels":[],"repeat":null,"title":"Dashboard Row","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":7,"w":12,"x":0,"y":97},"hiddenSeries":false,"id":53,"legend":{"alignAsTable":true,"avg":true,"current":false,"max":true,"min":false,"rightSide":true,"show":true,"total":false,"values":true},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pluginVersion":"7.1.4","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(irate(container_cpu_usage_seconds_total{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"}[30s])) by (namespace,container)","format":"time_series","intervalFactor":2,"legendFormat":"actual","refId":"A","step":3600},{"expr":"sum(container_spec_cpu_quota{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"} / container_spec_cpu_period{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"}) by (namespace,container)","format":"time_series","intervalFactor":2,"legendFormat":"limit","refId":"B","step":3600},{"expr":"sum(container_spec_cpu_shares{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\",container!=\"POD\"} / 1024) by (namespace,container) ","format":"time_series","intervalFactor":2,"legendFormat":"request","refId":"C","step":3600}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Cpu usage (total)","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":"cores","logBase":1,"max":null,"min":"0","show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":7,"w":12,"x":12,"y":97},"hiddenSeries":false,"id":54,"legend":{"alignAsTable":true,"avg":true,"current":false,"max":true,"min":false,"rightSide":true,"show":true,"total":false,"values":true},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pluginVersion":"7.1.4","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(container_memory_working_set_bytes{container=\"$container\",image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\"})","format":"time_series","intervalFactor":2,"legendFormat":"actual","refId":"A","step":3600},{"expr":"sum(container_spec_memory_limit_bytes{container=\"$container\",container!=\"\",cluster=\"$Cluster\", namespace=\"$namespace\"}) by (namespace,container) / count(container_spec_memory_limit_bytes{container=\"$container\",container!=\"\",cluster=\"$Cluster\", namespace=\"$namespace\"}) by (namespace,container)","format":"time_series","intervalFactor":2,"legendFormat":"limit","refId":"B","step":3600},{"expr":"sum(kube_pod_container_resource_requests_memory_bytes{container=\"$container\",container!=\"\",cluster=\"$Cluster\", namespace=\"$namespace\"}) by (namespace,container) / count(kube_pod_container_resource_requests_memory_bytes{container=\"$container\",container!=\"\",cluster=\"$Cluster\", namespace=\"$namespace\"}) by (namespace,container)","format":"time_series","intervalFactor":1,"legendFormat":"request","refId":"C"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Memory usage (total)","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":null,"logBase":1,"max":null,"min":"0","show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"datasource":null,"gridPos":{"h":1,"w":24,"x":0,"y":104},"id":69,"panels":[],"repeat":null,"title":"Network I/O pressure","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"grid":{},"gridPos":{"h":5,"w":24,"x":0,"y":105},"height":"200px","hiddenSeries":false,"id":32,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":true,"show":true,"sideWidth":200,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","percentage":false,"pluginVersion":"7.1.4","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum (rate (container_network_receive_bytes_total{pod=~\"^$Pod$\",cluster=\"$Cluster\"}[1m]))","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"Received","metric":"network","refId":"A","step":162},{"expr":"- sum (rate (container_network_transmit_bytes_total{pod=~\"^$Pod$\",cluster=\"$Cluster\"}[1m]))","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"Sent","metric":"network","refId":"B","step":162}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Network I/O pressure for $Pod","tooltip":{"msResolution":false,"shared":true,"sort":0,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"Bps","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"Bps","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"datasource":null,"gridPos":{"h":1,"w":24,"x":0,"y":110},"id":70,"panels":[],"repeat":null,"title":"Pods network I/O","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":111},"hiddenSeries":false,"id":16,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":true,"show":true,"sideWidth":200,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","percentage":false,"pluginVersion":"7.1.4","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum (rate (container_network_receive_bytes_total{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\"}[1m])) by (pod)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"-> {{ pod }}","metric":"network","refId":"A","step":162},{"expr":"- sum (rate (container_network_transmit_bytes_total{image!=\"\",name=~\"^k8s_.*\",pod=\"$Pod\",pod_template_hash=\"\",cluster=\"$Cluster\"}[1m])) by (pod)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"<- {{ pod }}","metric":"network","refId":"B","step":162}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Pods network I/O","tooltip":{"msResolution":false,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"Bps","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"datasource":null,"gridPos":{"h":1,"w":24,"x":0,"y":118},"id":71,"panels":[],"repeat":null,"title":"All processes network I/O","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"grid":{},"gridPos":{"h":10,"w":24,"x":0,"y":119},"hiddenSeries":false,"id":29,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":true,"show":true,"sideWidth":200,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","percentage":false,"pluginVersion":"7.1.4","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum (rate (container_network_receive_bytes_total{id!=\"/\",pod=~\"^$Pod$\",cluster=\"$Cluster\"}[1m])) by (id)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"-> {{ id }}","metric":"network","refId":"A","step":162},{"expr":"- sum (rate (container_network_transmit_bytes_total{id!=\"/\",pod=~\"^$Pod$\",cluster=\"$Cluster\"}[1m])) by (id)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"<- {{ id }}","metric":"network","refId":"B","step":162}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"All processes network I/O","tooltip":{"msResolution":false,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"Bps","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}}],"refresh":"5s","schemaVersion":26,"style":"dark","tags":["kubernetes"],"templating":{"list":[{"allValue":null,"current":{"selected":false,"text":"doug-ts-200-management-prometheus","value":"doug-ts-200-management-prometheus"},"datasource":"DS_Prometheus","definition":"","hide":0,"includeAll":false,"label":null,"multi":false,"name":"Cluster","options":[],"query":"label_values(machine_memory_bytes{},cluster)","refresh":1,"regex":"","skipUrlSync":false,"sort":1,"tagValuesQuery":"","tags":[],"tagsQuery":"","type":"query","useTags":false},{"allValue":".*","current":{"selected":false,"text":"gke-doug-ts-200-manageme-default-pool-20befdf4-1xhx","value":"gke-doug-ts-200-manageme-default-pool-20befdf4-1xhx"},"datasource":"DS_Prometheus","definition":"label_values(machine_memory_bytes{cluster=~\"$Cluster\"},kubernetes_io_hostname)","hide":2,"includeAll":false,"label":null,"multi":false,"name":"Node","options":[],"query":"label_values(machine_memory_bytes{cluster=~\"$Cluster\"},kubernetes_io_hostname)","refresh":1,"regex":"","skipUrlSync":false,"sort":0,"tagValuesQuery":"","tags":[],"tagsQuery":"","type":"query","useTags":false},{"allValue":null,"current":{"selected":false,"text":"tigera-elasticsearch","value":"tigera-elasticsearch"},"datasource":"DS_Prometheus","definition":"label_values(container_memory_usage_bytes{cluster=~\"$Cluster\",namespace=~\".+\",container!=\"POD\"},namespace)","hide":0,"includeAll":false,"label":null,"multi":false,"name":"namespace","options":[],"query":"label_values(container_memory_usage_bytes{cluster=~\"$Cluster\",namespace=~\".+\",container!=\"POD\"},namespace)","refresh":2,"regex":"","skipUrlSync":false,"sort":1,"tagValuesQuery":"","tags":[],"tagsQuery":"","type":"query","useTags":false},{"allValue":".*","current":{"selected":false,"text":"tigera-secure-es-9bd8b7baa502d70d-0","value":"tigera-secure-es-9bd8b7baa502d70d-0"},"datasource":"DS_Prometheus","definition":"label_values(container_memory_usage_bytes{cluster=~\"$Cluster\",namespace=~\"$namespace\",container!=\"POD\"},pod)","hide":0,"includeAll":false,"label":"Pod","multi":false,"name":"Pod","options":[],"query":"label_values(container_memory_usage_bytes{cluster=~\"$Cluster\",namespace=~\"$namespace\",container!=\"POD\"},pod)","refresh":1,"regex":"","skipUrlSync":false,"sort":0,"tagValuesQuery":"","tags":[],"tagsQuery":"","type":"query","useTags":false},{"allValue":null,"current":{"selected":false,"text":"elasticsearch","value":"elasticsearch"},"datasource":"DS_Prometheus","definition":"label_values(container_memory_usage_bytes{cluster=~\"$Cluster\",namespace=~\"$namespace\",container!=\"POD\",pod=\"$Pod\"},container)","hide":0,"includeAll":false,"label":null,"multi":false,"name":"container","options":[],"query":"label_values(container_memory_usage_bytes{cluster=~\"$Cluster\",namespace=~\"$namespace\",container!=\"POD\",pod=\"$Pod\"},container)","refresh":2,"regex":"","skipUrlSync":false,"sort":1,"tagValuesQuery":"","tags":[],"tagsQuery":"","type":"query","useTags":false}]},"time":{"from":"now-5m","to":"now"},"timepicker":{"refresh_intervals":["10s","30s","1m","5m","15m","30m","1h","2h","1d"],"time_options":["5m","15m","1h","6h","12h","24h","2d","7d","30d"]},"timezone":"browser","title":"Kubernetes Pod Metrics","uid":"gRGG1CBik","version":1}
  deploymentmetrics.json: |-
    {"annotations":{"list":[{"builtIn":1,"datasource":"-- Grafana --","enable":true,"hide":true,"iconColor":"rgba(0, 211, 255, 1)","name":"Annotations & Alerts","type":"dashboard"}]},"description":"Monitors Kubernetes deployments in cluster using Prometheus. Shows overall cluster CPU / Memory of deployments, replicas in each deployment. Uses Kube state metrics and cAdvisor metrics ","editable":true,"gnetId":741,"graphTooltip":0,"iteration":1592255799386,"links":[],"panels":[{"cacheTimeout":null,"colorBackground":false,"colorValue":true,"colors":["rgba(50, 172, 45, 0.97)","rgba(237, 129, 40, 0.89)","rgba(245, 54, 54, 0.9)"],"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"percent","gauge":{"maxValue":100,"minValue":0,"show":true,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":5,"w":8,"x":0,"y":0},"height":"180px","id":4,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (container_memory_working_set_bytes{cluster=~\"^$Cluster$\",container!=\"\",id!=\"/\",pod=~\"^$Deployment.*$\",container!=\"POD\"}) / sum (machine_memory_bytes{cluster=~\"^$Cluster$\",kubernetes_io_hostname=~\"^$Node$\"}) * 100","format":"time_series","interval":"10s","intervalFactor":1,"refId":"A","step":900}],"thresholds":"65, 90","title":"Deployment memory usage","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"cacheTimeout":null,"colorBackground":false,"colorValue":true,"colors":["rgba(50, 172, 45, 0.97)","rgba(237, 129, 40, 0.89)","rgba(245, 54, 54, 0.9)"],"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"percent","gauge":{"maxValue":100,"minValue":0,"show":true,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":5,"w":8,"x":8,"y":0},"height":"180px","id":6,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (rate (container_cpu_usage_seconds_total{cluster=~\"^$Cluster$\",pod=~\"^$Deployment.*$\",container!=\"POD\"}[1m])) / sum (machine_cpu_cores{cluster=~\"^$Cluster$\",kubernetes_io_hostname=~\"^$Node$\"}) * 100","format":"time_series","interval":"10s","intervalFactor":1,"refId":"A","step":900}],"thresholds":"65, 90","title":"Deployment CPU usage","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"percent","gauge":{"maxValue":100,"minValue":0,"show":true,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":5,"w":8,"x":16,"y":0},"height":"180px","id":37,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum(kube_deployment_status_replicas_available{cluster=~\"^$Cluster$\",deployment=~\"^$Deployment$\",pod_template_hash=\"\"}) / sum(kube_deployment_status_replicas{cluster=~\"^$Cluster$\",deployment=~\"^$Deployment$\",pod_template_hash=\"\"}) * 100","format":"time_series","interval":"","intervalFactor":2,"legendFormat":"","refId":"A","step":1800}],"thresholds":"","title":"Replicas","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"bytes","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":4,"x":0,"y":5},"height":"100px","id":38,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (container_memory_working_set_bytes{cluster=~\"^$Cluster$\",container!=\"\",id!=\"/\",pod=~\"^$Deployment.*$\",container!=\"POD\"})","format":"time_series","interval":"","intervalFactor":2,"legendFormat":"","refId":"A","step":1800}],"thresholds":"","title":"Used","type":"singlestat","valueFontSize":"50%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"bytes","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":4,"x":4,"y":5},"height":"100px","id":39,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (container_memory_working_set_bytes{cluster=~\"^$Cluster$\",kubernetes_io_hostname=~\"^$Node.*$\"})","format":"time_series","interval":"","intervalFactor":2,"legendFormat":"","refId":"A","step":1800}],"thresholds":"","title":"Total","type":"singlestat","valueFontSize":"50%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":4,"x":8,"y":5},"height":"100px","id":40,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":" cores","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (rate (container_cpu_usage_seconds_total{cluster=~\"^$Cluster$\",pod=~\"^$Deployment.*$\",container!=\"POD\"}[1m]))","format":"time_series","interval":"","intervalFactor":2,"legendFormat":"","refId":"A","step":1800}],"thresholds":"","title":"Used","type":"singlestat","valueFontSize":"50%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":4,"x":12,"y":5},"height":"100px","id":41,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":" cores","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (machine_cpu_cores{cluster=~\"^$Cluster$\",kubernetes_io_hostname=~\"^$Node$\"})","format":"time_series","interval":"","intervalFactor":2,"legendFormat":"","refId":"A","step":1800}],"thresholds":"","title":"Total","type":"singlestat","valueFontSize":"50%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":4,"x":16,"y":5},"height":"100px","id":42,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum(kube_deployment_status_replicas_available{cluster=~\"^$Cluster$\",deployment=~\"^$Deployment$\",pod_template_hash=\"\"})","format":"time_series","interval":"","intervalFactor":2,"legendFormat":"","refId":"A","step":1800}],"thresholds":"","title":"Available","type":"singlestat","valueFontSize":"50%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":4,"x":20,"y":5},"height":"100px","id":43,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum(kube_deployment_status_replicas{cluster=~\"^$Cluster$\",deployment=~\"^$Deployment$\",pod_template_hash=\"\"})","format":"time_series","interval":"","intervalFactor":2,"legendFormat":"","refId":"A","step":1800}],"thresholds":"","title":"Total","type":"singlestat","valueFontSize":"50%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":7,"w":24,"x":0,"y":8},"hiddenSeries":false,"id":44,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(kube_deployment_status_replicas{cluster=~\"^$Cluster$\",pod_template_hash=\"\",deployment=\"$Deployment\"}) by (deployment)","format":"time_series","intervalFactor":1,"legendFormat":"{{ deployment }}","metric":"kube_deployment_status_replicas","refId":"A","step":60}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Current Running Replicas","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":3,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":0,"fillGradient":0,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":15},"height":"","hiddenSeries":false,"id":17,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":true,"show":true,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum (rate (container_cpu_usage_seconds_total{cluster=~\"^$Cluster$\",container!=\"\",image!=\"\",name=~\"^k8s_.*\",pod=~\"^$Deployment.*$\",container!=\"POD\"}[1m])) by (pod)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"{{ pod }}","metric":"container_cpu","refId":"A","step":60}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Deployment CPU usage","tooltip":{"msResolution":true,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"none","label":"cores","logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":3,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":0,"fillGradient":0,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":22},"height":"","hiddenSeries":false,"id":24,"legend":{"alignAsTable":true,"avg":true,"current":true,"hideEmpty":false,"hideZero":false,"max":false,"min":false,"rightSide":true,"show":true,"sideWidth":null,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum (rate (container_cpu_usage_seconds_total{cluster=~\"^$Cluster$\",image!=\"\",name=~\"^k8s_.*\",container!=\"POD\",pod=~\"^$Deployment.*$\",container!=\"POD\"}[1m])) by (container, pod)","format":"time_series","hide":false,"interval":"10s","intervalFactor":1,"legendFormat":"pod: {{ pod }} | {{ container }}","metric":"container_cpu","refId":"A","step":60}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Containers CPU usage","tooltip":{"msResolution":true,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"none","label":"cores","logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":3,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":0,"fillGradient":0,"grid":{},"gridPos":{"h":13,"w":24,"x":0,"y":29},"hiddenSeries":false,"id":20,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":false,"show":true,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum (rate (container_cpu_usage_seconds_total{cluster=~\"^$Cluster$\",id!=\"/\",pod=~\"^$Deployment.*$\",container!=\"POD\",container!=\"\"}[1m])) by (container)","format":"time_series","hide":false,"interval":"10s","intervalFactor":1,"legendFormat":"{{ container  }}","metric":"container_cpu","refId":"A","step":60}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Average no of cores used per min","tooltip":{"msResolution":true,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"none","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":0,"fillGradient":0,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":42},"hiddenSeries":false,"id":25,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":true,"show":true,"sideWidth":200,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum (container_memory_working_set_bytes{cluster=~\"^$Cluster$\",image!=\"\",name=~\"^k8s_.*\",pod=~\"^$Deployment.*$\",container!=\"POD\"}) by (pod)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"{{ pod }}","metric":"container_memory_usage:sort_desc","refId":"A","step":60}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Deployment memory usage","tooltip":{"msResolution":false,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":0,"fillGradient":0,"grid":{},"gridPos":{"h":13,"w":24,"x":0,"y":49},"hiddenSeries":false,"id":28,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":false,"show":true,"sideWidth":200,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum (container_memory_working_set_bytes{cluster=~\"^$Cluster$\",container!=\"\",id!=\"/\",pod=~\"^$Deployment.*$\",container!=\"POD\"}) by (id)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"{{ id }}","metric":"container_memory_usage:sort_desc","refId":"A","step":60}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"All processes memory usage","tooltip":{"msResolution":false,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"fillGradient":0,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":62},"hiddenSeries":false,"id":16,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":true,"show":true,"sideWidth":200,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum (rate (container_network_receive_bytes_total{cluster=~\"^$Cluster$\",container!=\"\",image!=\"\",name=~\"^k8s_.*\",pod=~\"^$Deployment.*$\",container=\"POD\",}[1m])) by (pod)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"-> {{ pod }}","metric":"network","refId":"A","step":60},{"expr":"- sum (rate (container_network_transmit_bytes_total{cluster=~\"^$Cluster$\",container!=\"\",image!=\"\",name=~\"^k8s_.*\",pod=~\"^$Deployment.*$\",container=\"POD\",}[1m])) by (pod)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"<- {{ pod }}","metric":"network","refId":"B","step":60}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Deployment network I/O","tooltip":{"msResolution":false,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"Bps","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"fillGradient":0,"grid":{},"gridPos":{"h":13,"w":24,"x":0,"y":69},"hiddenSeries":false,"id":29,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":false,"show":true,"sideWidth":200,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum (rate (container_network_receive_bytes_total{cluster=~\"^$Cluster$\",container!=\"\",id!=\"/\",pod=~\"^$Deployment.*$\",container=\"POD\"}[1m])) by (id)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"-> {{ id }}","metric":"network","refId":"A","step":60},{"expr":"- sum (rate (container_network_transmit_bytes_total{cluster=~\"^$Cluster$\",container!=\"\",id!=\"/\",pod=~\"^$Deployment.*$\",container=\"POD\"}[1m])) by (id)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"<- {{ id }}","metric":"network","refId":"B","step":60}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"All processes network I/O","tooltip":{"msResolution":false,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"Bps","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}}],"refresh":false,"schemaVersion":25,"style":"dark","tags":["kubernetes"],"templating":{"list":[{"allValue":null,"current":{"selected":false,"text":"vt-ts-150-prometheus","value":"vt-ts-150-prometheus"},"datasource":"DS_Prometheus","definition":"","hide":0,"includeAll":false,"label":null,"multi":false,"name":"Cluster","options":[],"query":"label_values(machine_memory_bytes{},cluster)","refresh":1,"regex":"","skipUrlSync":false,"sort":1,"tagValuesQuery":"","tags":[],"tagsQuery":"","type":"query","useTags":false},{"allValue":".*","current":{"selected":false,"text":"default-http-backend","value":"default-http-backend"},"datasource":"DS_Prometheus","definition":"label_values(kube_deployment_labels{cluster=~\"$Cluster\"},deployment)","hide":0,"includeAll":true,"label":null,"multi":false,"name":"Deployment","options":[],"query":"label_values(kube_deployment_labels{cluster=~\"$Cluster\"},deployment)","refresh":2,"regex":"","skipUrlSync":false,"sort":0,"tagValuesQuery":"","tags":[],"tagsQuery":"","type":"query","useTags":false},{"allValue":".*","current":{"selected":false,"text":"All","value":"$__all"},"datasource":"DS_Prometheus","definition":"label_values(machine_memory_bytes{cluster=~\"$Cluster\"},kubernetes_io_hostname)","hide":2,"includeAll":true,"label":null,"multi":false,"name":"Node","options":[],"query":"label_values(machine_memory_bytes{cluster=~\"$Cluster\"},kubernetes_io_hostname)","refresh":2,"regex":"","skipUrlSync":false,"sort":0,"tagValuesQuery":"","tags":[],"tagsQuery":"","type":"query","useTags":false}]},"time":{"from":"now-15m","to":"now"},"timepicker":{"refresh_intervals":["10s","30s","1m","5m","15m","30m","1h","2h","1d"],"time_options":["5m","15m","1h","6h","12h","24h","2d","7d","30d"]},"timezone":"browser","title":"Kubernetes Deployment metrics","uid":"XOE4Cfmz","version":1}
  clustermetrics.json: |-
    {"annotations":{"list":[{"builtIn":1,"datasource":"-- Grafana --","enable":true,"hide":true,"iconColor":"rgba(0, 211, 255, 1)","name":"Annotations & Alerts","type":"dashboard"}]},"description":"Monitors Kubernetes cluster using Prometheus. Shows overall cluster CPU / Memory / Filesystem usage as well as individual pod, containers, systemd services statistics. Uses cAdvisor metrics only.","editable":true,"gnetId":315,"graphTooltip":0,"iteration":1592255067093,"links":[],"panels":[{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"fillGradient":0,"grid":{},"gridPos":{"h":5,"w":24,"x":0,"y":0},"height":"200px","hiddenSeries":false,"id":32,"legend":{"alignAsTable":false,"avg":true,"current":true,"max":false,"min":false,"rightSide":false,"show":false,"sideWidth":200,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":" sum (rate (container_network_receive_bytes_total{kubernetes_io_hostname=~\"^$Node$\",cluster=~\"^$Cluster$\"}[1m]))","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"Received","metric":"network","refId":"A","step":10},{"expr":"- sum (rate (container_network_transmit_bytes_total{kubernetes_io_hostname=~\"^$Node$\",cluster=~\"^$Cluster$\"}[1m]))","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"Sent","metric":"network","refId":"B","step":10}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Network I/O pressure","tooltip":{"msResolution":false,"shared":true,"sort":0,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"Bps","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"Bps","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"cacheTimeout":null,"colorBackground":false,"colorValue":true,"colors":["rgba(50, 172, 45, 0.97)","rgba(237, 129, 40, 0.89)","rgba(245, 54, 54, 0.9)"],"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"percent","gauge":{"maxValue":100,"minValue":0,"show":true,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":5,"w":8,"x":0,"y":5},"height":"180px","id":4,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (container_memory_working_set_bytes{id=\"/\",kubernetes_io_hostname=~\"^$Node$\",container!=\"POD\",cluster=~\"^$Cluster$\"}) / sum (machine_memory_bytes{kubernetes_io_hostname=~\"^$Node$\",cluster=~\"^$Cluster$\"}) * 100","format":"time_series","interval":"10s","intervalFactor":1,"refId":"A","step":30}],"thresholds":"65, 90","title":"Cluster memory usage","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"cacheTimeout":null,"colorBackground":false,"colorValue":true,"colors":["rgba(50, 172, 45, 0.97)","rgba(237, 129, 40, 0.89)","rgba(245, 54, 54, 0.9)"],"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"percent","gauge":{"maxValue":100,"minValue":0,"show":true,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":5,"w":8,"x":8,"y":5},"height":"180px","id":6,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (rate (container_cpu_usage_seconds_total{cluster=~\"^$Cluster$\",id=\"/\",kubernetes_io_hostname=~\"^$Node$\",container!=\"POD\"}[1m])) / sum (machine_cpu_cores{cluster=~\"^$Cluster$\",kubernetes_io_hostname=~\"^$Node$\"}) * 100","format":"time_series","interval":"10s","intervalFactor":1,"refId":"A","step":30}],"thresholds":"65, 90","title":"Cluster CPU usage (1m avg)","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"cacheTimeout":null,"colorBackground":false,"colorValue":true,"colors":["rgba(50, 172, 45, 0.97)","rgba(237, 129, 40, 0.89)","rgba(245, 54, 54, 0.9)"],"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"percent","gauge":{"maxValue":100,"minValue":0,"show":true,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":5,"w":8,"x":16,"y":5},"height":"180px","id":7,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (container_fs_usage_bytes{cluster=~\"^$Cluster$\",device=~\"^/dev/[sv]d[a-z][1-9]$\",id=\"/\",kubernetes_io_hostname=~\"^$Node$\",container!=\"POD\"}) / sum (container_fs_limit_bytes{cluster=~\"^$Cluster$\",container!=\"POD\",device=~\"^/dev/[sv]d[a-z][1-9]$\",id=\"/\",kubernetes_io_hostname=~\"^$Node$\"}) * 100","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"","metric":"","refId":"A","step":30}],"thresholds":"65, 90","title":"Cluster filesystem usage","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(50, 172, 45, 0.97)","rgba(237, 129, 40, 0.89)","rgba(245, 54, 54, 0.9)"],"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"bytes","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":4,"x":0,"y":10},"height":"1px","id":9,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"20%","prefix":"","prefixFontSize":"20%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (container_memory_working_set_bytes{cluster=~\"^$Cluster$\",id=\"/\",kubernetes_io_hostname=~\"^$Node$\",container!=\"POD\"})","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"","refId":"A","step":30}],"thresholds":"","title":"Used","type":"singlestat","valueFontSize":"50%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(50, 172, 45, 0.97)","rgba(237, 129, 40, 0.89)","rgba(245, 54, 54, 0.9)"],"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"bytes","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":4,"x":4,"y":10},"height":"1px","id":10,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (machine_memory_bytes{cluster=~\"^$Cluster$\",kubernetes_io_hostname=~\"^$Node$\"})","interval":"10s","intervalFactor":1,"legendFormat":"","refId":"A","step":30}],"thresholds":"","title":"Total","type":"singlestat","valueFontSize":"50%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(50, 172, 45, 0.97)","rgba(237, 129, 40, 0.89)","rgba(245, 54, 54, 0.9)"],"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":4,"x":8,"y":10},"height":"1px","id":11,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":" cores","postfixFontSize":"30%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (rate (container_cpu_usage_seconds_total{cluster=~\"^$Cluster$\",id=\"/\",kubernetes_io_hostname=~\"^$Node$\",container!=\"POD\"}[1m]))","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"","refId":"A","step":30}],"thresholds":"","title":"Used","type":"singlestat","valueFontSize":"50%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(50, 172, 45, 0.97)","rgba(237, 129, 40, 0.89)","rgba(245, 54, 54, 0.9)"],"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":4,"x":12,"y":10},"height":"1px","id":12,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":" cores","postfixFontSize":"30%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (machine_cpu_cores{cluster=~\"^$Cluster$\",kubernetes_io_hostname=~\"^$Node$\"})","interval":"10s","intervalFactor":1,"legendFormat":"","refId":"A","step":30}],"thresholds":"","title":"Total","type":"singlestat","valueFontSize":"50%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(50, 172, 45, 0.97)","rgba(237, 129, 40, 0.89)","rgba(245, 54, 54, 0.9)"],"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"bytes","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":4,"x":16,"y":10},"height":"1px","id":13,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (container_fs_usage_bytes{cluster=~\"^$Cluster$\",device=~\"^/dev/[sv]d[a-z][1-9]$\",id=\"/\",kubernetes_io_hostname=~\"^$Node$\",container!=\"POD\"})","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"","refId":"A","step":30}],"thresholds":"","title":"Used","type":"singlestat","valueFontSize":"50%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(50, 172, 45, 0.97)","rgba(237, 129, 40, 0.89)","rgba(245, 54, 54, 0.9)"],"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"bytes","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":4,"x":20,"y":10},"height":"1px","id":14,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"sum (container_fs_limit_bytes{cluster=~\"^$Cluster$\",device=~\"^/dev/[sv]d[a-z][1-9]$\",id=\"/\",kubernetes_io_hostname=~\"^$Node$\",container!=\"POD\"})","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"","refId":"A","step":30}],"thresholds":"","title":"Total","type":"singlestat","valueFontSize":"50%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":3,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":0,"fillGradient":0,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":13},"height":"","hiddenSeries":false,"id":17,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":true,"show":true,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":true,"targets":[{"expr":"sum (rate (container_cpu_usage_seconds_total{cluster=~\"^$Cluster$\",image!=\"\",name=~\"^k8s_.*\",kubernetes_io_hostname=~\"^$Node$\",container!=\"POD\"}[1m])) by (pod)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"{{ pod }}","metric":"container_cpu","refId":"A","step":10}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Pods CPU usage (1m avg)","tooltip":{"msResolution":true,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"none","label":"cores","logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":3,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":0,"fillGradient":0,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":20},"height":"","hiddenSeries":false,"id":23,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":true,"show":true,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":true,"targets":[{"expr":"sum (rate (container_cpu_usage_seconds_total{cluster=~\"^$Cluster$\",systemd_service_name!=\"\",kubernetes_io_hostname=~\"^$Node$\",container!=\"POD\"}[1m])) by (systemd_service_name)","format":"time_series","hide":false,"interval":"10s","intervalFactor":1,"legendFormat":"{{ systemd_service_name }}","metric":"container_cpu","refId":"A","step":10}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"System services CPU usage (1m avg)","tooltip":{"msResolution":true,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"none","label":"cores","logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":3,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":0,"fillGradient":0,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":27},"height":"","hiddenSeries":false,"id":24,"legend":{"alignAsTable":true,"avg":true,"current":true,"hideEmpty":false,"hideZero":false,"max":false,"min":false,"rightSide":true,"show":true,"sideWidth":null,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":true,"targets":[{"expr":"sum (rate (container_cpu_usage_seconds_total{cluster=~\"^$Cluster$\",image!=\"\",name=~\"^k8s_.*\",container!=\"POD\",kubernetes_io_hostname=~\"^$Node$\"}[1m])) by (container, pod)","format":"time_series","hide":false,"interval":"10s","intervalFactor":1,"legendFormat":"pod: {{ pod }} | {{ container }}","metric":"container_cpu","refId":"A","step":10},{"expr":"sum (rate (container_cpu_usage_seconds_total{cluster=~\"^$Cluster$\",image!=\"\",name!~\"^k8s_.*\",kubernetes_io_hostname=~\"^$Node$\"}[1m])) by (kubernetes_io_hostname, name, image)","format":"time_series","hide":false,"interval":"10s","intervalFactor":1,"legendFormat":"docker: {{ kubernetes_io_hostname }} | {{ image }} ({{ name }})","metric":"container_cpu","refId":"B","step":10},{"expr":"sum (rate (container_cpu_usage_seconds_total{cluster=~\"^$Cluster$\",rkt_container!=\"\",kubernetes_io_hostname=~\"^$Node$\"}[1m])) by (kubernetes_io_hostname, rkt_container)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"rkt: {{ kubernetes_io_hostname }} | {{ rkt_container }}","metric":"container_cpu","refId":"C","step":10}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Containers CPU usage (1m avg)","tooltip":{"msResolution":true,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"none","label":"cores","logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":3,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":0,"fillGradient":0,"grid":{},"gridPos":{"h":13,"w":24,"x":0,"y":34},"hiddenSeries":false,"id":20,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":false,"show":true,"sortDesc":false,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":true,"targets":[{"expr":"sum (rate (container_cpu_usage_seconds_total{cluster=~\"^$Cluster$\",id!=\"/\",kubernetes_io_hostname=~\"^$Node$\",container!=\"POD\"}[1m])) by (id)","format":"time_series","hide":false,"interval":"10s","intervalFactor":1,"legendFormat":"{{ id }}","metric":"container_cpu","refId":"A","step":10}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"All processes CPU usage (1m avg)","tooltip":{"msResolution":true,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"none","label":"cores","logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":0,"fillGradient":0,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":47},"hiddenSeries":false,"id":25,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":true,"show":true,"sideWidth":200,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":true,"targets":[{"expr":"sum (container_memory_working_set_bytes{cluster=~\"^$Cluster$\",image!=\"\",name=~\"^k8s_.*\",kubernetes_io_hostname=~\"^$Node$\",container!=\"POD\"}) by (pod)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"{{ pod }}","metric":"container_memory_usage:sort_desc","refId":"A","step":10}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Pods memory usage","tooltip":{"msResolution":false,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":0,"fillGradient":0,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":54},"hiddenSeries":false,"id":26,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":true,"show":true,"sideWidth":200,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":true,"targets":[{"expr":"sum (container_memory_working_set_bytes{cluster=~\"^$Cluster$\",systemd_service_name!=\"\",kubernetes_io_hostname=~\"^$Node$\"}) by (systemd_service_name)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"{{ systemd_service_name }}","metric":"container_memory_usage:sort_desc","refId":"A","step":10}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"System services memory usage","tooltip":{"msResolution":false,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":0,"fillGradient":0,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":61},"hiddenSeries":false,"id":27,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":true,"show":true,"sideWidth":200,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":true,"targets":[{"expr":"sum (container_memory_working_set_bytes{cluster=~\"^$Cluster$\",image!=\"\",name=~\"^k8s_.*\",container!=\"POD\",kubernetes_io_hostname=~\"^$Node$\"}) by (container, pod)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"pod: {{ pod }} | {{ container }}","metric":"container_memory_usage:sort_desc","refId":"A","step":10},{"expr":"sum (container_memory_working_set_bytes{cluster=~\"^$Cluster$\",image!=\"\",name!~\"^k8s_.*\",kubernetes_io_hostname=~\"^$Node$\"}) by (kubernetes_io_hostname, name, image)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"docker: {{ kubernetes_io_hostname }} | {{ image }} ({{ name }})","metric":"container_memory_usage:sort_desc","refId":"B","step":10},{"expr":"sum (container_memory_working_set_bytes{cluster=~\"^$Cluster$\",rkt_container!=\"\",kubernetes_io_hostname=~\"^$Node$\"}) by (kubernetes_io_hostname, rkt_container)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"rkt: {{ kubernetes_io_hostname }} | {{ rkt_container }}","metric":"container_memory_usage:sort_desc","refId":"C","step":10}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Containers memory usage","tooltip":{"msResolution":false,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":0,"fillGradient":0,"grid":{},"gridPos":{"h":13,"w":24,"x":0,"y":68},"hiddenSeries":false,"id":28,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":false,"show":true,"sideWidth":200,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":true,"targets":[{"expr":"sum (container_memory_working_set_bytes{cluster=~\"^$Cluster$\",id!=\"/\",kubernetes_io_hostname=~\"^$Node$\"}) by (id)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"{{ id }}","metric":"container_memory_usage:sort_desc","refId":"A","step":10}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"All processes memory usage","tooltip":{"msResolution":false,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"fillGradient":0,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":81},"hiddenSeries":false,"id":16,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":true,"show":true,"sideWidth":200,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum (rate (container_network_receive_bytes_total{cluster=~\"^$Cluster$\",image!=\"\",name=~\"^k8s_.*\",kubernetes_io_hostname=~\"^$Node$\"}[1m])) by (pod)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"-> {{ pod }}","metric":"network","refId":"A","step":10},{"expr":"- sum (rate (container_network_transmit_bytes_total{cluster=~\"^$Cluster$\",image!=\"\",name=~\"^k8s_.*\",kubernetes_io_hostname=~\"^$Node$\"}[1m])) by (pod)","format":"time_series","interval":"10s","intervalFactor":1,"legendFormat":"<- {{ pod }}","metric":"network","refId":"B","step":10}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Pods network I/O (1m avg)","tooltip":{"msResolution":false,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"Bps","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"fillGradient":0,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":88},"hiddenSeries":false,"id":30,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":true,"show":true,"sideWidth":200,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum (rate (container_network_receive_bytes_total{cluster=~\"^$Cluster$\",image!=\"\",name=~\"^k8s_.*\",kubernetes_io_hostname=~\"^$Node$\"}[1m])) by (container, pod)","format":"time_series","hide":false,"interval":"10s","intervalFactor":1,"legendFormat":"-> pod: {{ pod }} | {{ container }}","metric":"network","refId":"B","step":10},{"expr":"- sum (rate (container_network_transmit_bytes_total{cluster=~\"^$Cluster$\",image!=\"\",name=~\"^k8s_.*\",kubernetes_io_hostname=~\"^$Node$\"}[1m])) by (container, pod)","format":"time_series","hide":false,"interval":"10s","intervalFactor":1,"legendFormat":"<- pod: {{ pod }} | {{ container }}","metric":"network","refId":"D","step":10},{"expr":"sum (rate (container_network_receive_bytes_total{cluster=~\"^$Cluster$\",image!=\"\",name!~\"^k8s_.*\",kubernetes_io_hostname=~\"^$Node$\"}[1m])) by (kubernetes_io_hostname, name, image)","format":"time_series","hide":false,"interval":"10s","intervalFactor":1,"legendFormat":"-> docker: {{ kubernetes_io_hostname }} | {{ image }} ({{ name }})","metric":"network","refId":"A","step":10},{"expr":"- sum (rate (container_network_transmit_bytes_total{cluster=~\"^$Cluster$\",image!=\"\",name!~\"^k8s_.*\",kubernetes_io_hostname=~\"^$Node$\"}[1m])) by (kubernetes_io_hostname, name, image)","format":"time_series","hide":false,"interval":"10s","intervalFactor":1,"legendFormat":"<- docker: {{ kubernetes_io_hostname }} | {{ image }} ({{ name }})","metric":"network","refId":"C","step":10},{"expr":"sum (rate (container_network_transmit_bytes_total{cluster=~\"^$Cluster$\",rkt_container!=\"\",kubernetes_io_hostname=~\"^$Node$\"}[1m])) by (kubernetes_io_hostname, rkt_container)","format":"time_series","hide":false,"interval":"10s","intervalFactor":1,"legendFormat":"-> rkt: {{ kubernetes_io_hostname }} | {{ rkt_container }}","metric":"network","refId":"E","step":10},{"expr":"- sum (rate (container_network_transmit_bytes_total{cluster=~\"^$Cluster$\",rkt_container!=\"\",kubernetes_io_hostname=~\"^$Node$\"}[1m])) by (kubernetes_io_hostname, rkt_container)","format":"time_series","hide":false,"interval":"10s","intervalFactor":1,"legendFormat":"<- rkt: {{ kubernetes_io_hostname }} | {{ rkt_container }}","metric":"network","refId":"F","step":10}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Containers network I/O (1m avg)","tooltip":{"msResolution":false,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"Bps","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"fillGradient":0,"grid":{},"gridPos":{"h":13,"w":24,"x":0,"y":95},"hiddenSeries":false,"id":29,"legend":{"alignAsTable":true,"avg":true,"current":true,"max":false,"min":false,"rightSide":false,"show":true,"sideWidth":200,"sort":"current","sortDesc":true,"total":false,"values":true},"lines":true,"linewidth":2,"links":[],"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum (rate (container_network_receive_bytes_total{cluster=~\"^$Cluster$\",id!=\"/\",kubernetes_io_hostname=~\"^$Node$\"}[1m])) by (id)","interval":"10s","intervalFactor":1,"legendFormat":"-> {{ id }}","metric":"network","refId":"A","step":10},{"expr":"- sum (rate (container_network_transmit_bytes_total{cluster=~\"^$Cluster$\",id!=\"/\",kubernetes_io_hostname=~\"^$Node$\"}[1m])) by (id)","interval":"10s","intervalFactor":1,"legendFormat":"<- {{ id }}","metric":"network","refId":"B","step":10}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"All processes network I/O (1m avg)","tooltip":{"msResolution":false,"shared":true,"sort":2,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"Bps","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}}],"refresh":"5s","schemaVersion":25,"style":"dark","tags":["kubernetes"],"templating":{"list":[{"allValue":null,"current":{"selected":false,"text":"vt-ts-150-prometheus","value":"vt-ts-150-prometheus"},"datasource":"DS_Prometheus","definition":"label_values(machine_memory_bytes{},cluster)","hide":0,"includeAll":false,"label":"","multi":false,"name":"Cluster","options":[],"query":"label_values(machine_memory_bytes{},cluster)","refresh":1,"regex":"","skipUrlSync":false,"sort":1,"tagValuesQuery":"","tags":[],"tagsQuery":"","type":"query","useTags":false},{"allValue":".*","current":{"selected":true,"text":"gke-vt-ts-150-default-pool-bdbe5dd4-bg3m","value":"gke-vt-ts-150-default-pool-bdbe5dd4-bg3m"},"datasource":"DS_Prometheus","definition":"label_values(machine_memory_bytes{cluster=~\"$Cluster\"},kubernetes_io_hostname)","hide":0,"includeAll":true,"label":null,"multi":false,"name":"Node","options":[],"query":"label_values(machine_memory_bytes{cluster=~\"$Cluster\"},kubernetes_io_hostname)","refresh":1,"regex":"","skipUrlSync":false,"sort":0,"tagValuesQuery":"","tags":[],"tagsQuery":"","type":"query","useTags":false}]},"time":{"from":"now-5m","to":"now"},"timepicker":{"refresh_intervals":["10s","30s","1m","5m","15m","30m","1h","2h","1d"],"time_options":["5m","15m","1h","6h","12h","24h","2d","7d","30d"]},"timezone":"browser","title":"Kubernetes Cluster Monitoring","uid":"-2kz1fik","version":1}
  nodemetrics.json: |-
    {"annotations":{"list":[{"builtIn":1,"datasource":"-- Grafana --","enable":true,"hide":true,"iconColor":"rgba(0, 211, 255, 1)","name":"Annotations & Alerts","type":"dashboard"}]},"description":"Dashboard to view multiple servers","editable":true,"gnetId":405,"graphTooltip":0,"iteration":1591041793561,"links":[],"panels":[{"content":"","datasource":null,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"gridPos":{"h":3,"w":24,"x":0,"y":0},"id":11,"links":[],"maxPerRow":6,"mode":"html","repeat":"node","repeatDirection":"h","scopedVars":{"node":{"selected":true,"text":"10.128.0.44:9100","value":"10.128.0.44:9100"}},"style":{},"title":"$node","type":"text"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":24,"x":0,"y":18},"id":20,"interval":null,"links":[],"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"maxPerRow":6,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"repeat":"node","scopedVars":{"node":{"selected":true,"text":"10.128.0.44:9100","value":"10.128.0.44:9100"}},"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":false},"tableColumn":"","targets":[{"expr":"count(node_cpu_seconds_total{instance=~'$node',cluster=~'$Cluster', mode=\"system\"}) or count(node_cpu{instance=~'$node',cluster=~'$Cluster', mode=\"system\"})","instant":true,"interval":"","intervalFactor":2,"legendFormat":"","metric":"","refId":"A","step":14400,"target":""}],"thresholds":"","title":"CPU Cores","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":3,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":10,"fillGradient":0,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":36},"hiddenSeries":false,"id":7,"legend":{"alignAsTable":false,"avg":false,"current":false,"hideEmpty":false,"max":false,"min":false,"rightSide":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":0,"links":[],"maxPerRow":6,"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":true,"pointradius":5,"points":false,"renderer":"flot","repeat":"node","scopedVars":{"node":{"selected":true,"text":"10.128.0.44:9100","value":"10.128.0.44:9100"}},"seriesOverrides":[],"spaceLength":10,"stack":true,"steppedLine":false,"targets":[{"expr":"sum(irate(node_cpu_seconds_total{mode=\"system\",instance=~'$node',cluster=~'$Cluster'}[5m])) or sum(irate(node_cpu{mode=\"system\",instance=~'$node',cluster=~'$Cluster'}[5m]))","interval":"","intervalFactor":2,"legendFormat":"system","metric":"","refId":"A","step":1200,"target":""},{"expr":"sum(irate(node_cpu_seconds_total{mode=\"user\",instance=~'$node',cluster=~'$Cluster'}[5m])) or sum(irate(node_cpu{mode=\"user\",instance=~'$node',cluster=~'$Cluster'}[5m]))","interval":"","intervalFactor":2,"legendFormat":"user","refId":"B","step":1200},{"expr":"sum(irate(node_cpu_seconds_total{mode=\"nice\",instance=~'$node',cluster=~'$Cluster'}[5m])) or sum(irate(node_cpu{mode=\"nice\",instance=~'$node',cluster=~'$Cluster'}[5m]))","interval":"","intervalFactor":2,"legendFormat":"nice","refId":"C","step":1200},{"expr":"sum(irate(node_cpu_seconds_total{mode=\"iowait\",instance=~'$node',cluster=~'$Cluster'}[5m])) or sum(irate(node_cpu{mode=\"iowait\",instance=~'$node',cluster=~'$Cluster'}[5m]))","interval":"","intervalFactor":2,"legendFormat":"iowait","refId":"E","step":1200},{"expr":"sum(irate(node_cpu_seconds_total{mode=\"steal\",instance=~'$node',cluster=~'$Cluster'}[5m])) or sum(irate(node_cpu{mode=\"steal\",instance=~'$node',cluster=~'$Cluster'}[5m]))","intervalFactor":2,"legendFormat":"steal","refId":"H","step":1200},{"expr":"sum(irate(node_cpu_seconds_total{mode=\"idle\",instance=~'$node',cluster=~'$Cluster'}[5m])) or sum(irate(node_cpu{mode=\"idle\",instance=~'$node',cluster=~'$Cluster'}[5m]))","interval":"","intervalFactor":2,"legendFormat":"idle","refId":"D","step":1200},{"expr":"sum(irate(node_cpu_seconds_total{mode=\"irq\",instance=~'$node',cluster=~'$Cluster'}[5m])) or sum(irate(node_cpu{mode=\"irq\",instance=~'$node',cluster=~'$Cluster'}[5m]))","interval":"","intervalFactor":2,"legendFormat":"irq","refId":"F","step":1200},{"expr":"sum(irate(node_cpu_seconds_total{mode=\"softirq\",instance=~'$node',cluster=~'$Cluster'}[5m])) or sum(irate(node_cpu{mode=\"softirq\",instance=~'$node',cluster=~'$Cluster'}[5m]))","interval":"","intervalFactor":2,"legendFormat":"softirq","refId":"G","step":1200},{"expr":"sum(irate(node_cpu_seconds_total{mode=\"guest\",instance=~'$node',cluster=~'$Cluster'}[5m])) or sum(irate(node_cpu{mode=\"guest\",instance=~'$node',cluster=~'$Cluster'}[5m]))","interval":"","intervalFactor":2,"legendFormat":"guest","refId":"I","step":1200}],"thresholds":[{"colorMode":"custom","fill":true,"fillColor":"rgba(216, 200, 27, 0.27)","op":"gt","value":0}],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"CPU","tooltip":{"msResolution":false,"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":"%","logBase":1,"max":100,"min":0,"show":true},{"format":"short","logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{"Slab":"#E5A8E2","Swap":"#E24D42"},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":2,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"fillGradient":0,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":78},"hiddenSeries":false,"id":17,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":2,"links":[],"maxPerRow":6,"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","repeat":"node","scopedVars":{"node":{"selected":true,"text":"10.128.0.44:9100","value":"10.128.0.44:9100"}},"seriesOverrides":[{"alias":"/Apps|Buffers|Cached|Free|Slab|SwapCached|PageTables|VmallocUsed/","fill":5,"stack":true},{"alias":"Swap","fill":5,"stack":true}],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"( node_memory_MemTotal_bytes{instance=~'$node',cluster=~'$Cluster'} - node_memory_MemFree_bytes{instance=~'$node',cluster=~'$Cluster'} - node_memory_Buffers_bytes{instance=~'$node',cluster=~'$Cluster'} - node_memory_Cached_bytes{instance=~'$node',cluster=~'$Cluster'} - node_memory_SwapCached_bytes{instance=~'$node',cluster=~'$Cluster'} - node_memory_Slab_bytes{instance=~'$node',cluster=~'$Cluster'} - node_memory_PageTables_bytes{instance=~'$node',cluster=~'$Cluster'} - node_memory_VmallocUsed_bytes{instance=~'$node',cluster=~'$Cluster'} ) or ( node_memory_MemTotal{instance=~'$node',cluster=~'$Cluster'} - node_memory_MemFree{instance=~'$node',cluster=~'$Cluster'} - node_memory_Buffers{instance=~'$node',cluster=~'$Cluster'} - node_memory_Cached{instance=~'$node',cluster=~'$Cluster'} - node_memory_SwapCached{instance=~'$node',cluster=~'$Cluster'} - node_memory_Slab{instance=~'$node',cluster=~'$Cluster'} - node_memory_PageTables{instance=~'$node',cluster=~'$Cluster'} - node_memory_VmallocUsed{instance=~'$node',cluster=~'$Cluster'} )","interval":"","intervalFactor":2,"legendFormat":"Apps","metric":"","refId":"A","step":1200,"target":""},{"expr":"node_memory_Buffers_bytes{instance=~'$node',cluster=~'$Cluster'} or node_memory_Buffers{instance=~'$node',cluster=~'$Cluster'}","interval":"","intervalFactor":2,"legendFormat":"Buffers","refId":"B","step":1200},{"expr":"node_memory_Cached_bytes{instance=~'$node',cluster=~'$Cluster'} or node_memory_Cached{instance=~'$node',cluster=~'$Cluster'}","interval":"","intervalFactor":2,"legendFormat":"Cached","refId":"D","step":1200},{"expr":"node_memory_MemFree_bytes{instance=~'$node',cluster=~'$Cluster'} or node_memory_MemFree{instance=~'$node',cluster=~'$Cluster'}","hide":false,"interval":"","intervalFactor":2,"legendFormat":"Free","refId":"E","step":1200},{"expr":"node_memory_Slab_bytes{instance=~'$node',cluster=~'$Cluster'} or node_memory_Slab{instance=~'$node',cluster=~'$Cluster'}","interval":"","intervalFactor":2,"legendFormat":"Slab","refId":"F","step":1200},{"expr":"node_memory_SwapCached_bytes{instance=~'$node',cluster=~'$Cluster'} or node_memory_SwapCached{instance=~'$node',cluster=~'$Cluster'}","interval":"","intervalFactor":2,"legendFormat":"SwapCached","refId":"G","step":1200},{"expr":"node_memory_PageTables_bytes{instance=~'$node',cluster=~'$Cluster'} or node_memory_PageTables{instance=~'$node',cluster=~'$Cluster'}","interval":"","intervalFactor":2,"legendFormat":"PageTables","refId":"H","step":1200},{"expr":"node_memory_VmallocUsed_bytes{instance=~'$node',cluster=~'$Cluster'} or node_memory_VmallocUsed{instance=~'$node',cluster=~'$Cluster'}","interval":"","intervalFactor":2,"legendFormat":"VmallocUsed","metric":"","refId":"I","step":1200},{"expr":"(node_memory_SwapTotal_bytes{instance=~'$node',cluster=~'$Cluster'} - node_memory_SwapFree{instance=~'$node',cluster=~'$Cluster'}) or (node_memory_SwapTotal{instance=~'$node',cluster=~'$Cluster'} - node_memory_SwapFree{instance=~'$node',cluster=~'$Cluster'})","interval":"","intervalFactor":2,"legendFormat":"Swap","metric":"","refId":"C","step":1200},{"expr":"node_memory_Committed_AS_bytes{instance=~'$node',cluster=~'$Cluster'} or node_memory_Committed_AS{instance=~'$node',cluster=~'$Cluster'}","interval":"","intervalFactor":2,"legendFormat":"Committed","metric":"","refId":"J","step":1200},{"expr":"node_memory_Mapped_bytes{instance=~'$node',cluster=~'$Cluster'} or node_memory_Mapped{instance=~'$node',cluster=~'$Cluster'}","interval":"","intervalFactor":2,"legendFormat":"Mapped","refId":"K","step":1200},{"expr":"node_memory_Active_bytes{instance=~'$node',cluster=~'$Cluster'} or node_memory_Active{instance=~'$node',cluster=~'$Cluster'}","interval":"","intervalFactor":2,"legendFormat":"Active","metric":"","refId":"L","step":1200},{"expr":"node_memory_Inactive_bytes{instance=~'$node',cluster=~'$Cluster'} or node_memory_Inactive{instance=~'$node',cluster=~'$Cluster'}","interval":"","intervalFactor":2,"legendFormat":"Inactive","metric":"","refId":"M","step":1200}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Memory","tooltip":{"msResolution":false,"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":"GB","logBase":1,"max":null,"min":null,"show":true},{"format":"short","logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"fillGradient":0,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":120},"hiddenSeries":false,"id":13,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":2,"links":[],"maxPerRow":6,"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","repeat":"node","scopedVars":{"node":{"selected":true,"text":"10.128.0.44:9100","value":"10.128.0.44:9100"}},"seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"node_load1{instance=~'$node',cluster=~'$Cluster'}","format":"time_series","interval":"","intervalFactor":2,"legendFormat":"load","metric":"","refId":"A","step":1200,"target":""}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Load","tooltip":{"msResolution":false,"shared":true,"sort":0,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","logBase":1,"max":null,"min":null,"show":true},{"format":"short","logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","decimals":3,"editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"fillGradient":0,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":162},"hiddenSeries":false,"id":9,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":2,"links":[],"maxPerRow":6,"nullPointMode":"connected","options":{"dataLinks":[]},"percentage":false,"pointradius":5,"points":false,"renderer":"flot","repeat":"node","scopedVars":{"node":{"selected":true,"text":"10.128.0.44:9100","value":"10.128.0.44:9100"}},"seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"100.0 - 100 * (node_filesystem_avail_bytes{instance=~'$node',cluster=~'$Cluster',device !~'tmpfs',device!~'by-uuid'} / node_filesystem_size_bytes{instance=~'$node',cluster=~'$Cluster',device !~'tmpfs',device!~'by-uuid'}) or 100.0 - 100 * (node_filesystem_avail{instance=~'$node',cluster=~'$Cluster',device !~'tmpfs',device!~'by-uuid'} / node_filesystem_size{instance=~'$node',cluster=~'$Cluster',device !~'tmpfs',device!~'by-uuid'})","interval":"","intervalFactor":2,"legendFormat":"{{mountpoint}}","metric":"","refId":"A","step":1200,"target":""}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Disk Space Used","tooltip":{"msResolution":true,"shared":true,"sort":0,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"percent","logBase":1,"max":100,"min":0,"show":true},{"format":"short","logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":204},"id":19,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":2,"links":[],"maxPerRow":6,"nullPointMode":"connected","percentage":false,"pointradius":5,"points":false,"renderer":"flot","repeat":"node","scopedVars":{"node":{"selected":true,"text":"10.128.0.44:9100","value":"10.128.0.44:9100"}},"seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"irate(node_disk_io_time_seconds_total{instance=~'$node',cluster=~'$Cluster'}[5m])*100 or irate(node_disk_io_time_ms{instance=~'$node',cluster=~'$Cluster'}[5m])/10","interval":"","intervalFactor":2,"legendFormat":"{{device}}","metric":"","refId":"A","step":1200,"target":""}],"thresholds":[],"timeFrom":null,"timeShift":null,"title":"Disk Utilization per Device","tooltip":{"msResolution":false,"shared":false,"sort":0,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"percent","logBase":1,"max":100,"min":null,"show":true},{"format":"short","logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":246},"id":14,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":2,"links":[],"maxPerRow":6,"nullPointMode":"connected","percentage":false,"pointradius":5,"points":false,"renderer":"flot","repeat":"node","scopedVars":{"node":{"selected":true,"text":"10.128.0.44:9100","value":"10.128.0.44:9100"}},"seriesOverrides":[{"alias":"/.*_read$/","transform":"negative-Y"}],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"irate(node_disk_reads_completed_total{instance=~'$node',cluster=~'$Cluster'}[5m]) or irate(node_disk_reads_completed{instance=~'$node',cluster=~'$Cluster'}[5m])","interval":"","intervalFactor":4,"legendFormat":"{{device}}_read","metric":"","refId":"A","step":2400,"target":""},{"expr":"irate(node_disk_writes_completed_total{instance=~'$node',cluster=~'$Cluster'}[5m]) or irate(node_disk_writes_completed{instance=~'$node',cluster=~'$Cluster'}[5m])","intervalFactor":2,"legendFormat":"{{device}}_write","metric":"","refId":"B","step":1200}],"thresholds":[],"timeFrom":null,"timeShift":null,"title":"Disk IOs per Device","tooltip":{"msResolution":false,"shared":false,"sort":0,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":"IO/second read (-) / write (+)","logBase":1,"max":null,"min":null,"show":true},{"format":"short","logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":288},"id":18,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":2,"links":[],"maxPerRow":6,"nullPointMode":"connected","percentage":false,"pointradius":5,"points":false,"renderer":"flot","repeat":"node","scopedVars":{"node":{"selected":true,"text":"10.128.0.44:9100","value":"10.128.0.44:9100"}},"seriesOverrides":[{"alias":"/.*_read/","transform":"negative-Y"}],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"irate(node_disk_read_bytes_total{instance=~'$node',cluster=~'$Cluster'}[5m]) or irate(node_disk_sectors_read{instance=~'$node',cluster=~'$Cluster'}[5m]) * 512","interval":"","intervalFactor":4,"legendFormat":"{{device}}_read","refId":"B","step":2400},{"expr":"irate(node_disk_written_bytes_total{instance=~'$node',cluster=~'$Cluster'}[5m]) or irate(node_disk_sectors_written{instance=~'$node',cluster=~'$Cluster'}[5m]) * 512","interval":"","intervalFactor":4,"legendFormat":"{{device}}_write","metric":"","refId":"A","step":2400,"target":""}],"thresholds":[],"timeFrom":null,"timeShift":null,"title":"Disk Throughput per Device","tooltip":{"msResolution":false,"shared":false,"sort":0,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":"Bytes/second read (-) / write (+)","logBase":1,"max":null,"min":null,"show":true},{"format":"short","logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":330},"id":25,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":2,"links":[],"maxPerRow":6,"nullPointMode":"connected","percentage":false,"pointradius":5,"points":false,"renderer":"flot","repeat":"node","scopedVars":{"node":{"selected":true,"text":"10.128.0.44:9100","value":"10.128.0.44:9100"}},"seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"node_disk_io_now{instance=~'$node',cluster=~'$Cluster'}","interval":"","intervalFactor":1,"legendFormat":"{{device}}","metric":"","refId":"A","step":1200,"target":""}],"thresholds":[],"timeFrom":null,"timeShift":null,"title":"Disk Queue Length","tooltip":{"msResolution":false,"shared":true,"sort":0,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","logBase":1,"max":null,"min":0,"show":true},{"format":"short","logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":372},"id":22,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":2,"links":[],"maxPerRow":6,"nullPointMode":"connected","percentage":false,"pointradius":5,"points":false,"renderer":"flot","repeat":"node","scopedVars":{"node":{"selected":true,"text":"10.128.0.44:9100","value":"10.128.0.44:9100"}},"seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"irate(node_context_switches_total{instance=~'$node',cluster=~'$Cluster'}[5m]) or irate(node_context_switches{instance=~'$node',cluster=~'$Cluster'}[5m])","interval":"","intervalFactor":2,"legendFormat":"context switches","metric":"","refId":"A","step":1200,"target":""}],"thresholds":[],"timeFrom":null,"timeShift":null,"title":"Context Switches","tooltip":{"msResolution":false,"shared":true,"sort":0,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","logBase":1,"max":null,"min":null,"show":true},{"format":"short","logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":414},"id":12,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":2,"links":[],"maxPerRow":6,"nullPointMode":"connected","percentage":false,"pointradius":5,"points":false,"renderer":"flot","repeat":"node","scopedVars":{"node":{"selected":true,"text":"10.128.0.44:9100","value":"10.128.0.44:9100"}},"seriesOverrides":[{"alias":"/.*_in/","transform":"negative-Y"}],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"irate(node_network_receive_bytes_total{instance=~'$node',cluster=~'$Cluster'}[5m])*8 or irate(node_network_receive_bytes{instance=~'$node',cluster=~'$Cluster'}[5m])*8","interval":"","intervalFactor":2,"legendFormat":"{{device}}_in","metric":"","refId":"A","step":1200,"target":""},{"expr":"irate(node_network_transmit_bytes_total{instance=~'$node',cluster=~'$Cluster'}[5m])*8 or irate(node_network_transmit_bytes{instance=~'$node',cluster=~'$Cluster'}[5m])*8","interval":"","intervalFactor":2,"legendFormat":"{{device}}_out","refId":"B","step":1200}],"thresholds":[],"timeFrom":null,"timeShift":null,"title":"Network Traffic","tooltip":{"msResolution":false,"shared":true,"sort":0,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bits","label":"bits in (-) / bits out (+)","logBase":1,"max":null,"min":null,"show":true},{"format":"short","logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":456},"id":21,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":2,"links":[],"maxPerRow":6,"nullPointMode":"connected","percentage":false,"pointradius":5,"points":false,"renderer":"flot","repeat":"node","scopedVars":{"node":{"selected":true,"text":"10.128.0.44:9100","value":"10.128.0.44:9100"}},"seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"node_netstat_Tcp_CurrEstab{instance=~'$node',cluster=~'$Cluster'}","intervalFactor":2,"legendFormat":"established","refId":"A","step":1200,"target":""}],"thresholds":[],"timeFrom":null,"timeShift":null,"title":"Netstat","tooltip":{"msResolution":false,"shared":true,"sort":0,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","logBase":1,"max":null,"min":null,"show":true},{"format":"short","logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":498},"id":23,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":2,"links":[],"maxPerRow":6,"nullPointMode":"connected","percentage":false,"pointradius":5,"points":false,"renderer":"flot","repeat":"node","scopedVars":{"node":{"selected":true,"text":"10.128.0.44:9100","value":"10.128.0.44:9100"}},"seriesOverrides":[{"alias":"/.*Out.*/","transform":"negative-Y"},{"alias":"Udp_NoPorts","yaxis":2}],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"irate(node_netstat_Udp_InDatagrams{instance=~'$node',cluster=~'$Cluster'}[5m])","intervalFactor":2,"legendFormat":"Udp_InDatagrams","refId":"A","step":1200,"target":""},{"expr":"irate(node_netstat_Udp_InErrors{instance=~'$node',cluster=~'$Cluster'}[5m])","intervalFactor":2,"legendFormat":"Udp_InErrors","refId":"B","step":1200},{"expr":"irate(node_netstat_Udp_OutDatagrams{instance=~'$node',cluster=~'$Cluster'}[5m])","interval":"","intervalFactor":2,"legendFormat":"Udp_OutDatagrams","refId":"C","step":1200},{"expr":"irate(node_netstat_Udp_NoPorts{instance=~'$node',cluster=~'$Cluster'}[5m])","intervalFactor":2,"legendFormat":"Udp_NoPorts","refId":"D","step":1200}],"thresholds":[],"timeFrom":null,"timeShift":null,"title":"UDP Stats","tooltip":{"msResolution":false,"shared":true,"sort":0,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","logBase":1,"max":null,"min":null,"show":true},{"format":"short","logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"DS_Prometheus","editable":true,"error":false,"fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"grid":{},"gridPos":{"h":7,"w":24,"x":0,"y":540},"id":24,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":2,"links":[],"maxPerRow":6,"nullPointMode":"connected","percentage":false,"pointradius":5,"points":false,"renderer":"flot","repeat":"node","scopedVars":{"node":{"selected":true,"text":"10.128.0.44:9100","value":"10.128.0.44:9100"}},"seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"node_nf_conntrack_entries_limit{instance=~'$node',cluster=~'$Cluster'} - node_nf_conntrack_entries{instance=~'$node',cluster=~'$Cluster'}","format":"time_series","intervalFactor":2,"legendFormat":"free","refId":"A","step":1200,"target":""}],"thresholds":[],"timeFrom":null,"timeShift":null,"title":"Conntrack","tooltip":{"msResolution":false,"shared":true,"sort":0,"value_type":"cumulative"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","logBase":1,"max":null,"min":null,"show":true},{"format":"short","logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}}],"refresh":"5s","schemaVersion":25,"style":"dark","tags":["kubernetes"],"templating":{"list":[{"allValue":null,"current":{"selected":true,"text":"vt12-ts-prometheus","value":"vt12-ts-prometheus"},"datasource":"DS_Prometheus","definition":"","hide":0,"includeAll":false,"label":null,"multi":false,"name":"Cluster","options":[],"query":"label_values(machine_memory_bytes{},cluster)","refresh":1,"regex":"","skipUrlSync":false,"sort":1,"tagValuesQuery":"","tags":[],"tagsQuery":"","type":"query","useTags":false},{"allFormat":"glob","allValue":null,"current":{"selected":true,"tags":[],"text":"10.128.0.44:9100","value":["10.128.0.44:9100"]},"datasource":"DS_Prometheus","definition":"label_values(node_exporter_build_info{cluster=~\"$Cluster\"}, instance)","hide":0,"includeAll":false,"label":"","multi":true,"multiFormat":"regex values","name":"node","options":[],"query":"label_values(node_exporter_build_info{cluster=~\"$Cluster\"}, instance)","refresh":1,"regex":"","skipUrlSync":false,"sort":1,"tagValuesQuery":"","tags":[],"tagsQuery":"","type":"query","useTags":false}]},"time":{"from":"now-5m","to":"now"},"timepicker":{"now":true,"refresh_intervals":["10s","30s","1m","5m","15m","30m","1h","2h","1d"],"time_options":["5m","15m","1h","6h","12h","24h","2d","7d","30d"]},"timezone":"browser","title":"Kubernetes Nodes","uid":"5oUKj2mk","version":1}

Above configmap creates following dashboards automatically when it is mounted on /etc/grafana/provisioning/dashboards directory in Grafana pod’s container.

Kubernetes Cluster Monitoring
Kubernetes Nodes

Create Grafana dashboards configmap using the following command.

Shell

cloudshell:~/prometheus$ kubectl apply -f grafana-dashboards.yaml
configmap/grafana-dashboards created

Copy the following content into a file called grafana-datasources.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasource-conf
  labels:
    name: grafana-datasource-conf
  namespace: monitoring
data:
  datasources.yaml: |-
    apiVersion: 1
    deleteDatasources:
      - name: Prometheus
        orgId: 1
    datasources:
    - name: DS_Prometheus
      type: prometheus
      access: proxy
      orgId: 1
      url: http://prometheus-service-internal.monitoring.svc.cluster.local:8080
      version: 1
      editable: true

Above configmap creates Prometheus data source automatically when it is mounted on /etc/grafana/provisioning/datasource directory in Grafana pod’s container.

Since we have already created Prometheus internal service in earlier post, using prometheus-service-internal.monitoring.svc.cluster.local fqdn on port 8080, here we are using same end point as a data source.

Create Grafana data source configmap for Prometheus, using the following command.

Shell

cloudshell:~/prometheus$ kubectl apply -f grafana-datasources.yaml
configmap/grafana-datasource-conf created

Copy the following content into a file called grafana.yaml. The following yaml file configuration will create a Grafana deployment and service, once applied. upon successful completion we can access Grafana web UI using external load balancer IP address.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      securityContext:
        fsGroup: 472
        supplementalGroups:
          - 0
      containers:
        - name: grafana
          image: grafana/grafana:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              name: http-grafana
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /robots.txt
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 2
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 3000
            timeoutSeconds: 1
          resources:
            requests:
              cpu: 50m
              memory: 750Mi
          volumeMounts:
            - mountPath: /var/lib/grafana
              name: grafana-storage
            - mountPath: /etc/grafana/provisioning/datasources
              name: grafana-datasources
              readOnly: false
            - mountPath: /etc/grafana/provisioning/dashboards
              name: grafana-dashboards
              readOnly: false
      volumes:
        - name: grafana-storage
          emptyDir: {}
        - name: grafana-datasources
          configMap:
              defaultMode: 420
              name: grafana-datasource-conf
        - name: grafana-dashboards
          configMap:
              defaultMode: 420
              name: grafana-dashboards
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
spec:
  ports:
    - port: 3000
      protocol: TCP
      targetPort: http-grafana
  selector:
    app: grafana
  sessionAffinity: None
  type: LoadBalancer

Create Grafana deployment and service, using the following command.

Shell

cloudshell:~/prometheus$ kubectl apply -f grafana.yaml
deployment.apps/grafana configured
service/grafana unchanged

After couple of seconds you can go to GKE console -> ‘Services & Ingress’ for External load balancer IP to access Grafana console.

Once you click on the endpoint it will take you to the Grafana console, where you can login with default user name and password which are admin/admin.

Then navigate to dashboards section to see the dashboards we have automatically created using grafana-dashboards.yml file.

Conclusion

In this quick start demo we have configured Grafana Server on a GKE cluster to consume Prometheus time series data to create Kubernetes cluster monitoring dashboards automatically using configmaps.

You can find more information about Grafana in official documentation. You can find more pre-configured dashboards here. you can download the json file and include them in grafana-dashboards.yml file to automate dashboard creation.

Sudheer Valla

July 9, 2022

Kubernetes, Monitoring

DevOps, GKE, Kubernetes, Prometheus