Prometheus + Kubernetes + Grafana

Persistent Prometheus & Grafana on Kubernetes

What is Prometheus?

Prometheus is an open-source system monitoring and alerting toolkit originally built at SoundCloud. Since its inception, many companies and organizations have adopted Prometheus for monitoring their systems. This tool is used for complex queries and has its own language called PromQL.

What is Grafana?

Grafana is an open-source solution for running data analytics, pulling up metrics that make sense of the massive amount of data & to monitor our apps with the help of cool customizable dashboards like graphs, tables etc. Grafana is capable of connecting to various types of data sources like Prometheus, Influx DB, My SQL etc.

As the industry is moving towards so containerisation for its applications, similarly monitoring tools can also be set upon them. By default storages of pods in Kubernetes are not persistent so our configuration data will be lost every time we face any issue. Hence, in this article, we will guide you to make your configuration persistent and working.

Basic terminologies we need to know before we proceed with the task:

  • Pod: Pod is the basic unit in Kubernetes inside which the containers run. Pods alone are not of much use as they don’t have any special features.
  • ReplicaSet: This is a feature that looks that the number of specified pods are running at any point in time.
  • Deployment: It is an intelligent unit of Kubernetes that supports features like scaling of pods, replica sets, updates rollouts etc.
  • Persistent Volume Claim (PVC): PVs are volume plugins like Volumes, but have a lifecycle independent of any individual pod that uses the PV. A PVC) is a request for storage by a user.
  • Service: Service is an abstract way to expose an application running on a set of Pods as a network service.
  • ConfigMap: ConfigMaps allow you to decouple configuration artifacts from image content to keep containerized applications portable. This is what helps us make things persistent.
  • Kusmotization.yaml: It contains the sequence of manifest files that can be run together by a single command.

Now that we know the basic terminologies, using these we are going to create a setup of monitoring and visualisation that will be persistent until a change is done.

I am running and testing this setup on Windows 10 and require the following setup on the system.

  1. Minikube (
  2. Kubectl (

Let’s begin with our setup.

Let’s start my minikube first and check if my kubectl is configured properly.

minikube startkubectl get nodes
Minikube Started and Kubectl tested

I will be using official Docker images of Prometheus and Grafana.

Let’s start with our manifest files. First, we will deploy Prometheus.


First, we will create a PVC for persistent storage of all the logs. I have specified accessModes and storage size as per my requirement.


Now, we will create a configmap to pass the persistent configuration to the Prometheus deployment. We have used file as the data for configmap which will be used as configuration file for the Prometheus service in the pods. The file contains its scrape configuration and its targets from where it will be getting the metrics.


Now we will create the file for Deployment of Prometheus pods. We have passed the file path that is to be used as the configuration for the Prometheus using “args” as it is mentioned in the image arguments. We have done the same thing for the path where the collected metrics are stored.

To pass this file we have volume mounts and volume attributes. These will be used to mount the PVC as well.


The last one here is for the service that will be used to expose the pods to the external network. We will be using NodePort for the same. In our case, it will then work on minikube IP:port no as requested by us.

The final will be the Kustomization file that will help us ease our job.

Provisioning the Deployment || Running Dashboard

You can test the persistent configuration by deleting the pod as Deployment will restart another pod. And you can check the configuration file will be the same. For changing configuration, just make changes in configmap and restart pods.

Now moving on to persistent Grafana setup.


First, we will create a PVC to store the settings changed in Grafana dashboard. This can be done for storing other configuration settings as well.


This configmap will be used to create a persistent data source in Grafana. Similar files can be made for adding dashboard, plugins etc.


This file will create the required deployment. Mount paths are set to predefined paths for persistent storage in contrast to Prometheus where I used user-defined paths for the same


This service will similarly expose my Grafana pods to the network.


The final will be the Kustomization file that will help us ease our job.

Provisioning the Deployment || Running Dashboard

Default credentials are username — admin and password — admin

Pre created Datasource || Visualising metrics

So finally our setup is done and is completely persistent as per our needs. I have hardcoded the IP’s and port numbers which can be automated as well via various means.

All the accumulated codes can be found here.

For any queries, corrections or suggestions, connect with me on my LinkedIn.

Worked in collaboration with Daksh Jain.




Just a tech enthusiast… Research, implement and share is what I like to do

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Everyones favourite penguin.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ashish Kumar

Ashish Kumar

Just a tech enthusiast… Research, implement and share is what I like to do

More from Medium