Provisioning and Monitoring EKS via Terraform
What we will be doing?
- Provisioning Elastic Kubernetes Service(EKS) via Terraform
- Provisioning Elastic File System(EFS) via Terraform
- Provisioning Kube State Metrics on EKS via Manifests
- Provisioning Prometheus on EKS via Terraform
- Provisioning Grafana on EKS via Helm Charts
What do we aim to reach?
Our idea is to integrate the mentioned technologies and bring them under a single provisioning setup. These are some of the basic tools required to monitor the smoothing functioning of a Kubernetes deployment. By smooth functioning, we refer to advanced features like High Availability, Auto Scaling etc. The integration will be done and explained as mentioned above.
We will require the following things installed in our local system before we start :
- AWS Account
- Terraform v0.12.26 (https://www.terraform.io/downloads.html)
- Kubectl v1.18.0 (https://kubernetes.io/docs/tasks/tools/install-kubectl/)
- Helm v2.16.9 (https://github.com/helm/helm/releases)
- Eksctl v0.23.0 (https://github.com/weaveworks/eksctl/releases)
NOTE: I have used Windows 10 as my local system.
Let’s start with code and provisioning of EKS and EFS (via Terraform)
As EKS and EFS are services provided by AWS, hence we will use its provider. We will need our current region and the available availability zones at the point of time so we will data sources for the same with the required attributes.
http provider has just been initialised here and will be used and explained later.
aws configure --profile <profilename>Configure your profile with the following to use in the AWS provider. The credentials can be created and found on the AWS dashboard.
“http://ipv4.icanhazip.com” shows the public IP address of the host system which is then stored in a local to be used later in ingress CIDR block for the EKS master.
We have stored a name for EKS cluster in a variable to be used later. This is done so we don’t have to care for spelling mistakes is names which are very difficult to debug.
Now we will create a Virtual Private Cloud(VPC) for the EKS. We will create one with our choice of CIDR block. Make sure dns_hostnames and dns_support are enabled so that hostnames are also mapped to the instances in the VPC and are resolvable by anyone trying to connect to it.
Next, we create one subnet in each of the availability zones we got from our data sources. It will be done using the count indexes and the CIDR blocks will be allotted as per required automatically as well. Make sure public_ip mapping is enabled as well so outside connectivity is open for now. Tags have been used for internal use by the cluster as well as for identification purposes.
Internet Gateway is created in the VPC to allow all the instances with public IP to connect to the internet.
Route table and Route Table Associations are used to create and define how the packet or data flow will take place. We have created in such a way that the internet is accessible both ways.
EKS Cluster requires some powers to make changes and work in our account like creating and deleting instances whenever required. Therefore, we will create roles and assign the required policies.
The role is created and the following policies are attached.
arn:aws:iam::aws:policy/AmazonEKSClusterPolicy — This policy provides Kubernetes with the permissions it requires to manage resources on your behalf. Kubernetes requires Ec2:CreateTags permissions to place identifying information on EC2 resources including but not limited to Instances, Security Groups, and Elastic Network Interfaces.
arn:aws:iam::aws:policy/AmazonEKSServicePolicy — This policy allows Amazon Elastic Container Service for Kubernetes to create and manage the necessary resources to operate EKS Clusters.
Then we create a security group for the EKS cluster which can communicate with all its worker nodes and only the host system used to create the cluster can communicate to it. This has been done using the IP we stored in the http local.
In the next step, we will create our EKS cluster with the required connectivity and roles which will then connect to worker nodes.
Similarly, the worker nodes also require some powers. Hence we will create a role with the following policies as well.
AmazonEC2ContainerRegistryReadOnly — Provides read-only access to Amazon EC2 Container Registry repositories.
AmazonEKSWorkerNodePolicy — This policy allows Amazon EKS worker nodes to connect to Amazon EKS Clusters.
AmazonEKS_CNI_Policy — This policy provides the Amazon VPC CNI Plugin (amazon-vpc-cni-k8s) the permissions it requires to modify the IP address configuration on your EKS worker nodes. This permission set allows the CNI to list, describe, and modify Elastic Network Interfaces on your behalf. More information on the AWS VPC CNI Plugin is available here: https://github.com/aws/amazon-vpc-cni-k8s
Next, we will create a generate a key pair locally, store the public key in a local file and push the public key to AWS to create a keypair on AWS. This keypair will be attached to worker nodes so we can connect to them via SSH if required.
Then we will provision our EKS worker node groups which in turn will provide nodes in the required cluster. We can define what type of worker node we require, including their instance types, disk sizes etc. as per the requirements.
This complete setup of Cluster and Nodes is highly available and has many other features which are provided by AWS themselves.
This file will give us the methods of configuring our local system to connect to the cluster and nodes. You can configure using configmap and edit in the kubeconfig file.
I have also added in a small feature which will auto-configure your kubectl. I have set interpreter as Powershell since I am working on windows. It can be changed as per the Operating System.
Now we will configure EFS on AWS. Firstly we will create a security group for the same which is open for the internet in both ways (not recommended — edit the CIDR as per requirement). Then we will configure for the file system. We will create it, set its mount targets and the access points as per our needs.
Setting up EFS Provisioner and Prometheus
We have used AWS provider to use data sources to get the required values automatically that are to be used in the setup.
Kubernetes provider has been initialised as well to work inside the cluster.
We have also created a namespace in our cluster where are efs provisioner will be set up and Prometheus pods will run as well.
We have to grant some permissions for efs provisioning to work on EKS as PVC. It is necessary to provide a unique name for it. Then we give the ClusterRole details where the subjects (for eg. permissions) will be added to and the last is the subjects itself that are to be added which in our case is the ServiceAccount.
Now we will be creating the deployment for EFS provisioner. First, we will create a StorageClass in the cluster that will help the cluster decide which type of storage does it offer like gp2, efs etc.
Then we will create a deployment with a provided efs-provisioner image. We can create as many replicas as we require and keep the deployment strategy as Recreate. The env variables that we have passed are necessary for the deployment to work and we have picked the values using AWS data sources. In the last, we will give it the mount points for the pods.
After this deployment now our EFS based PVC is ready to be used.
To test this we will launch Prometheus as well which will be using this storage.
First, we will create a PVC of the required size and access mode as ReadWriteMany from our storage class that uses EFS.
Next, we will create a persistent configuration for the Prometheus pods using a config map and give the required points. For now, we configured it for localhost and for kube-state-metrics that will be created later on to monitor our cluster and its nodes.
And finally, we will configure our deployment. We have used a self-created image here that is available at DockerHub. We can use the official Prometheus image as well and change the configurations accordingly. Both the configuration file as well as the metrics that will be monitored will remain persistent in the PVC created previously as well as in the EFS.
Setting up Kube State Metrics (via Manifest files)
What is it?
Kube State metrics is s service which talks to Kubernetes API server to get all the details about all the API objects like deployments, pods, daemonsets etc. Basically it provides Kubernetes API object metrics which you cannot get directly from native Kubernetes monitoring components.
Kube state metrics service exposes all the metrics on /metrics URI. Prometheus can scrape all the metrics exposed by Kube state metrics.
We will be using the following repository.
Run the following commands and your setup will be provisioned and running:
git clone https://github.com/devopscube/kube-state-metrics-configs.gitkubectl apply -f kube-state-metrics-configs/
As we can see now our Kube State Metrics endpoint is up and running.
Setting up Grafana (via Helm)
What is it?
Grafana is open-source visualization and analytics software. It allows you to query, visualize, alert on, and explore your metrics no matter where they are stored. In plain English, it provides you with tools to turn your time-series database (TSDB) data into beautiful graphs and visualizations which ease the job of monitoring our system and services.
Helm is a tool that streamlines installing and managing Kubernetes applications. Think of it like Apt/Yum/Homebrew for K8S. It can help to deploy something as small as a pod and as big as a full WebApp Stack with servers, databases, and all resources.
Run the following commands and your grafana will be provisioned and running:
kubectl -n kube-system create serviceaccount tillerkubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tillerhelm repo add bitnami https://charts.bitnami.com/bitnamihelm init --service-account tiller --upgrade# wait for tiller pod to starthelm install --name graf bitnami/grafana --namespace grafana-ns --set admin.user=admin --set admin.password=qwerty --set presistence.storageClass=gp2 --set service.type=LoadBalancer --set metrics.enabled=true --set metrics.serviceMonitor.namespace=tf-prometheus-configmap
For a description of the deployment, refer
- Setup the Service Account for Tiller in the kube-system namespace. Tiller is the service that actually communicates with the Kubernetes API to manage our Helm packages.
- Creating a Cluster Role Binding for Tiller.
- Add the Repository.
- Initialise Helm and upgrade the Tiller account.
