Kubernetes Monitoring with Grafana Cloud

Table of Contents

Push-Based Kubernetes Monitoring with Grafana Cloud

Introduction Setting Up Grafana Cloud Create Access Policies Configure Terraform Provider Create a Grafana Stack Create Access Policy for Grafana Alloy Expose Connection Details Deploy the k8s-monitoring Helm Chart Understanding Grafana Alloy What the k8s-monitoring Helm Chart Deploys Deploying with Terraform Customizing Your Monitoring Pipeline Managing Costs

Introduction

This guide demonstrates how to create a scalable, centralized monitoring solution for your Kubernetes clusters using a push-based architecture implemented through Infrastructure as Code.

We'll use three key components:

With this architecture, you can collect logs, metrics, traces, and more across multiple Kubernetes clusters and centralize it in one place:

Setting Up Grafana Cloud

Let's start by configuring Grafana Cloud to receive our monitoring data.

Create Access Policies

Firstly, you need to configure access policies in Grafana Cloud to allow Terraform to manage resources. Access policies define what actions can be performed on which resources in Grafana Cloud. Sign up at Grafana Cloud, then go to “My Account” → “Security” → “Access Policies”, and create terraform-access-policy that grants Terraform the permissions needed to create and manage resources in Grafana Cloud:

Configure Terraform Provider

Next, we'll set up the Grafana Terraform provider to interact with Grafana Cloud:

Create a Grafana Stack

Now let's create a Grafana stack—this is a logical grouping of services like metrics, logs, traces, etc.:

Create Access Policy for Grafana Alloy

Also, we need to allow Grafana Alloy to send data to our stack. This policy defines the specific permissions (scopes) that Alloy needs to push metrics, logs, and traces:

Expose Connection Details

Now we have all the necessary tokens and URLs to push data from your Kubernetes clusters into Grafana Cloud's stack via Grafana Alloy. These outputs provide the connection details needed for your monitoring setup:

Deploy the k8s-monitoring Helm Chart

With Grafana Cloud configured, we can now set up monitoring in our Kubernetes clusters.

Understanding Grafana Alloy

Grafana Alloy uses a configuration language called River (similar to HCL syntax in Terraform) to define how telemetry data is collected and processed. While you could manually configure it as shown below, this quickly becomes complex for production environments:

This approach above can quickly become tedious and time-consuming to write and maintain. That's why, instead of writing this complex configuration by hand, we'll use the k8s-monitoring Helm chart.

What the k8s-monitoring Helm Chart Deploys

The k8s-monitoring chart deploys several components to provide comprehensive monitoring of your Kubernetes cluster like:
- Prometheus for metrics collection.
- Loki for log collection.
- Tempo for distributed tracing.
- Pyroscope for continuous profiling.
- Core Kubernetes monitoring components like:
  - kube-state-metrics for exposing K8s object state metrics to show cluster state.
  - node-exporter for hardware metrics.
  - kubelet for exposing node and container runtime metrics.
  - cAdvisor for container resource usage metrics like CPU, memory, network, and disk.

Examplalues for the k8s-monitoring Helm chart might look like:

Deploying with Terraform

Here's how to deploy the chart using Terraform:

After deploying, you'll see several pods in your monitoring namespace:

Customizing Your Monitoring Pipeline

When implementing monitoring across different environments and clusters, you'll often need to customize how metrics are collected and labeled. The k8s-monitoring Helm chart allows you to apply these customizations without writing complex River configurations directly. You can implement them by adding extra relabeling rules to your values:

Managing Costs

Monitoring in the cloud comes with costs, especially when collecting data from large clusters. Here are some strategies to keep expenses under control: