Kubernetes Monitoring with Grafana Cloud

Introduction Setting Up Grafana Cloud Create Access Policies Configure Terraform Provider Create a Grafana Stack Create Access Policy for Grafana Alloy Expose Connection Details Deploy the k8s-monitoring Helm Chart Understanding Grafana Alloy What the k8s-monitoring Helm Chart Deploys Deploying with Terraform Customizing Your Monitoring Pipeline Managing Costs

Introduction

This guide demonstrates how to create a scalable, centralized monitoring solution for your Kubernetes clusters using a push-based architecture. We'll use three key components:

Grafana Cloud — an observability platform that stores and visualizes all our telemetry data.

Grafana Alloy — a flexible telemetry collection agent designed to discover, collect, process, and forward observability data from various sources:

K8s-monitoring Helm chart — a pre-configured setup for deploying monitoring components in Kubernetes clusters, simplifying the deployment and configuration of Grafana Alloy.

With this architecture, you can collect logs, metrics, traces, and more across multiple Kubernetes clusters and centralize it in one place:

This diagram shows how metrics can be pulled from applications in a Kubernetes cluster and pushed to Grafana Cloud or its self-hosted alternative

Setting Up Grafana Cloud

Let's start by configuring Grafana Cloud to receive our monitoring data.

Create Access Policies

Firstly, you need to configure access policies in Grafana Cloud to allow Terraform to manage resources. Access policies define what actions can be performed on which resources in Grafana Cloud. Sign up at Grafana Cloud, then go to “My Account” → “Security” → “Access Policies”, and create terraform-access-policy that grants Terraform the permissions needed to create and manage resources in Grafana Cloud:

For production use, you should define more specific permissions

Configure Terraform Provider

Next, we'll set up the Grafana Terraform provider to interact with Grafana Cloud:


provider "grafana" {
  alias                     = "cloud"
  cloud_access_policy_token = var.grafana_cloud_access_policy_token
}

Create a Grafana Stack

Now let's create a Grafana stack which is a logical grouping of services like metrics, logs, traces, etc.:


resource "grafana_cloud_stack" "this" {
  name        = "Your name for Grafana Cloud Stack"
  slug        = "yourgcloudstack"
  region_slug = "us"
}

# For access to your Grafana stack
resource "grafana_cloud_stack_service_account" "cloud_sa" {
  provider = grafana.cloud

  stack_slug  = resource.grafana_cloud_stack.this[0].slug
  name        = "cloud service account"
  role        = "Admin"
  is_disabled = false
}

# Token for authenticating the service account
resource "grafana_cloud_stack_service_account_token" "cloud_sa" {
  provider = grafana.cloud

  stack_slug         = resource.grafana_cloud_stack.this[0].slug
  name               = "terraform service account key"
  service_account_id = grafana_cloud_stack_service_account.cloud_sa[0].id
}

Create Access Policy for Grafana Alloy

Also, we need to allow Grafana Alloy to send data to our stack. This policy defines the specific permissions (scopes) that Alloy needs to push metrics, logs, and traces:


resource "grafana_cloud_access_policy" "grafana_alloy" {
  provider = grafana.cloud

  region       = resource.grafana_cloud_stack.this[0].region_slug
  name         = "grafana-alloy-access-policy"
  display_name = "Grafana Alloy Access Policy (created by Terraform)"

  scopes = [
    "metrics:write",
    "metrics:import",
    "logs:write",
    "traces:write",
  ]
  
  # Define which resources the policy applies to
  realm {
    type       = "stack"
    identifier = resource.grafana_cloud_stack.this[0].id
  }
}

resource "grafana_cloud_access_policy_token" "grafana_alloy" {
  provider = grafana.cloud

  region           = resource.grafana_cloud_stack.this[0].region_slug
  access_policy_id = grafana_cloud_access_policy.grafana_alloy[0].policy_id
  name             = "grafana-alloy-access-token"
  display_name     = "Grafana Alloy Access Token (created by Terraform)"
}

Expose Connection Details

Now we have all the necessary tokens and URLs to push data from your Kubernetes clusters into Grafana Cloud's stack via Grafana Alloy. These outputs provide the connection details needed for your monitoring setup:


output "alloy_access_token" {
  description = "Grafana Alloy access token."
  value       = grafana_cloud_access_policy_token.grafana_alloy[0].token
  sensitive   = true
}

output "prometheus_url" {
  description = "Grafana Cloud Prometheus URL."
  value       = resource.grafana_cloud_stack.this[0].prometheus_url
}

output "prometheus_user_id" {
  description = "Grafana Cloud Prometheus user ID."
  value       = resource.grafana_cloud_stack.this[0].prometheus_user_id
}

output "loki_url" {
  description = "Grafana Cloud Loki URL."
  value       = resource.grafana_cloud_stack.this[0].logs_url
}

output "loki_user_id" {
  description = "Grafana Cloud Loki user ID."
  value       = resource.grafana_cloud_stack.this[0].logs_user_id
}

This Terraform configuration defines outputs for Prometheus (Mimir) and Loki, providing user IDs and endpoint URLs. While Grafana Cloud also supports Tempo for tracing, k6 for load testing, and more, this setup focuses solely on metrics and logs to keep things simple.

Deploy the k8s-monitoring Helm Chart

With Grafana Cloud configured, we can now set up monitoring in our Kubernetes clusters.

⚠️

To deploy Helm charts with Terraform, you'll need to configure the Helm provider. See the Terraform Helm provider documentation for setup instructions.

📌

While this guide shows deploying the Helm chart via Terraform for simplicity and to illustrate the concept, in production environments you might prefer using ArgoCD with Kustomize for managing Helm charts (this approach offers better separation of concerns and more granular control over Kubernetes resources).

Understanding Grafana Alloy

Grafana Alloy uses a configuration language called River (similar to HCL syntax in Terraform) to define how telemetry data is collected and processed. While you could manually configure it as shown below, this quickly becomes complex for production environments:


//@ Prometheus Scrapping
// Discover Kubernetes pods to collect metrics from
// https://grafana.com/docs/agent/latest/flow/reference/components/discovery.kubernetes
discovery.kubernetes "pod_metrics" {
  role = "pod"
  // Limit namespaces where pods are discovered
  namespaces {
    own_namespace = false  // whether you want to search for Pods in the Namespace Grafana Agent Flow is running in
    names         = ["default", "kube-system", "monitoring", "celery"]
  }
}

// Expose pod labels as metric labels
// https://grafana.com/docs/agent/latest/flow/reference/components/discovery.relabel
discovery.relabel "pod_metrics" {
  targets = discovery.kubernetes.pod_metrics.targets
  rule {
    action = "labelmap"
    regex = "__meta_kubernetes_pod_label_(.+)"
  }
  rule {
    action = "replace"
    source_labels = ["__meta_kubernetes_pod_node_name"]
    target_label = "node"
  }
  rule {
    action = "replace"
    replacement = "${cluster_name}"
    target_label = "cluster"
  }
}

// Scrape metrics from Kubernetes pods and send to a prometheus.remote_write component
// https://grafana.com/docs/agent/latest/flow/reference/components/prometheus.scrape
prometheus.scrape "pod_metrics" {
  targets    = discovery.relabel.pod_metrics.output
  forward_to = [prometheus.remote_write.default.receiver]
  honor_labels = true
}

// Collect and send metrics to a Prometheus remote_write endpoint
// https://grafana.com/docs/agent/latest/flow/reference/components/prometheus.remote_write
prometheus.remote_write "default" {
    // If you have more than one endpoint to write metrics to, repeat the endpoint block for additional endpoints
    endpoint {
        url = "${prometheus_url}"
        basic_auth {
            username = "${prometheus_user_id}"
            password = "${grafana_agent_access_token}"
        }
    }
}

//@ Logs Scrapping
// Discover Kubernetes pods to collect logs from
// https://grafana.com/docs/agent/latest/flow/reference/components/discovery.kubernetes
discovery.kubernetes "pod_logs" {
  role = "pod"
  // Limit namespaces where pods are discovered
  namespaces {
    own_namespace = false  // whether you want to search for Pods in the Namespace Grafana Agent Flow is running in
    names         = ["celery"]
  }
}

// Apply relabeling for log scraping
// https://grafana.com/docs/agent/latest/flow/reference/components/discovery.relabel
discovery.relabel "pod_logs" {
  targets = discovery.kubernetes.pod_logs.targets
  rule {
    action = "labelmap"
    regex = "__meta_kubernetes_pod_label_(.+)"
  }
  rule {
    action        = "replace"
    source_labels = ["__meta_kubernetes_pod_node_name"]
    target_label  = "__host__"
  }
  rule {
    action        = "replace"
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }
  rule {
    action        = "replace"
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label  = "pod"
  }
  rule {
    action        = "replace"
    source_labels = ["__meta_kubernetes_container_name"]
    target_label  = "container"
  }
  rule {
    action        = "replace"
    replacement   = "/var/log/pods/*$1/*.log"
    separator     = "/"
    source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"]
    target_label  = "__path__"
  }
}

// Tails logs from Kubernetes containers using the Kubernetes API
// https://grafana.com/docs/agent/latest/flow/reference/components/discovery.relabel
loki.source.kubernetes "pods" {
  targets    = discovery.relabel.pod_logs.output
  forward_to = [loki.write.default.receiver]
}

// Receives log entries from other loki components and sends them over the network using Loki’s logproto format
// https://grafana.com/docs/agent/latest/flow/reference/components/loki.write/
loki.write "default" {
  endpoint {
    url = "${loki_url}"
    basic_auth {
      username = "${loki_user_id}"
      password = "${grafana_agent_access_token}"
    }
  }
}

Data collection pipeline that automatically discovers, relabels, and forwards metrics (Prometheus) and logs (Loki) from Kubernetes pods to Grafana Cloud endpoints using Grafana Alloy.

Here's how you would deploy such a configuration using Terraform:


resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

locals {
  grafana_agent_config = templatefile("${path.module}/config/grafana_agent_config.river.tpl", {
    cluster_name           = "your_cluster_n"
    grafana_agent_access_token = var.grafana_agent_access_token
    prometheus_url             = var.prometheus_url
    prometheus_user_id         = var.prometheus_user_id
    loki_url                   = var.loki_url
    loki_user_id               = var.loki_user_id
  })
}

# https://artifacthub.io/packages/helm/grafana/grafana-agent
# https://github.com/grafana/agent/tree/main/operations/helm/charts/grafana-agent
resource "helm_release" "grafana_agent" {
  repository = "https://grafana.github.io/helm-charts"
  chart      = "grafana-agent"
  version    = "~> 0.29.0"
  name       = "grafana-agent"
  namespace  = kubernetes_namespace.monitoring.metadata[0].name
  values = [yamlencode({
    agent = {
      configMap = {
        content = local.grafana_agent_config
      }
    }
  })]
}

This approach above can quickly become tedious and time-consuming to write and maintain. That's why, instead of writing this complex configuration by hand, we'll use the k8s-monitoring Helm chart.

What the k8s-monitoring Helm Chart Deploys

The k8s-monitoring chart deploys several components to provide comprehensive monitoring of your Kubernetes cluster like:

Prometheus for metrics collection.

Loki for log collection.

Tempo for distributed tracing.

Pyroscope for continuous profiling.

Core Kubernetes monitoring components like:
- kube-state-metrics for exposing K8s object state metrics to show cluster state.
- node-exporter for hardware metrics.
- kubelet for exposing node and container runtime metrics.
- cAdvisor for container resource usage metrics like CPU, memory, network, and disk.

and even more (check out their GitHub repo).

Examples for the k8s-monitoring Helm chart might look like:


cluster:
  # -- The name of this cluster, which will be set in all labels. Required.
  name: ${cluster_name}

externalServices:
  prometheus:
    # -- Prometheus host where metrics will be sent
    host: "${prometheus_host}"
    basicAuth:
      username: "${prometheus_username}"
      password: "${grafana_agent_access_token}"

  loki:
    # -- Loki host where logs and events will be sent
    host: "${loki_host}"
    basicAuth:
      username: "${loki_username}"
      password: "${grafana_agent_access_token}"

# Settings related to capturing and forwarding metrics
metrics:
  # -- Capture and forward metrics
  enabled: true

  # -- How frequently to scrape metrics
  scrapeInterval: 60s

  # Annotation-based autodiscovery allows for discovering metric sources solely on their annotations and does
  # not require adding any extra configuration.
  autoDiscover:
    # Enable annotation-based autodiscovery
    enabled: true

    # -- Annotations that are used to discover and configure metric scraping targets. Add these annotations
    # to your services or pods to control how autodiscovery will find and scrape metrics from your service or pod.
    annotations:
      # -- Annotation for enabling scraping for this service or pod. Value should be either "true" or "false"
      scrape: "k8s.grafana.com/scrape"
      # -- Annotation for setting or overriding the metrics path. If not set, it defaults to /metrics
      metricsPath: "k8s.grafana.com/metrics.path"
      # -- Annotation for setting the metrics port by number.
      metricsPortNumber: "k8s.grafana.com/metrics.portNumber"

  # Metrics from Grafana Alloy
  alloy:
    enabled: false

  # Cluster object metrics from Kube State Metrics
  kube-state-metrics:
    enabled: true

  # Node metrics from Node Exporter
  node-exporter:
    enabled: true

  # Cluster metrics from the Kubelet
  kubelet:
    enabled: true
    
  # Container metrics from cAdvisor
  cadvisor:
    enabled: true

  # Metrics from the API Server
  apiserver:
    enabled: false

  # Metrics from the Kube Controller Manager
  kubeControllerManager:
    enabled: false

  # Metrics from the Kube Proxy
  kubeProxy:
    enabled: false

  # Metrics from the Kube Scheduler
  kubeScheduler:
    enabled: false

  # Cost related metrics from OpenCost
  cost:
    enabled: false

# Settings related to capturing and forwarding logs
logs:
  enabled: true

  # Settings for Kubernetes pod logs
  pod_logs:
    enabled: true

    # Controls the behavior of discovering pods for logs.
    # When set to "all", every pod (filtered by the namespaces list below) will have their logs gathered, but you can
    # use the annotation to remove a pod from that list.
    # When set to "annotation", only pods with the annotation set to true will be gathered.
    # Possible values: "all" "annotation"
    discovery: "all"

    # The annotation to control the behavior of gathering logs from this pod. If you put this annotation on to your pod,
    # it will either enable or disable auto gathering of logs from this pod.
    annotation: "k8s.grafana.com/logs.autogather"

  # Settings for scraping Kubernetes cluster events
  cluster_events:
    enabled: false

# Settings related to capturing and forwarding traces
traces:
  enabled: false

# Settings for the Node Exporter deployment
# You can use this sections to make modifications to the Node Exporter deployment.
# See https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-node-exporter for available values.
prometheus-node-exporter:
  # -- Should this helm chart deploy Node Exporter to the cluster.
  # Set this to false if your cluster already has Node Exporter, or if you do
  # not want to scrape metrics from Node Exporter.
  enabled: true

  tolerations:
    - effect: NoSchedule
      operator: Exists
    - effect: NoExecute
      operator: Exists

# Settings for the Grafana Alloy instance that gathers pod logs.
# See https://github.com/grafana/alloy/tree/main/operations/helm/charts/alloy for available values.
# @ignored -- This skips including these values in README.md
alloy-logs:
  controller:
    type: daemonset
    nodeSelector:
      kubernetes.io/os: linux

    # Allow this pod to be scheduled on GPU nodes
    tolerations:
      - effect: NoSchedule
        operator: Exists
      - effect: NoExecute
        operator: Exists

Deploying with Terraform

Here's how to deploy the chart using Terraform:


resource "kubernetes_namespace" "monitoring" {
  metadata {
    name = "monitoring"
  }
}

resource "helm_release" "k8s_monitoring" {
  repository = "https://grafana.github.io/helm-charts"
  chart      = "k8s-monitoring"
  version    = "~> 1.6.13"

  name      = "k8s-monitoring"
  namespace = kubernetes_namespace.monitoring.metadata[0].name

  values = [
    templatefile("${path.module}/config/k8s_monitoring.yaml.tpl", {
      cluster_name                = "your_cluster"
      grafana_access_policy_token = var.grafana_access_policy_token
      grafana_prometheus_host     = var.grafana_prometheus_host
      grafana_prometheus_username = var.grafana_prometheus_username
      grafana_loki_host           = var.grafana_loki_host
      grafana_loki_username       = var.grafana_loki_username
    })
  ]
}

After deploying, you'll see several pods in your monitoring namespace:


$ kubectl get pods -n monitoring 
NAME                                                 READY   STATUS    RESTARTS   AGE
k8s-monitoring-alloy-0                               2/2     Running   0          4d2h
k8s-monitoring-alloy-logs-6qgz9                      2/2     Running   0          4d2h
k8s-monitoring-alloy-logs-8mpqd                      2/2     Running   0          4d1h
k8s-monitoring-alloy-logs-gdj2b                      2/2     Running   0          4d2h
k8s-monitoring-alloy-logs-ljg9b                      2/2     Running   0          4d2h
k8s-monitoring-alloy-logs-tnn4d                      2/2     Running   0          4d1h
k8s-monitoring-alloy-logs-vglxx                      2/2     Running   0          4d2h
k8s-monitoring-alloy-logs-vrldw                      2/2     Running   0          4d1h
k8s-monitoring-kube-state-metrics-55f4b6bdd7-pswx5   1/1     Running   0          4d2h
k8s-monitoring-prometheus-node-exporter-54nhd        1/1     Running   0          4d2h
k8s-monitoring-prometheus-node-exporter-nfhv4        1/1     Running   0          4d2h
k8s-monitoring-prometheus-node-exporter-pbh6l        1/1     Running   0          4d2h
k8s-monitoring-prometheus-node-exporter-qj86w        1/1     Running   0          4d2h
k8s-monitoring-prometheus-node-exporter-rlhmj        1/1     Running   0          4d2h
k8s-monitoring-prometheus-node-exporter-tpdz9        1/1     Running   0          4d2h
k8s-monitoring-prometheus-node-exporter-wwm7s        1/1     Running   0          4d2h

Monitoring components in the monitoring namespace, including Grafana Alloy (metrics and logs), kube-state-metrics, and node exporter.

Customizing Your Monitoring Pipeline

When implementing monitoring across different environments and clusters, you'll often need to customize how metrics are collected and labeled. The k8s-monitoring Helm chart allows you to apply these customizations without writing complex River configurations directly. You can implement them by adding extra relabeling rules to your values:


# Settings related to capturing and forwarding metrics
metrics:
  # -- Rule blocks to be added to the discovery.relabel component for all metric sources.
  # See https://grafana.com/docs/agent/latest/flow/reference/components/discovery.relabel/#rule-block
  extraRelabelingRules: |-
    rule {
      action = "replace"
      source_labels = ["__meta_kubernetes_pod_node_name"]
      target_label = "node"
    }
    rule {  // NB(khaykingleb): to differentiate the environments of pods in the GPU cluster
      action = "replace"
      source_labels = ["__meta_kubernetes_pod_label_Env"]
      target_label = "env"
    }

This configuration assigns a node label based on the pod’s Kubernetes node and an env label to differentiate environments in a GPU cluster, enhancing observability in Grafana Alloy.

These rules follow the same syntax as River configuration but are applied within the managed Helm chart deployment. This gives you the flexibility of the River language while keeping deployment simple and maintainable.

Managing Costs

Monitoring in the cloud comes with costs, especially when collecting data from large clusters. Here are some strategies to keep expenses under control:

Start small: enable only essential namespaces/components, begin by monitoring only critical applications and infrastructure components.

Monitor usage: regularly check the usage dashboards in Grafana Cloud to understand how much data you're ingesting and which metrics are taking up the most space.

Filter metrics: drop logs or metrics you don't need.

⚠️

Selective metric/logs shipping is generally an anti-pattern as you risk losing visibility into problems. If Grafana Cloud ingestion costs are a concern, the better approach is to deploy a dedicated monitoring cluster and forward all data there through Grafana Alloy. That way you keep all metrics/logs available, while controlling cost.

Use the metricsTuning settings of the k8s-monitoring chart to limit which metrics are collected:


# Cluster object metrics from Kube State Metrics
kube-state-metrics:
  # Adjustments to the scraped metrics to filter the amount of data sent to storage.
  metricsTuning:
    # -- Filter the list of metrics from Kube State Metrics to a useful, minimal set. See https://github.com/grafana/k8s-monitoring-helm/blob/85d679e23af4e79eeaae5f2207237e30fef06ff8/charts/k8s-monitoring/README.md#allow-list-for-kube-state-metrics
    useDefaultAllowList: true
    # -- Metrics to keep. Can use regex.
    includeMetrics:
      - kube_pod_status_qos_class
      - kube_namespace_created
      - kube_deployment_status_replicas_unavailable
      - kube_pod_container_status_restarts_total
      - kube_node_labels
    # -- Metrics to drop. Can use regex.
    excludeMetrics:
      - kube_lease_owner
      - kube_lease_renew_time
      - kube_pod_tolerations
      - kube_pod_status_ready
      - kube_pod_status_scheduled
      - kube_pod_owner
      - kube_pod_start_time
      - kube_pod_container_state_started
      - kube_node_status_allocatable
      - kube_node_spec_unschedulable
      - kube_pod_created
      - kube_pod_ips
      - kube_pod_restart_policy
      - kube_pod_service_account
      - kube_pod_status_initialized_time
      - kube_pod_status_scheduled_time
      - kube_pod_status_container_ready_time
      - kube_pod_status_ready_time
      - kube_horizontalpodautoscaler_status_condition
      - kube_replicaset_created
      - kube_replicaset_metadata_generation
      - kube_replicaset_spec_replicas
      - kube_replicaset_status_fully_labeled_replicas
      - kube_replicaset_status_observed_generation
      - kube_replicaset_status_ready_replicas
      - kube_replicaset_status_replicas
      - kube_daemonset_created
      - kube_daemonset_metadata_generation
      - kube_deployment_status_condition
      - kube_namespace_status_phase
      - kube_endpoint_ports
      - kube_endpoint_address
      - kube_secret_created
      - kube_secret_metadata_resource_version

# Node metrics from Node Exporter
node-exporter:
  # Adjustments to the scraped metrics to filter the amount of data sent to storage.
  metricsTuning:
    # -- Filter the list of metrics from Node Exporter to the minimal set required for Kubernetes Monitoring. See https://github.com/grafana/k8s-monitoring-helm/blob/85d679e23af4e79eeaae5f2207237e30fef06ff8/charts/k8s-monitoring/README.md#allow-list-for-node-exporter
    useDefaultAllowList: true
    # -- Filter the list of metrics from Node Exporter to the minimal set required for Kubernetes Monitoring as well as the Node Exporter integration.
    useIntegrationAllowList: false
    # -- Metrics to keep. Can use regex.
    includeMetrics:
      - node_uname_info
      - node_time_seconds
      - node_boot_time_seconds
      - node_cpu_core_throttles_total
      - node_load1
      - node_context_switches_total
      - node_filefd_maximum
      - node_timex_estimated_error_seconds
      - node_network_receive_bytes_total
      - node_network_receive_errs_total
      - node_network_receive_packets_total
      - node_network_receive_drop_total
      - node_netstat_Tcp_CurrEstab
      - node_nf_conntrack_entries
      - node_disk_read_bytes_total
      - node_disk_written_bytes_total
      - node_disk_reads_completed_total
      - node_disk_writes_completed_total
      - node_disk_io_now
    # -- Metrics to drop. Can use regex.
    excludeMetrics:
      - node_filesystem_readonly
      - node_filesystem_free_bytes
      - node_scrape_collector_duration_seconds
      - node_scrape_collector_success
      - node_cpu_guest_seconds_total

# Cluster metrics from the Kubelet
kubelet:
  # Adjustments to the scraped metrics to filter the amount of data sent to storage.
  metricsTuning:
    # -- Filter the list of metrics from the Kubelet to the minimal set required for Kubernetes Monitoring. See https://github.com/grafana/k8s-monitoring-helm/blob/85d679e23af4e79eeaae5f2207237e30fef06ff8/charts/k8s-monitoring/README.md#allow-list-for-kubelet
    useDefaultAllowList: true
    # -- Metrics to keep. Can use regex.
    includeMetrics: []
    # -- Metrics to drop. Can use regex.
    excludeMetrics:
      - kubelet_pod_worker_duration_seconds_bucket
      - kubelet_cgroup_manager_duration_seconds_bucket
      - kubelet_runtime_operations_total
      - kubelet_pleg_relist_duration_seconds_bucket
      - kubelet_pleg_relist_interval_seconds_bucket
      - kubelet_pod_start_duration_seconds_bucket
      - rest_client_requests_total
      - storage_operation_duration_seconds_count
      - volume_manager_total_volumes

# Container metrics from cAdvisor
cadvisor:
  # Adjustments to the scraped metrics to filter the amount of data sent to storage.
  metricsTuning:
    # -- Filter the list of metrics from cAdvisor to the minimal set required for Kubernetes Monitoring. See https://github.com/grafana/k8s-monitoring-helm/blob/85d679e23af4e79eeaae5f2207237e30fef06ff8/charts/k8s-monitoring/README.md#allow-list-for-cadvisor
    useDefaultAllowList: true
    # -- Metrics to keep. Can use regex.
    includeMetrics:
      - machine_cpu_cores
      - container_oom_events_total
      - container_network_receive_errors_total
      - container_cpu_cfs_throttled_seconds_total
    # -- Metrics to drop. Can use regex.
    excludeMetrics:
      - container_memory_cache
      - container_memory_swap
      - container_fs_reads_total
      - container_fs_writes_total
      - container_fs_reads_bytes_total
      - container_fs_writes_bytes_total

Limit namespaces from which you collect logs:


logs:
  pod_logs:
    # -- Only capture logs from pods in these namespaces (`[]` means all namespaces)
    namespaces:
      - celery
      - supabase

<Kubernetes Monitoring with Grafana Cloud

Contents