Prometheus + Grafana 监控体系

修改记录原文

#Prometheus #Grafana #监控 #Alertmanager #Kubernetes

本页目录

场景
前置条件
配置
坑

场景

生产环境 K8s 集群需要全方位可观测性：集群资源、应用指标、业务指标、告警通知、仪表盘可视化。

栈选择：Prometheus（采集+存储+查询）+ Grafana（可视化）+ Alertmanager（告警路由去重）+ kube-state-metrics + node-exporter + kube-prometheus-stack（Helm 一键部署）。

前置条件

K8s 1.20+，Helm 3
集群资源够：Prometheus 约 2C4G+50G 存储 / Grafana 约 500M
存储类支持 PVC（Prometheus 需持久化）
域名 + TLS（Grafana 对外访问需 HTTPS）

配置

1. 一键部署 kube-prometheus-stack

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# 创建命名空间
kubectl create namespace monitoring

# 生产环境推荐配置
helm install kps prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace \
  --version 45.0.0 \
  -f values-prod.yaml

values-prod.yaml 关键配置

# prometheus 配置
prometheus:
  prometheusSpec:
    retention: 30d              # 保留 30 天
    retentionSize: 50GB         # 最大 50G
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: ssd     # 必须有快速存储类
          resources:
            requests:
              storage: 100Gi
    resources:
      requests:
        cpu: 1000m
        memory: 4Gi
      limits:
        cpu: 2000m
        memory: 8Gi
    # 远程存储（可选，长期保留）
    # remoteWrite:
    #   - url: https://thanos-receive.example.com/api/v1/receive

  # 服务发现额外配置
  additionalScrapeConfigs:
    - job_name: 'myapp'
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true

# alertmanager 配置
alertmanager:
  alertmanagerSpec:
    replicas: 2
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: ssd
          resources:
            requests:
              storage: 10Gi
    config:
      global:
        resolve_timeout: 5m
      route:
        group_by: ['alertname', 'namespace']
        group_wait: 30s
        group_interval: 5m
        repeat_interval: 4h
        receiver: 'default'
        routes:
          - match:
              severity: critical
            receiver: 'pagerduty'
            continue: true
          - match:
              severity: warning
            receiver: 'slack'
      receivers:
        - name: 'default'
          email_configs:
            - to: 'ops@example.com'
              send_resolved: true
        - name: 'slack'
          slack_configs:
            - api_url: 'https://hooks.slack.com/services/xxx'
              channel: '#alerts'
              send_resolved: true
        - name: 'pagerduty'
          pagerduty_configs:
            - service_key: 'xxx'

# grafana 配置
grafana:
  enabled: true
  persistence:
    type: pvc
    storageClassName: ssd
    size: 10Gi
  adminPassword: "StrongPass123"
  sidecar:
    dashboards:
      enabled: true
      searchNamespace: ALL
    datasources:
      enabled: true
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
        - name: 'default'
          orgId: 1
          folder: 'Kubernetes'
          type: file
          disableDeletion: false
          updateIntervalSeconds: 10
          allowUiUpdates: true
          options:
            path: /var/lib/grafana/dashboards/default

  # 预装常用仪表盘
  dashboards:
    default:
      k8s-cluster:
        gnetId: 315
        revision: 3
        datasource: Prometheus
      k8s-pods:
        gnetId: 6417
        datasource: Prometheus
      node-exporter:
        gnetId: 1860
        datasource: Prometheus

# node-exporter（主机指标）
prometheus-node-exporter:
  enabled: true
  resources:
    requests:
      cpu: 100m
      memory: 128Mi

# kube-state-metrics（K8s 对象状态）
kubeStateMetrics:
  enabled: true

# kubelet 监控
kubelet:
  enabled: true
  serviceMonitor:
    enabled: true

# 安装
helm install kps prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace \
  -f values-prod.yaml --version 45.0.0

2. 验证部署

# 检查组件
kubectl get pods -n monitoring

# Prometheus UI
kubectl port-forward -n monitoring svc/kps-kube-prometheus-prometheus 9090:9090
# 浏览器访问 http://localhost:9090

# Grafana
kubectl port-forward -n monitoring svc/kps-grafana 3000:80
# 用户 admin / StrongPass123
# 浏览器访问 http://localhost:3000

3. 域名 + HTTPS（生产环境）

# ingress-grafana.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana
  namespace: monitoring
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - grafana.example.com
      secretName: grafana-tls
  rules:
    - host: grafana.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: kps-grafana
                port:
                  number: 80

kubectl apply -f ingress-grafana.yaml

3. 常用 PromQL（复制即用）

集群资源概览

# 集群 CPU 使用率
sum(rate(container_cpu_usage_seconds_total{container!="",container!="POD"}[5m])) 
/ sum(kube_node_status_capacity{resource="cpu"}) * 100

# 集群内存使用率
sum(container_memory_working_set_bytes{container!="",container!="POD"}) 
/ sum(kube_node_status_capacity{resource="memory"}) * 100

# 节点 CPU 使用率
sum by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m])) 
/ sum by (instance) (rate(node_cpu_seconds_total[5m])) * 100

# Pod 重启次数
increase(kube_pod_container_status_restarts_total[1h])

# Pod OOMKilled
kube_pod_container_status_terminated_reason{reason="OOMKilled"} == 1

# PVC 使用率
kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes * 100

应用指标（需应用暴露 /metrics）

# 请求 QPS
sum(rate(http_requests_total{job="myapp"}[5m])) by (path, method)

# 请求延迟 P99
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job="myapp"}[5m])) by (le, path))

# 错误率
sum(rate(http_requests_total{job="myapp",code=~"5.."}[5m])) 
/ sum(rate(http_requests_total{job="myapp"}[5m]))

4. Grafana 仪表盘导入

# UI 导入：+ Import → Dashboard JSON → 上传 JSON 文件
# 或配置 sidecar 自动发现

# 常用 Dashboard ID（Grafana.com）
# 315: Kubernetes Cluster Overview
# 315: Kubernetes Cluster Monitoring
# 6417: Kubernetes Pods
# 1860: Node Exporter Full
# 13948: Kubernetes / Compute Resources / Cluster
# 13946: Kubernetes / Compute Resources / Namespace
# 14282: Kubernetes / Network
# 15661: Kubernetes / Persistent Volumes
# 11074: etcd
# 12559: Kubernetes / API Server
# 11078: Kubernetes / Scheduler
# 11079: Kubernetes / Controller Manager
# 11102: Kubernetes / Kubelet

5. Alertmanager 告警规则（常用）

# custom-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: custom-rules
  namespace: monitoring
spec:
  groups:
    - name: k8s.rules
      rules:
        - alert: KubePodCrashLooping
          expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Pod {{ $labels.pod }} 崩溃重启"
            description: "{{ $labels.namespace }}/{{ $labels.pod }} 最近 15 分钟重启 {{ $value }} 次"

        - alert: KubePodNotReady
          expr: kube_pod_status_phase{phase="Running"} == 0
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Pod {{ $labels.pod }} 长时间 NotReady"
            description: "{{ $labels.namespace }}/{{ $labels.pod }} 超过 5 分钟 NotReady"

        - alert: KubeCPUOvercommit
          expr: sum(kube_pod_container_resource_requests_cpu_cores) / sum(kube_node_status_capacity_cpu_cores) > 1.5
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "CPU 过度承诺"
            description: "集群 CPU 请求总量超过容量 1.5 倍"

        - alert: KubeMemoryOvercommit
          expr: sum(kube_pod_container_resource_requests_memory_bytes) / sum(kube_node_status_capacity_memory_bytes) > 1.5
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "内存过度承诺"

        - alert: PVCUsageHigh
          expr: (kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes) > 0.85
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "PVC 使用率超过 85%"
            description: "{{ $labels.persistentvolumeclaim }} 使用率 {{ $value | humanizePercentage }}"

        - alert: NodeDiskPressure
          expr: kube_node_status_condition{condition="DiskPressure",status="true"} == 1
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "节点 {{ $labels.node }} 磁盘压力"

        - alert: NodeMemoryPressure
          expr: kube_node_status_condition{condition="MemoryPressure",status="true"} == 1
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "节点 {{ $labels.node }} 内存压力"

        - alert: PodOOMKilled
          expr: kube_pod_container_status_terminated_reason{reason="OOMKilled"} == 1
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: "Pod {{ $labels.pod }} OOMKilled"
            description: "{{ $labels.namespace }}/{{ $labels.pod }} 被 OOM Killer 杀掉"

    - name: node.rules
      rules:
        - alert: NodeDown
          expr: up{job="node-exporter"} == 0
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "节点 {{ $labels.instance }} 宕机"
            description: "{{ $labels.instance }} 超过 2 分钟无心跳"

        - alert: NodeCPUHigh
          expr: (1 - avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) > 0.9
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "节点 {{ $labels.instance }} CPU 持续高负载"

        - alert: NodeMemoryHigh
          expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.9
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "节点 {{ $labels.instance }} 内存使用率超过 90%"

        - alert: NodeDiskFull
          expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) < 0.1
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "节点 {{ $labels.instance }} 磁盘剩余不足 10%"

kubectl apply -f custom-rules.yaml -n monitoring

5. 推送到钉钉 / 企业微信 / 飞书

# alertmanager 配置片段
receivers:
  - name: 'dingtalk'
    webhook_configs:
      - url: 'https://oapi.dingtalk.com/robot/send?access_token=xxx'
        send_resolved: true
        http_config:
          basic_auth:
            username: ''
            password: ''
  - name: 'wechat'
    webhook_configs:
      - url: 'https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxx'
        send_resolved: true

6. Grafana 仪表盘变量（动态筛选）

{
  "type": "query",
  "name": "namespace",
  "label": "Namespace",
  "datasource": "Prometheus",
  "query": "label_values(kube_pod_info, namespace)",
  "multi": true,
  "includeAll": true,
  "allValue": ".*"
}

面板查询引用：

sum(rate(container_cpu_usage_seconds_total{namespace=~"$namespace",container!="",container!="POD"}[5m])) by (pod)

7. 长期存储（Thanos / VictoriaMetrics）

# Thanos Sidecar（对接对象存储）
prometheus:
  prometheusSpec:
    retention: 24h           # 本地只留 24h
    retentionSize: 50GB
    # 远程写入 Thanos
    remoteWrite:
      - url: https://thanos-receive.example.com/api/v1/receive

# VictoriaMetrics 单机版（轻量替代）
docker run -d --name vmsingle \
  -p 8428:8428 \
  -v /data/victoria:/victoria \
  victoriametrics/victoria-metrics \
  -retentionPeriod=90d \
  -storageDataPath=/victoria \
  -httpListenAddr=:8428

7. 常用运维操作

# Prometheus 重载配置（不重启）
curl -X POST http://prometheus:9090/-/reload

# 清理 Prometheus 数据
kubectl exec -n monitoring prometheus-0 -- rm -rf /prometheus/*
kubectl delete pod -n monitoring prometheus-0

# Grafana 重置密码
kubectl exec -n monitoring deploy/kps-grafana -- grafana-cli admin reset-admin-password NewPass123

# 导出 Grafana Dashboard
curl -H "Authorization: Bearer $TOKEN" \
  http://grafana/api/dashboards/uid/xxx > dashboard.json

# 备份 Grafana
kubectl exec -n monitoring deploy/kps-grafana -- \
  tar czf - /var/lib/grafana | gzip > grafana-backup.tar.gz

坑

坑	现象	避坑
Prometheus OOM	内存不足被 Kill	设置 `resources.limits.memory`、开启 `--storage.tsdb.retention.size`
告警风暴	一次故障发几百条告警	配置 `group_by`、`group_wait`、`group_interval`、`repeat_interval`
Grafana 看不了历史	PVC 没挂 / 没开启 persistence	`grafana.persistence.enabled=true` + PVC
告警风暴刷屏钉钉	告警没分组、没抑制	配置 `group_by`、`group_wait`、`repeat_interval`、`inhibit_rules`
Prometheus 磁盘满	磁盘满导致写入失败	设置 `retentionSize`、开启 `remoteWrite`
Grafana 看不到数据	datasource 没配 / namespace 没选	检查 Datasource 设置、namespace 变量
告警不触发	表达式写错 / for 时长太长	用 `promtool check rules` 校验、缩短 `for`
Alertmanager HA 不生效	没配 gossip / 没共享存储	配置 `cluster`、`peer`、`storage`
Grafana 看不到历史数据	没配 PVC / retention 太短	`grafana.persistence.enabled=true` + `retention`
Namespace 变量不生效	变量没绑定 datasource / label 名不对	变量 Query 用 `label_values(metric, label)`
告警恢复不发 Resolved	`send_resolved: false` / 无 Resolved 模板	`send_resolved: true` + 模板里写 `{{ if eq .Status "resolved" }}`