Prometheus + Grafana 监控体系 · 修改记录

这次没有正文改动。

 @@ -0,0 +1,522 @@
+---
+tags: [Prometheus, Grafana, 监控, Alertmanager, Kubernetes]
+order: 30
+---
+
+# Prometheus + Grafana 监控体系
+
+## 场景
+
+生产环境 K8s 集群需要全方位可观测性：集群资源、应用指标、业务指标、告警通知、仪表盘可视化。
+
+**栈选择**：Prometheus（采集+存储+查询）+ Grafana（可视化）+ Alertmanager（告警路由去重）+ kube-state-metrics + node-exporter + kube-prometheus-stack（Helm 一键部署）。
+
+## 前置条件
+
+- K8s 1.20+，Helm 3
+- 集群资源够：Prometheus 约 2C4G+50G 存储 / Grafana 约 500M
+- 存储类支持 PVC（Prometheus 需持久化）
+- 域名 + TLS（Grafana 对外访问需 HTTPS）
+
+## 配置
+
+### 1. 一键部署 kube-prometheus-stack
+
+```bash
+helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+helm repo update
+
+# 创建命名空间
+kubectl create namespace monitoring
+
+# 生产环境推荐配置
+helm install kps prometheus-community/kube-prometheus-stack \
+  -n monitoring --create-namespace \
+  --version 45.0.0 \
+  -f values-prod.yaml
+```
+
+**values-prod.yaml 关键配置**
+
+```yaml
+# prometheus 配置
+prometheus:
+  prometheusSpec:
+    retention: 30d              # 保留 30 天
+    retentionSize: 50GB         # 最大 50G
+    storageSpec:
+      volumeClaimTemplate:
+        spec:
+          storageClassName: ssd     # 必须有快速存储类
+          resources:
+            requests:
+              storage: 100Gi
+    resources:
+      requests:
+        cpu: 1000m
+        memory: 4Gi
+      limits:
+        cpu: 2000m
+        memory: 8Gi
+    # 远程存储（可选，长期保留）
+    # remoteWrite:
+    #   - url: https://thanos-receive.example.com/api/v1/receive
+
+  # 服务发现额外配置
+  additionalScrapeConfigs:
+    - job_name: 'myapp'
+      kubernetes_sd_configs:
+        - role: pod
+      relabel_configs:
+        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
+          action: keep
+          regex: true
+
+# alertmanager 配置
+alertmanager:
+  alertmanagerSpec:
+    replicas: 2
+    storage:
+      volumeClaimTemplate:
+        spec:
+          storageClassName: ssd
+          resources:
+            requests:
+              storage: 10Gi
+    config:
+      global:
+        resolve_timeout: 5m
+      route:
+        group_by: ['alertname', 'namespace']
+        group_wait: 30s
+        group_interval: 5m
+        repeat_interval: 4h
+        receiver: 'default'
+        routes:
+          - match:
+              severity: critical
+            receiver: 'pagerduty'
+            continue: true
+          - match:
+              severity: warning
+            receiver: 'slack'
+      receivers:
+        - name: 'default'
+          email_configs:
+            - to: 'ops@example.com'
+              send_resolved: true
+        - name: 'slack'
+          slack_configs:
+            - api_url: 'https://hooks.slack.com/services/xxx'
+              channel: '#alerts'
+              send_resolved: true
+        - name: 'pagerduty'
+          pagerduty_configs:
+            - service_key: 'xxx'
+
+# grafana 配置
+grafana:
+  enabled: true
+  persistence:
+    type: pvc
+    storageClassName: ssd
+    size: 10Gi
+  adminPassword: "StrongPass123"
+  sidecar:
+    dashboards:
+      enabled: true
+      searchNamespace: ALL
+    datasources:
+      enabled: true
+  dashboardProviders:
+    dashboardproviders.yaml:
+      apiVersion: 1
+      providers:
+        - name: 'default'
+          orgId: 1
+          folder: 'Kubernetes'
+          type: file
+          disableDeletion: false
+          updateIntervalSeconds: 10
+          allowUiUpdates: true
+          options:
+            path: /var/lib/grafana/dashboards/default
+
+  # 预装常用仪表盘
+  dashboards:
+    default:
+      k8s-cluster:
+        gnetId: 315
+        revision: 3
+        datasource: Prometheus
+      k8s-pods:
+        gnetId: 6417
+        datasource: Prometheus
+      node-exporter:
+        gnetId: 1860
+        datasource: Prometheus
+
+# node-exporter（主机指标）

…… 改动较长，只显示前一部分