236｜Kubernetes 调度与资源管理：调度器原理、亲和性、污点容忍、QoS 与自动伸缩

用户/Controller 创建 Pod → API Server 写入 etcd
    → Scheduler Watch 到未调度的 Pod（nodeName 为空）
        → 执行调度算法，选出最优节点
            → 向 API Server 写入 Binding（设置 nodeName）
                → 目标节点 kubelet 感知到 Binding，拉取镜像启动容器

Sort → PreFilter → Filter → PostFilter → PreScore → Score → NormalizeScore → Reserve → Permit → PreBind → Bind → PostBind

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  schedulerName: my-custom-scheduler  # 默认是 default-scheduler
  containers:
  - name: app
    image: nginx

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  affinity:
    nodeAffinity:
      # 硬约束：节点必须有 gpu-type=nvidia-a100 标签
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: gpu-type
            operator: In
            values:
            - nvidia-a100
            - nvidia-h100
      # 软偏好：优先选择 ssd=true 的节点
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 80
        preference:
          matchExpressions:
          - key: disk-type
            operator: In
            values:
            - ssd
      - weight: 20
        preference:
          matchExpressions:
          - key: zone
            operator: In
            values:
            - zone-a
  containers:
  - name: training
    image: pytorch/pytorch:latest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  labels:
    app: redis
spec:
  replicas: 3
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      affinity:
        # 反亲和性：不要把同一个 Redis 的多个副本调度到同一节点
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - redis
            topologyKey: kubernetes.io/hostname
      containers:
      - name: redis
        image: redis:7

# 将 Web 应用调度到与 Redis 同一节点，减少网络延迟
affinity:
  podAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app
          operator: In
          values:
          - redis
      topologyKey: kubernetes.io/hostname

kubectl taint nodes node1 key1=value1:NoSchedule
#                    │     │      │      └── Effect（效果）
#                    │     │      └── Value（值）
#                    │     └── Key（键）
#                    └── 节点名

tolerations:
- key: node.kubernetes.io/not-ready
  operator: Exists
  effect: NoExecute
  tolerationSeconds: 300   # 节点 NotReady 后 300 秒才被驱逐

apiVersion: v1
kind: Pod
metadata:
  name: special-pod
spec:
  tolerations:
  # 容忍所有带有 key=special 且 effect=NoSchedule 的污点
  - key: special
    operator: Equal
    value: "true"
    effect: NoSchedule
  # 容忍所有带有 key=gpu 的污点，不管 value 和 effect
  - key: gpu
    operator: Exists
  containers:
  - name: app
    image: nginx

# K8s 自动为 Pod 添加的默认容忍
tolerations:
- key: node.kubernetes.io/not-ready
  operator: Exists
  effect: NoExecute
  tolerationSeconds: 300
- key: node.kubernetes.io/unreachable
  operator: Exists
  effect: NoExecute
  tolerationSeconds: 300

# 给 GPU 节点打上标签和污点
kubectl label node gpu-node-1 node-role=gpu
kubectl label node gpu-node-2 node-role=gpu
kubectl taint node gpu-node-1 gpu=true:NoSchedule
kubectl taint node gpu-node-2 gpu=true:NoSchedule

# 给高内存节点打标签
kubectl label node mem-node-1 node-role=memory-intensive
kubectl label node mem-node-2 node-role=memory-intensive

# GPU 任务的 Pod 配置
spec:
  tolerations:
  - key: gpu
    operator: Equal
    value: "true"
    effect: NoSchedule
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node-role
            operator: In
            values:
            - gpu

# Guaranteed：requests == limits
spec:
  containers:
  - name: app
    image: myapp:latest
    resources:
      requests:
        cpu: "500m"
        memory: "512Mi"
      limits:
        cpu: "500m"
        memory: "512Mi"

# Burstable：requests < limits
spec:
  containers:
  - name: app
    image: myapp:latest
    resources:
      requests:
        cpu: "250m"
        memory: "256Mi"
      limits:
        cpu: "1000m"
        memory: "1Gi"

# BestEffort：没有 resources 配置
spec:
  containers:
  - name: app
    image: myapp:latest

BestEffort → Burstable（使用量超过 requests 的） → Burstable（使用量不超过 requests 的） → Guaranteed

# 进入容器查看 OOM Score
cat /proc/1/oom_score
cat /proc/1/oom_score_adj

# kubelet 配置
evictionHard:
  memory.available: "200Mi"       # 可用内存 < 200Mi 时驱逐
  nodefs.available: "10%"         # 磁盘可用 < 10% 时驱逐
  nodefs.inodesFree: "5%"         # inode 可用 < 5% 时驱逐
  imagefs.available: "15%"        # 镜像文件系统可用 < 15% 时驱逐

evictionSoft:
  memory.available: "500Mi"
evictionSoftGracePeriod:
  memory.available: "1m30s"       # 超过阈值 90 秒后才驱逐

HPA 控制器每隔 15 秒（可通过 --horizontal-pod-autoscaler-sync-period 调整）
    → 从 Metrics API 获取当前指标值
        → 计算 desiredReplicas = ceil(currentReplicas × (currentMetric / targetMetric))
            → 调整 Deployment 的 replicas 字段

desiredReplicas = ceil(currentReplicas × (currentMetricValue / desiredMetricValue))

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 2
  maxReplicas: 10
  metrics:
  # CPU 指标
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  # 内存指标
  - type: Resource
    resource:
      name: memory
      target:
        type: AverageValue
        averageValue: 512Mi
  # 自定义指标（需要 Prometheus Adapter 等）
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "1000"
  behavior:
    # 扩容行为
    scaleUp:
      stabilizationWindowSeconds: 0    # 扩容不等待，立即响应
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15              # 每 15 秒最多翻倍
      - type: Pods
        value: 4
        periodSeconds: 15              # 每 15 秒最多增加 4 个 Pod
      selectPolicy: Max
    # 缩容行为
    scaleDown:
      stabilizationWindowSeconds: 300  # 缩容前等待 5 分钟，防止抖动
      policies:
      - type: Pods
        value: 1
        periodSeconds: 60              # 每分钟最多减少 1 个 Pod

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: myapp-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  updatePolicy:
    updateMode: Auto
  resourcePolicy:
    containerPolicies:
    - containerName: '*'
      minAllowed:
        cpu: "100m"
        memory: "128Mi"
      maxAllowed:
        cpu: "2"
        memory: "4Gi"
      controlledResources: ["cpu", "memory"]

CA 工作流程：
1. 每 10 秒检查是否有 Pending Pod
2. 如果有 Pending Pod 且现有节点无法满足：
   → 判断节点组（Node Group）是否在最大节点数限制内
   → 调用云厂商 API 增加节点
3. 每 10 秒检查是否有低利用率节点：
   → 节点上 Pod 的 requests 总和 < 节点容量的 50%（默认）
   → 节点上的 Pod 可以安全迁移到其他节点
   → 调用云厂商 API 删除节点

流量增加 → HPA 扩容 Pod → Pod Pending（资源不足） → CA 增加节点 → Pod 调度成功
流量减少 → HPA 缩容 Pod → 节点空闲 → CA 回收节点

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kafka-consumer-scaler
spec:
  scaleTargetRef:
    name: kafka-consumer
  minReplicaCount: 0    # 支持缩容到 0
  maxReplicaCount: 30
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: kafka:9092
      consumerGroup: my-group
      topic: orders
      lagThreshold: "100"    # 积压超过 100 条就扩容

┌─────────────────────────────────────────────────────────────────┐
│                     K8s 调度与资源管理全景                        │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌──────────┐    ┌──────────┐    ┌──────────┐                  │
│  │ 亲和性    │    │ 污点容忍  │    │ 拓扑分布  │                  │
│  │ Affinity │    │ Taint/   │    │ Topology │                  │
│  │ Pod 选择  │    │ Toleration│    │ Spread   │                  │
│  │ 节点      │    │ 节点拒绝  │    │ 跨域均衡  │                  │
│  └────┬─────┘    └────┬─────┘    └────┬─────┘                  │
│       │               │               │                         │
│       └───────────────┼───────────────┘                         │
│                       ▼                                         │
│              ┌────────────────┐                                 │
│              │   调度器        │                                 │
│              │  Filter→Score  │                                 │
│              │  →Bind         │                                 │
│              └───────┬────────┘                                 │
│                      │                                          │
│       ┌──────────────┼──────────────┐                           │
│       ▼              ▼              ▼                           │
│  ┌──────────┐ ┌──────────┐ ┌──────────────┐                    │
│  │  QoS     │ │ 资源配额  │ │ 自动伸缩      │                    │
│  │ G/B/BE   │ │ RQ/LQ    │ │ HPA/VPA/CA   │                    │
│  │ 驱逐保护  │ │ 隔离限制  │ │ 弹性伸缩      │                    │
│  └──────────┘ └──────────┘ └──────────────┘                    │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

utilization = (当前所有 Pod 的 CPU 使用量总和) / (sum(Pod[i].spec.containers[*].resources.requests.cpu)) × 100%

desiredReplicas = ceil(currentReplicas × currentUtilization / targetUtilization)

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
spec:
  serviceName: kafka
  replicas: 3
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: kafka
            topologyKey: topology.kubernetes.io/zone
      containers:
      - name: kafka
        image: confluentinc/cp-kafka:7.5.0

1. kubectl apply -f deployment.yaml
   → 客户端将 YAML 序列化为 JSON，发送 POST 请求到 API Server

2. API Server 接收请求
   → 认证（Authentication）：验证用户身份
   → 授权（Authorization）：RBAC 检查用户是否有 create deployments 权限
   → 准入控制（Admission Control）：例如 LimitRanger 设置默认 requests/limits，
     ResourceQuota 检查命名空间配额是否超限
   → 写入 etcd，返回 Deployment 对象

3. Deployment Controller（kube-controller-manager 内）
   → Watch 到新 Deployment → 创建 ReplicaSet → ReplicaSet Controller 创建 Pod
   → Pod 写入 etcd 时 nodeName 为空，status.phase = Pending

4. Scheduler 介入
   → Watch 到 nodeName 为空的 Pod，放入调度队列
   → Sort 阶段：按优先级排序（PriorityClass）
   → PreFilter：预处理 Pod 信息
   → Filter（预选）：
     · PodFitsResources：节点剩余 CPU/内存 >= Pod 的 requests
     · PodMatchNodeSelector：匹配 nodeSelector / Node Affinity
     · PodToleratesNodeTaints：匹配 Taint/Toleration
     · CheckNodeCondition：节点 Ready？
     · PodFitsHostPorts：HostPort 冲突？
     · 其他 Filter 插件...
   → PostFilter：如果 Filter 后无可用节点 → 抢占（Preemption）低优先级 Pod
   → Score（优选）：
     · NodeResourcesFit：资源均衡度打分
     · ImageLocality：镜像已有加分
     · InterPodAffinity：Pod 亲和性匹配加分
     · 其他 Score 插件...
   → Reserve：预留资源（乐观并发）
   → Permit：等待/批准
   → Bind：向 API Server 提交 Binding 对象，设置 nodeName

5. 目标节点 kubelet
   → Watch 到 nodeName 指向自己的 Pod
   → 再次执行 Admission（如 PodSecurityPolicy 检查）
   → 调用 CRI（Container Runtime Interface）拉取镜像、创建容器
   → 调用 CNI（Container Network Interface）配置网络
   → 调用 CSI（Container Storage Interface）挂载存储卷
   → 更新 Pod status 为 Running

6. 资源管理相关
   → kubelet 通过 Cgroup 设置 CPU/内存限制（limits）
   → kubelet 上报 Node Status（包含剩余资源）给 API Server
   → 如果发生 OOM：内核 OOM Killer 按 oom_score_adj 杀进程
   → kubelet 监控资源使用，触发驱逐（Eviction）

# GPU 节点：只接受 GPU 任务
kubectl taint node gpu-{1..8} workload-type=gpu:NoSchedule

# 在线服务节点：拒绝批处理任务
kubectl taint node service-{1..20} workload-type=online:NoSchedule

# 批处理节点：可以被抢占，接受在线服务溢出
kubectl taint node batch-{1..10} workload-type=batch:NoSchedule

# GPU 训练任务 Pod
spec:
  tolerations:
  - key: workload-type
    operator: Equal
    value: gpu
    effect: NoSchedule
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: workload-type
            operator: In
            values: [gpu]
    # 高优先级训练任务可以抢占批处理
  priorityClassName: high-priority

# 在线服务 Pod
spec:
  tolerations:
  - key: workload-type
    operator: Equal
    value: online
    effect: NoSchedule
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: workload-type
            operator: In
            values: [online]
    # 反亲和性：同一服务的 Pod 分散到不同节点
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app: my-service
          topologyKey: kubernetes.io/hostname

# 批处理任务 Pod
spec:
  tolerations:
  - key: workload-type
    operator: Equal
    value: batch
    effect: NoSchedule
  priorityClassName: low-priority  # 可被抢占