摘要

本文讲解Kubernetes中，Pod的升级和回滚实践。

实践内容

当集群中的某个服务需要升级时，我们需要停止目前与该服务相关的所有Pod，然后下载新版本镜像并创建新的Pod。如果集群规模比较大，则这个工作变成了一个挑战，而且先全部停止然后逐步升级的方式会导致较长时间的服务不可用。Kubernetes提供了滚动升级功能来解决上述问题。

如果Pod是通过Deployment创建的，则用户可以在运行时修改Deployment的Pod定义（spec.template）或镜像名称，并应用到Deployment对象上，系统即可完成Deployment的自动更新操作。如果在更新过程中发生了错误，则还可以通过回滚操作恢复Pod的版本。

（1） Deployment的升级

以Deployment nginx为例（nginx-deployment.yaml）：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
        tier: nginx-label 
    matchExpressions:
      - {key: tier, operator: In, values: [nginx-label]}      
  template:
    metadata:
      labels:
        app: nginx
        tier: nginx-label 
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        imagePullPolicy: IfNotPresent        
        ports:
        - containerPort: 80

报错

$ kubectl create -f nginx-deployment.yaml 
error: unable to recognize "nginx-deployment.yaml": no matches for kind "Deployment" in version "apps/v1beta1"

原因：

这个主要是由于版本升级的原因.由
apiVersion: apps/v1beta1
kind: Deployment
改为

apiVersion: apps/v1
kind: Deployment

已运行的Pod副本数量有3个：

$ kubectl get pods
NAME                                READY   STATUS      RESTARTS   AGE
cm-test-pod                         0/1     Completed   0          2d1h
frontend-7d7c57fc94-hldfv           1/1     Running     0          5m3s
mysql-f9mzh                         1/1     Running     0          3d
myweb-gcfh5                         1/1     Running     0          3d2h
nginx-deployment-6f68987bf5-2fftn   1/1     Running     0          29s
nginx-deployment-6f68987bf5-hrxx6   1/1     Running     0          29s
nginx-deployment-6f68987bf5-wt427   1/1     Running     0          29s

现在Pod镜像需要被更新为Nginx:1.9.1，我们可以通过kubectl set image命令为Deployment设置新的镜像名称：

$ kubectl set image deployment/nginx-deployment nginx=nginx:1.9.1
deployment.apps/nginx-deployment image updated

另一种更新的方法是使用kubectl edit命令修改Deployment的配置，将spec.template.spec.containers[0].image从Nginx:1.9.1更改为Nginx:1.7.9：

$ kubectl edit deployment/nginx-deployment

...
 containers:
      - image: nginx:1.7.9
        imagePullPolicy: IfNotPresent
...

deployment.apps/nginx-deployment edited

以下描述由nginx:1.7.9到Nginx:1.9.1的过程。一旦镜像名（或Pod定义）发生了修改，则将触发系统完成Deployment所有运行Pod的滚动升级操作。可以使用kubectl rollout status命令查看Deployment的更新过程：

$ kubectl rollout status deployment/nginx-deployment
deployment "nginx-deployment" successfully rolled out
root@k8s-master:/data/k8s# kubectl set image deployment/nginx-deployment nginx=nginx:1.9.1
deployment.apps/nginx-deployment image updated
root@k8s-master:/data/k8s# kubectl rollout status deployment/nginx-deployment
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "nginx-deployment" rollout to finish: 1 old replicas are pending termination...
deployment "nginx-deployment" successfully rolled out

查看Pod使用的镜像，已经更新为Nginx:1.9.1了：

$ kubectl get pods
NAME                                READY   STATUS      RESTARTS   AGE
cm-test-pod                         0/1     Completed   0          2d1h
frontend-7d7c57fc94-hldfv           1/1     Running     0          18m
mysql-f9mzh                         1/1     Running     0          3d
myweb-gcfh5                         1/1     Running     0          3d2h
nginx-deployment-77986bc785-7c7lp   1/1     Running     0          76s
nginx-deployment-77986bc785-9w849   1/1     Running     0          80s
nginx-deployment-77986bc785-sgfnt   1/1     Running     0          78s

$ kubectl describe pod nginx-deployment-77986bc785-7c7lp
Name:         nginx-deployment-77986bc785-7c7lp
...
Image:          nginx:1.9.1
...

那么，Deployment是如何完成Pod更新的呢？我们可以使用kubectl describe deployments/nginx-deployment命令仔细观察Deployment的更新过程。初始创建Deployment时，系统创建了一个ReplicaSet（nginx-deployment-4087004473），并按用户的需求创建了3个Pod副本。当更新Deployment时，系统创建了一个新的ReplicaSet（nginx-deployment-3599678771），并将其副本数量扩展到1，然后将旧的ReplicaSet缩减为2。之后，系统继续按照相同的更新策略对新旧两个ReplicaSet进行逐个调整。最后，新的ReplicaSet运行了3个新版本Pod副本，旧的ReplicaSet副本数量则缩减为0。如图3.8所示。

运行kubectl get rs命令，查看两个ReplicaSet的最终状态：

$ kubectl get rs
NAME                          DESIRED   CURRENT   READY   AGE
frontend-7d7c57fc94           1         1         1       28m
nginx-deployment-6f68987bf5   0         0         0       24m
nginx-deployment-77986bc785   3         3         3       19m

在整个升级的过程中，系统会保证至少有两个Pod可用，并且最多同时运行4个Pod，这是Deployment通过复杂的算法完成的。Deployment需要确保在整个更新过程中只有一定数量的Pod可能处于不可用状态。在默认情况下，Deployment确保可用的Pod总数至少为所需的副本数量（DESIRED）减1，也就是最多1个不可用（maxUnavailable=1）。Deployment还需要确保在整个更新过程中Pod的总数量不会超过所需的副本数量太多。

（2） Deployment的回滚

有时（例如新的Deployment不稳定时）我们可能需要将Deployment回滚到旧版本。在默认情况下，所有Deployment的发布历史记录都被保留在系统中，以便于我们随时进行回滚（可以配置历史记录数量）。假设在更新Deployment镜像时，将容器镜像名误设置成Nginx:1.91（一个不存在的镜像）：

假设在更新Deployment镜像时，将容器镜像名误设置成Nginx:1.91（一个不存在的镜像）：

$ kubectl set image deployment/nginx-deployment nginx=nginx:1.91
deployment.apps/nginx-deployment image updated

则这时Deployment的部署过程会卡住：

$ kubectl rollout status deployment/nginx-deployment
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...

由于执行过程卡住，所以需要执行Ctrl-C命令来终止这个查看命令。查看ReplicaSet，可以看到新建的ReplicaSet（nginx-deployment-67886c9675）：

# kubectl get rs
NAME                          DESIRED   CURRENT   READY   AGE
frontend-7d7c57fc94           1         1         1       34m
nginx-deployment-67886c9675   1         1         0       101s
nginx-deployment-6f68987bf5   0         0         0       29m
nginx-deployment-77986bc785   3         3         3       25m

再查看创建的Pod，会发现新的ReplicaSet创建的1个Pod被卡在镜像拉取过程中,ImagePullBackOff。

$ kubectl get pods
NAME                                READY   STATUS             RESTARTS   AGE
cm-test-pod                         0/1     Completed          0          2d1h
frontend-7d7c57fc94-hldfv           1/1     Running            0          36m
mysql-f9mzh                         1/1     Running            0          3d1h
myweb-gcfh5                         1/1     Running            0          3d2h
nginx-deployment-67886c9675-zkhn5   0/1     ImagePullBackOff   0          3m35s
nginx-deployment-77986bc785-7c7lp   1/1     Running            0          19m
nginx-deployment-77986bc785-9w849   1/1     Running            0          19m
nginx-deployment-77986bc785-sgfnt   1/1     Running            0          19m

为了解决上面这个问题，我们需要回滚到之前稳定版本的Deployment。首先，用kubectl rollout history命令检查这个Deployment部署的历史记录：

$ kubectl rollout history deployment/nginx-deployment
deployment.apps/nginx-deployment 
REVISION  CHANGE-CAUSE
3         <none>
4         <none>
5         <none>

注意，在创建Deployment时使用--record参数，就可以在CHANGE-CAUSE列看到每个版本使用的命令了。

另外，Deployment的更新操作是在Deployment进行部署（Rollout）时被触发的，这意味着当且仅当Deployment的Pod模板（即spec.template）被更改时才会创建新的修订版本，例如更新模板标签或容器镜像。其他更新操作（如扩展副本数）将不会触发Deployment的更新操作，这也意味着我们将Deployment回滚到之前的版本时，只有Deployment的Pod模板部分会被修改。

如果需要查看特定版本的详细信息，则可以加上--revision=<N>参数：

$ kubectl rollout history deployment/nginx-deployment --revision=4
deployment.apps/nginx-deployment with revision #4
Pod Template:
  Labels:   app=nginx
    pod-template-hash=77986bc785
    tier=nginx-label
  Containers:
   nginx:
    Image:  nginx:1.9.1
    Port:   80/TCP
    Host Port:  0/TCP
    Environment:    <none>
    Mounts: <none>
  Volumes:  <none>

现在我们决定撤销本次发布并回滚到上一个部署版本：

$ kubectl rollout undo deployment/nginx-deployment
deployment.apps/nginx-deployment rolled back

这样，该Deployment就回滚到之前的稳定版本了，可以从Deployment的事件信息中查看到回滚到版本2的操作过程：

$ kubectl describe deployment/nginx-deployment

Name:                   nginx-deployment
Namespace:              default
...
NewReplicaSet:   nginx-deployment-77986bc785 (3/3 replicas created)
Events:
  Type    Reason             Age                From                   Message
  ----    ------             ----               ----                   -------
  Normal  ScalingReplicaSet  37m                deployment-controller  Scaled down replica set nginx-deployment-6f68987bf5 to 2
  Normal  ScalingReplicaSet  37m                deployment-controller  Scaled up replica set nginx-deployment-77986bc785 to 2
  Normal  ScalingReplicaSet  37m                deployment-controller  Scaled up replica set nginx-deployment-77986bc785 to 3
  Normal  ScalingReplicaSet  33m                deployment-controller  Scaled up replica set nginx-deployment-6f68987bf5 to 1
  Normal  ScalingReplicaSet  33m                deployment-controller  Scaled down replica set nginx-deployment-77986bc785 to 2
  Normal  ScalingReplicaSet  33m (x2 over 42m)  deployment-controller  Scaled up replica set nginx-deployment-6f68987bf5 to 3
  Normal  ScalingReplicaSet  30m (x2 over 37m)  deployment-controller  Scaled up replica set nginx-deployment-77986bc785 to 1
  Normal  ScalingReplicaSet  30m (x2 over 37m)  deployment-controller  Scaled down replica set nginx-deployment-6f68987bf5 to 1
  Normal  ScalingReplicaSet  30m (x6 over 33m)  deployment-controller  (combined from similar events): Scaled up replica set nginx-deployment-77986bc785 to 3
  Normal  ScalingReplicaSet  30m (x2 over 37m)  deployment-controller  Scaled down replica set nginx-deployment-6f68987bf5 to 0
  Normal  ScalingReplicaSet  14m                deployment-controller  Scaled up replica set nginx-deployment-67886c9675 to 1
  Normal  ScalingReplicaSet  3m21s              deployment-controller  Scaled down replica set nginx-deployment-67886c9675 to 0

（3）kubectl rollout命令列表

kubectl rollout
Manage the rollout of a resource.
  
 Valid resource types include:

  *  deployments
  *  daemonsets
  *  statefulsets

Examples:
  # Rollback to the previous deployment
  kubectl rollout undo deployment/abc
  
  # Check the rollout status of a daemonset
  kubectl rollout status daemonset/foo

Available Commands:
  history     View rollout history
  pause       Mark the provided resource as paused
  restart     Restart a resource
  resume      Resume a paused resource
  status      Show the status of the rollout
  undo        Undo a previous rollout

Usage:
  kubectl rollout SUBCOMMAND [options]

Use "kubectl <command> --help" for more information about a given command.
Use "kubectl options" for a list of global command-line options (applies to all commands).

运行kubectl get rs命令，查看两个ReplicaSet的最终状态：

$ kubectl get rs
NAME                          DESIRED   CURRENT   READY   AGE
frontend-7d7c57fc94           1         1         1       28m
nginx-deployment-6f68987bf5   0         0         0       24m
nginx-deployment-77986bc785   3         3         3       19m

参考

《Kubernetes权威指南：从Docker到Kubernetes实践全接触（第4版）》

第7课 Kubernetes之Pod的升级和回滚实践

摘要

实践内容

（1） Deployment的升级

（2） Deployment的回滚

（3）kubectl rollout命令列表

参考

相关推荐

在.net core中使用nginx做负载均衡

React 18 超全升级指南

LTUI v1.7 发布，一个基于 Lua 的跨平台字符终端 UI 界面库

生成对抗网络(GAN)的半监督学习

《若依ruoyi》第二十五章:Spring boot 上传下载封装详解二

QT进阶之路 : 布局详解

如何在 Element UI 中使用栅格布局实现响应式设计?

IDEA 中 Jetty 的配置操作手册 idea jfinal

如何在Dify平台上创建智能Agent:一步步教你实现超级智能体搭建

k8s自动化运维三