ArgoWorkflow教程(七)---高效的步骤间文件共享策略

意琦行收录于 Devops

2024-06-03 00:00:00 约 1600 字预计阅读 4 分钟 - 次阅读

之前我们分析了使用 artifact 实现步骤间文件共享，今天分享一下如何使用 PVC 实现更加高效的步骤间文件共享。

1. 概述

之前在 artifact 篇我们演示了如何使用 artifact 实现步骤间文件传递，今天介绍一种更为简单的文件传递方式：PVC 共享。

artifact 毕竟是借助 S3 实现中转，效率上肯定是低于直接共享 PVC 的，而且 artifact 一般用于结果输出，将最终结果保存到 S3，而不是单纯的用来共享文件。

2. 使用 artifact 共享文件

之前已经分享过了怎么通过 artifact 在不同步骤之间传递文件，这里在回顾一下。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifact-passing-
spec:
  entrypoint: artifact-example
  templates:
  - name: artifact-example
    steps:
    - - name: generate-artifact
        template: whalesay
    - - name: consume-artifact
        template: print-message
        arguments:
          artifacts:
          # bind message to the hello-art artifact
          # generated by the generate-artifact step
          - name: message
            from: "{{steps.generate-artifact.outputs.artifacts.hello-art}}"

  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [sh, -c]
      args: ["cowsay hello world | tee /tmp/hello_world.txt"]
    outputs:
      artifacts:
      # generate hello-art artifact from /tmp/hello_world.txt
      # artifacts can be directories as well as files
      - name: hello-art
        path: /tmp/hello_world.txt

  - name: print-message
    inputs:
      artifacts:
      # unpack the message input artifact
      # and put it at /tmp/message
      - name: message
        path: /tmp/message
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["cat /tmp/message"]

可以看到，artifact 方式共享文件步骤间传递参数是比较类似。

导出 artifact

1
2
3
4
5
6
outputs:
  artifacts:
  # generate hello-art artifact from /tmp/hello_world.txt
  # artifacts can be directories as well as files
  - name: hello-art
    path: /tmp/hello_world.txt

后续步骤引用导出的 artifact

1
2
3
4
5
6
arguments:
  artifacts:
  # bind message to the hello-art artifact
  # generated by the generate-artifact step
  - name: message
    from: "{{steps.generate-artifact.outputs.artifacts.hello-art}}"

以及步骤中怎么将 artifact 引入，比如下面 demo 则是将 artifact 做为 /tmp/message 挂载到 Pod 中。

1
2
3
4
5
6
inputs:
  artifacts:
  # unpack the message input artifact
  # and put it at /tmp/message
  - name: message
    path: /tmp/message

3. 使用 PVC 高效共享文件

顾名思义，就是不同步骤都挂载同一个 PVC，这样自然就实现了文件共享。

ArgoWorkflow 中的每一步都会单独启动一个 Pod 来运行

也有两种方式：

1）动态创建 PVC：Workflow 运行时创建 PVC，运行结束后删除 PVC
2）使用已有 PVC：不会创建也不会删除

动态创建 PVC

完整 Demo 如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: volumes-pvc-
spec:
  entrypoint: volumes-pvc-example
  volumeClaimTemplates:                 # define volume, same syntax as k8s Pod spec
  - metadata:
      name: workdir                     # name of volume claim
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi                  # Gi => 1024 * 1024 * 1024

  templates:
  - name: volumes-pvc-example
    steps:
    - - name: generate
        template: whalesay
    - - name: print
        template: print-message

  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [sh, -c]
      args: ["echo generating message in volume; cowsay hello world | tee /mnt/vol/hello_world.txt"]
      # Mount workdir volume at /mnt/vol before invoking docker/whalesay
      volumeMounts:                     # same syntax as k8s Pod spec
      - name: workdir
        mountPath: /mnt/vol

  - name: print-message
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo getting message from volume; find /mnt/vol; cat /mnt/vol/hello_world.txt"]
      # Mount workdir volume at /mnt/vol before invoking docker/whalesay
      volumeMounts:                     # same syntax as k8s Pod spec
      - name: workdir
        mountPath: /mnt/vol

首先定义了一个 PVC 模版，Workflow 运行时就会使用该模版创建一个 PVC

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
spec:
  entrypoint: volumes-pvc-example
  volumeClaimTemplates:                 # define volume, same syntax as k8s Pod spec
  - metadata:
      name: workdir                     # name of volume claim
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi                  # Gi => 1024 * 1024 * 1024

然后其他步骤需要将该 PVC 挂载到对应目录

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [sh, -c]
      args: ["echo generating message in volume; cowsay hello world | tee /mnt/vol/hello_world.txt"]
      # Mount workdir volume at /mnt/vol before invoking docker/whalesay
      volumeMounts:                     # same syntax as k8s Pod spec
      - name: workdir
        mountPath: /mnt/vol

  - name: print-message
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo getting message from volume; find /mnt/vol; cat /mnt/vol/hello_world.txt"]
      # Mount workdir volume at /mnt/vol before invoking docker/whalesay
      volumeMounts:                     # same syntax as k8s Pod spec
      - name: workdir
        mountPath: /mnt/vol

这样就实现了文件共享，非常简单。

等 Workflow 运行结束后，Argo 会自动将创建出的 PVC 删除。

使用已有 PVC

在某些情况下，我们可以希望访问已经存在的卷，而不是动态创建/销毁卷。

完整 Demo 如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
# Define Kubernetes PVC
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: my-existing-volume
spec:
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 1Gi

---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: volumes-existing-
spec:
  entrypoint: volumes-existing-example
  volumes:
  # Pass my-existing-volume as an argument to the volumes-existing-example template
  # Same syntax as k8s Pod spec
  - name: workdir
    persistentVolumeClaim:
      claimName: my-existing-volume

  templates:
  - name: volumes-existing-example
    steps:
    - - name: generate
        template: whalesay
    - - name: print
        template: print-message

  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [sh, -c]
      args: ["echo generating message in volume; cowsay hello world | tee /mnt/vol/hello_world.txt"]
      volumeMounts:
      - name: workdir
        mountPath: /mnt/vol

  - name: print-message
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo getting message from volume; find /mnt/vol; cat /mnt/vol/hello_world.txt"]
      volumeMounts:
      - name: workdir
        mountPath: /mnt/vol

首先就是手动创建一个 PVC

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# Define Kubernetes PVC
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: my-existing-volume
spec:
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 1Gi

然后在 Workflow 中定义要使用这个 PVC

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: volumes-existing-
spec:
  entrypoint: volumes-existing-example
  volumes:
  # Pass my-existing-volume as an argument to the volumes-existing-example template
  # Same syntax as k8s Pod spec
  - name: workdir
    persistentVolumeClaim:
      claimName: my-existing-volume

可以看做是使用 persistentVolumeClaim 来替换了之前的 volumeClaimTemplates

然后就是步骤将这个 PVC 挂载到对应目录

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [sh, -c]
      args: ["echo generating message in volume; cowsay hello world | tee /mnt/vol/hello_world.txt"]
      volumeMounts:
      - name: workdir
        mountPath: /mnt/vol

  - name: print-message
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo getting message from volume; find /mnt/vol; cat /mnt/vol/hello_world.txt"]
      volumeMounts:
      - name: workdir
        mountPath: /mnt/vol

这一步和使用动态创建 PVC 时没有任何变化。

4. 小结

本文主要分析了 Argo 中的 Workflow 中怎么使用 PVC 共享文件。

1）定义 PVC 模版或者指定使用已有的 PVC
2）步骤中将 PVC 挂载到对应目录使用