近期在做CD调研，因为我们CI是使用Gitlab Runner做的，所以今天看下使用Gitlab Runner 做 CD

环境信息

  • Gitlab 社区版 11.2.5
  • Gitlab-runner v11.3.1
    • 安装于Kubernetes集群中, 安装过程见链接
  • Kubernetes v.1.20.4 阿里云ACK托管版

思路

逻辑其实很简单，基于原有CI流程(.gitlab-ci.yml)增加一步stages即可。

  • stages： kubectl set image

实现过程中有以下几点需要考虑

  1. Gitlab 需要 集成 Kubernetes
    • Gitlab本身可以直接添加外部Kubernetes集群，但需要为每个项目都配置比较麻烦，放弃。
    • 考虑使用公网APIserver入口 使用Kubeconf从Gitlab Runner直接调用
      • ACK APIserver暴露公网入口过程见文档
      • 通过configmap方式挂载到 CD流程启动的Runner Pod中
  2. CD 流程需要手动触发
    • 并不是每个版本都需要发布到生产环境
    • 使用when: manual

Runner配置

为kubeconf创建configmap

参考上述文档生成公网kubeconf配置文件 并 配置到namespace中

我的runner部署在kubernetes中，每个stages都会生成一个新的容器执行操作，所以需要配置configmap挂载

如果runner是直接虚拟机部署的，可以放到系统某个目录下直接指定文件调用即可

内容大致为

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: gitlab
  labels:
    app: kubeconf-k8s
  name: kubeconf-k8s
data:
  config: |
    apiVersion: v1
    clusters:
    - cluster:
        server: https://******
    具体内容每个集群不一致，按自己的kubeconf来
    多个集群可以选用不同的name，挂载到pod的不同目录即可

应用到Namespace

kubectl -n gitlab apply -f kubeconf.yaml

修改Runner Pod 模板

修改runner 应用 yaml 添加配置到config.toml

kubectl -n gitlab edit deployments.apps runner

在command部分增加以下3行 echo

        lifecycle:
          postStart:
            exec:
              command:
              - /bin/bash
              - -c
              - sleep 10;
              #### 原有内容不要删除，仅增加以下行 ####
                echo '    [[runners.kubernetes.volumes.config_map]]' >> /etc/gitlab-runner/conf
ig.toml;
                echo '      name = "kubeconf-serverk8s"' >> /etc/gitlab-runner/config.toml;
                echo '      mount_path = "/k8s/kubeconf-serverk8s"' >> /etc/gitlab-runner/confi
g.toml;

最终效果如下

[root@prod-public-runner-k8s-node01 ~]# kubectl -n gitlab exec -it runner-5dc5856fbd-dzbcb  cat /etc/gitlab-runner/config.toml

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "runner-5dc5856fbd-dzbcb"
  request_concurrency = 4
  url = "http://git.******.com"
  token = "8826046ccadc5bfeaa7feba68100ea"
  executor = "kubernetes"
  cache_dir = "/tmp/gitlab/cache"
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.kubernetes]
    host = "https://172.17.254.185:6443"
    cert_file = "/etc/gitlab-runner/certs/runner.pem"
    key_file = "/etc/gitlab-runner/certs/runner-key.pem"
    ca_file = "/etc/gitlab-runner/certs/ca.pem"
    bearer_token_overwrite_allowed = false
    image = ""
    namespace = "gitlab"
    namespace_overwrite_allowed = ""
    privileged = true
    pull_policy = "if-not-present"
    service_account = "executor"
    service_account_overwrite_allowed = ""
    pod_annotations_overwrite_allowed = ""
    [runners.kubernetes.volumes]
    [[runners.kubernetes.volumes.pvc]]
      name = "gitlab-cache"
      mount_path = "/tmp/gitlab/cache"
    [[runners.kubernetes.volumes.host_path]]
      name = "docker-sock"
      mount_path = "/var/run/docker.sock"
      host_path = "/var/run/docker.sock"
    [[runners.kubernetes.volumes.pvc]]
      name = "maven-cache"
      mount_path = "/root/.m2"
      host_path = "/tmp/maven/cache"
    [[runners.kubernetes.volumes.pvc]]
      name = "npm-cache"
      mount_path = "/npm/node_modules"
      host_path = "/tmp/npm/cache"
    [[runners.kubernetes.volumes.config_map]]
      name = "kubeconf-k8s"
      mount_path = "/k8s/kubeconf-serverk8s"

Runner部分修改完毕，应用后会重新注册一个Runner到Gitlab

CICD流程stages创建Pod时将会挂载以上配置；

.gitlab-ci.yml CD 流程配置

生成kubectl镜像

stages执行kubelet时需要使用image，这里直接build一个

  • 先从Github下载自己需要的client版本，我使用的v1.20.4的kubernetes-client-linux-amd64.tar.gz

  • 解压获得kubectl

  • 编写Dockerfile 并 build push到自己的仓库

    FROM centos:8
    ADD kubectl /bin/

增加stages

在原有.gitlabci.yml基础上增加一个stages

deploy:
    image: harbor.***.net/public/kubelet:v1.20 #指定刚build的镜像地址
    stage: deploy
    variables:
      GIT_STRATEGY: none
    script:
      - echo ${DOCKERE_IMAGE}
      - kubectl --kubeconfig /k8s/kubeconf-serverk8s/config -n my-app set image deploy cicd-test cicd-test=${DOCKERE_IMAGE}
    when: manual
    only:
      - /^v[0-9]+(\.[0-9]+)+$/

完整配置如下，仅测试deploy stages，codebuild 未写

variables:
  DDING_ROBOT_TOKEN: ************
  GIT_STRATEGY: fetch
  GIT_DEPTH: 10
  CI_DEBUG_TRACE: "false"
  GIT_SUBMODULE_STRATEGY: recursive
  GIT_REPO: ${CI_PROJECT_NAME}
  GIT_BRANCH: ${CI_COMMIT_REF_NAME}
  GIT_GROUP: "${CI_PROJECT_NAMESPACE}"
  AUTHOR_NAME: ${GITLAB_USER_LOGIN}
  AUTHOR_EMAIL: ${GITLAB_USER_EMAIL}
  AUTHOR_USERNAME: ${GITLAB_USER_NAME}
  DOCKER_REPO: "harbor.***.net"
  HELM_REPO: "https://harbor.***.net/chartrepo/"

cache:
  untracked: true
  key: $GIT_REPO-$GIT_BRANCH-$CI_COMMIT_SHA
  paths:
    - conf

stages:
    - git version
    - go build
    - docker build
    - helm push
    - notice
    - deploy

before_script:
    - export GIT_VERSION="`cat git_version`"
    - echo "${GIT_GROUP}"|egrep "cloud|ai" >/dev/null && export GIT_GROUP="ai"
    - export BUILD_VERSION="$GIT_BRANCH.$GIT_VERSION-`echo ${CI_COMMIT_SHA}| cut -c1-8`"
    - export DOCKERE_IMAGE="${DOCKER_REPO}/${GIT_GROUP}/${GIT_REPO}:${BUILD_VERSION}"
    - export SERVICE_PORT=`cat Dockerfile |grep EXPOSE|awk '{print $NF}'`
    - export REQUEST_CPU=`cat Dockerfile |grep REQUEST|awk -F'[ |,]' '{print $4}'`
    - export REQUEST_MEM=`cat Dockerfile |grep REQUEST|awk -F'[ |,]' '{print $5}'`
    - export LIMIT_CPU=`cat Dockerfile |grep LIMIT|awk -F'[ |,]' '{print $4}'`
    - export LIMIT_MEM=`cat Dockerfile |grep LIMIT|awk -F'[ |,]' '{print $5}'`

git_version:
    image: registry.cn-beijing.aliyuncs.com/***/centos-git:latest
    stage: git version
    script:
    - git rev-list HEAD |wc -l > ./git_version
    only:
      - /^v[0-9]+(\.[0-9]+)+$/

code build:
    image: registry.cn-beijing.aliyuncs.com/***/golang:1.9.2
    stage: go build
    script:
      - echo $GIT_VERSION > ./GIT_VERSION
    only:
      - /^v[0-9]+(\.[0-9]+)+$/

docker build:
    image: docker:18.06.1-ce
    stage: docker build
    script:
      - docker login -u ${DOCKER_REPO_USERNAME} -p ${DOCKER_REPO_PASSWORD} ${DOCKER_REPO}
      - docker build -t ${DOCKERE_IMAGE} -f Dockerfile .
      - docker push ${DOCKERE_IMAGE}
    only:
      - /^v[0-9]+(\.[0-9]+)+$/

helm push:
    image: registry.cn-beijing.aliyuncs.com/***/centos-helm:2.12.0
    stage: helm push
    script:
    - sed -i "s/Docker_Image/${DOCKER_REPO}\/${GIT_GROUP}\/${GIT_REPO}/g" gitlab-ci/helm-templates/values.yaml
    - sed -i "s/BUILD_VERSION/${BUILD_VERSION}/g" gitlab-ci/helm-templates/values.yaml
    - sed -i "s/GIT_REPO/${GIT_REPO}/g" gitlab-ci/helm-templates/values.yaml
    - sed -i "s/Service_External_Port/${SERVICE_PORT}/g" gitlab-ci/helm-templates/values.yaml
    - sed -i "s/Service_Internal_Port/${SERVICE_PORT}/g" gitlab-ci/helm-templates/values.yaml
    - sed -i "s/helm-templates/${GIT_REPO}/g" gitlab-ci/helm-templates/Chart.yaml
    - sed -i "s/REQUEST_CPU/${REQUEST_CPU}/g" gitlab-ci/helm-templates/values.yaml
    - sed -i "s/REQUEST_MEM/${REQUEST_MEM}/g" gitlab-ci/helm-templates/values.yaml
    - sed -i "s/LIMIT_CPU/${LIMIT_CPU}/g" gitlab-ci/helm-templates/values.yaml
    - sed -i "s/LIMIT_MEM/${LIMIT_MEM}/g" gitlab-ci/helm-templates/values.yaml
    - helm push gitlab-ci/helm-templates ${HELM_REPO}/${GIT_GROUP} --username=${HELM_REPO_USERNAME} --password=${HELM_REPO_PASSWORD} -v="${BUILD_VERSION}"
    only:
      - /^v[0-9]+(\.[0-9]+)+$/

build success:
    image: registry.cn-beijing.aliyuncs.com/***/ci_notice
    stage: notice
    variables:
      GIT_STRATEGY: none
    script:
      - notice succeed "${DOCKERE_IMAGE}"
    when: on_success
    only:
      - /^v[0-9]+(\.[0-9]+)+$/

build failed:
    image: registry.cn-beijing.aliyuncs.com/***/ci_notice
    stage: notice
    variables:
      GIT_STRATEGY: none
    script:
      - notice fail "${DOCKERE_IMAGE}"
    when: on_failure
    only:
      - /^v[0-9]+(\.[0-9]+)+$/
deploy:
    image: harbor.***.net/public/kubelet:v1.20
    stage: deploy
    variables:
      GIT_STRATEGY: none
    script:
      - echo ${DOCKERE_IMAGE}
      - kubectl --kubeconfig /k8s/kubeconf-serverk8s/config -n my-app set image deploy cicd-test cicd-test=${DOCKERE_IMAGE}
    when: manual
    only:
      - /^v[0-9]+(\.[0-9]+)+$/

流程测试

Git 提交代码到指定分支，或在流水线界面直接运行流水线

CI流水线完成后将会暂停到CD Deploy stages

此时需要手动点击Play按钮，或者点击进入作业后点击触发此手动操作

cd-pipeline

cd-pipeline-info

触发后的完整过程如下

deploy-log

此时应用的版本已经被更新；