argo-dashboard

之前做过Gitlab的CI,但k8s上的CD一直没有做,所有上线均三方在线手工操作,目前计划增加CD功能

以下是整个Argo CD 的安装和测试记录

这里只验证了CD部分,CI由Gitlab实现后期会做结合

了解Argocd

  • Argo CD是一个基于Kubernetes的声明式的GitOps工具

  • 大致逻辑为 kubernetes的Deployment/SVC等yaml文件存放于Git仓库中,研发更新Git仓库中的Image版本即可完成发布(Argocd会监听对比Git和Kubernetes中的Yaml,不一致将会推送覆盖)

安装Argocd

环境

  • kubernetes v1.20.4 集群
  • argocd latest (v2.6.1)

安装

官方安装方法:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

查看被安装的资源

[root@prod-public-runner-k8s-node01 system]# kubectl -n argocd get all
NAME READY STATUS RESTARTS AGE
pod/argocd-application-controller-0 1/1 Running 0 41h
pod/argocd-applicationset-controller-7b5f77c495-sqqkm 1/1 Running 0 41h
pod/argocd-dex-server-58df896df-64c4d 1/1 Running 0 41h
pod/argocd-notifications-controller-5dcb877fd8-vnlqk 1/1 Running 0 42h
pod/argocd-redis-58654b9b4b-xkdj5 1/1 Running 0 41h
pod/argocd-repo-server-7459b57cd-xfbgx 1/1 Running 0 41h
pod/argocd-server-67545b758b-d8js7 1/1 Running 0 42h

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/argocd-applicationset-controller ClusterIP 10.254.58.14 <none> 7000/TCP,8080/TCP 2d
service/argocd-dex-server ClusterIP 10.254.59.220 <none> 5556/TCP,5557/TCP,5558/TCP 2d
service/argocd-metrics ClusterIP 10.254.35.183 <none> 8082/TCP 2d
service/argocd-notifications-controller-metrics ClusterIP 10.254.35.247 <none> 9001/TCP 2d
service/argocd-redis ClusterIP 10.254.35.102 <none> 6379/TCP 2d
service/argocd-repo-server ClusterIP 10.254.21.22 <none> 8081/TCP,8084/TCP 2d
service/argocd-server ClusterIP 10.254.27.164 <none> 80/TCP,443/TCP 2d
service/argocd-server-metrics ClusterIP 10.254.35.168 <none> 8083/TCP 2d

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/argocd-applicationset-controller 1/1 1 1 2d
deployment.apps/argocd-dex-server 1/1 1 1 2d
deployment.apps/argocd-notifications-controller 1/1 1 1 2d
deployment.apps/argocd-redis 1/1 1 1 2d
deployment.apps/argocd-repo-server 1/1 1 1 2d
deployment.apps/argocd-server 1/1 1 1 2d

NAME DESIRED CURRENT READY AGE
replicaset.apps/argocd-applicationset-controller-688c5855db 0 0 0 2d
replicaset.apps/argocd-applicationset-controller-7b5f77c495 1 1 1 41h
replicaset.apps/argocd-dex-server-58df896df 1 1 1 41h
replicaset.apps/argocd-dex-server-6dbbc6c548 0 0 0 2d
replicaset.apps/argocd-notifications-controller-5dcb877fd8 1 1 1 42h
replicaset.apps/argocd-notifications-controller-6769d68cc6 0 0 0 2d
replicaset.apps/argocd-redis-58654b9b4b 1 1 1 41h
replicaset.apps/argocd-redis-689cb75c98 0 0 0 2d
replicaset.apps/argocd-repo-server-5d66c4c699 0 0 0 47h
replicaset.apps/argocd-repo-server-7459b57cd 1 1 1 41h
replicaset.apps/argocd-repo-server-88d94bc9c 0 0 0 2d
replicaset.apps/argocd-server-5d9f577d6c 0 0 0 2d
replicaset.apps/argocd-server-67545b758b 1 1 1 42h

NAME READY AGE
statefulset.apps/argocd-application-controller 1/1 2d

遇到的问题

[root@prod-public-runner-k8s-node01 system]# kubectl -n argocd logs -f pod/argocd-repo-server-7459b57cd-xfbgx
time="2023-02-16T14:15:51Z" level=info msg="ArgoCD Repository Server is starting" built="2023-02-16T17:14:51Z" commit=2bf51f401d6700f8e8b9565d9fc3f66dcf60a0b6 port=8081 version=v2.5.0-rc1+2bf51f4
time="2023-02-16T14:15:51Z" level=info msg="Generating self-signed TLS certificate for this session"
time="2023-02-16T14:15:51Z" level=info msg="Initializing GnuPG keyring at /app/config/gpg/keys"
time="2023-02-16T14:15:51Z" level=info msg="gpg --no-permission-warning --logger-fd 1 --batch --gen-key /tmp/gpg-key-recipe735869143" dir= execID=30201
time="2023-02-16T14:15:57Z" level=error msg="`gpg --no-permission-warning --logger-fd 1 --batch --gen-key /tmp/gpg-key-recipe735869143` failed exit status 2" execID=30201
time="2023-02-16T14:15:57Z" level=info msg=Trace args="[gpg --no-permission-warning --logger-fd 1 --batch --gen-key /tmp/gpg-key-recipe735869143]" dir= operation_name="exec gpg" time_ms=6010.954976

解决方法

kubectl -n argocd edit deploy argocd-repo-server
/spec/template/spec/containers/0/securityContext/seccompProfile/typeby的值改为Unconfined

seccompProfile:
type: Unconfined

访问Argocd

访问Argocd的方法有两种

  • Web UI
  • Argocd CLI

Web UI访问

配置Ingress

[root@prod-public-runner-k8s-node01 ~]#  kubectl -n argocd get ingress argocd-server-ingress -o yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
name: argocd-server-ingress
namespace: argocd
spec:
rules:
- host: argocd.roobo.net
http:
paths:
- backend:
service:
name: argocd-server
port:
name: https
path: /
pathType: Prefix
tls:
- hosts:
- argocd.roobo.net
secretName: roobo.net

获取admin密码

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d && echo

使用admin/密码即可通过Ingress端口访问Argocd Web UI。

Argocd CLI访问

下载客户端

wget https://github.com/argoproj/argo-cd/releases/download/v2.6.1/argocd-linux-amd64

sudo cp argocd-linux-amd64 /usr/local/bin/argocd

sudo chmod 777 /usr/local/bin/argocd

登录

argocd login <argocd-server> --grpc-web

一键登录
argocd login <argocd-server> --username admin --password $(kubectl --kubeconfig=$KCONFIG -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d ) --insecure --grpc-web

修改密码

argocd login <argocd-server>
argocd account list
argocd account update-password \
--account <name> \
--current-password <current-admin> \
--new-password <new-user-password>

使用CLI创建测试应用

创建Argocd应用

Web方式大家都很熟悉,这里以Cli 命令行方式创建演示
更多命令行说明可参考官网文档

添加kubernetes集群

默认情况下Argocd已经将当前k8s集群添加到了Clusters列表中,如果想CD到其他集群需要执行命令添加

1. 首先把目标肌群的kube.config下载下来
2. $clustername 是 kube.config 中/contexts/name 字段的值 (集群名称)
argocd cluster add $clustername --kubeconfig kube.config

[root@prod-public-runner-k8s-node01 ~]# argocd cluster add 266043569625004212-c7ded9566e35f4533b4dd76e1abdae02a --kubeconfig server-k8s.kube.config
WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `266043569625004212-c7ded9566e35f4533b4dd76e1abdae02a` with full cluster level privileges. Do you want to continue [y/N]? y
INFO[0001] ServiceAccount "argocd-manager" already exists in namespace "kube-system"
INFO[0001] ClusterRole "argocd-manager-role" updated
INFO[0001] ClusterRoleBinding "argocd-manager-role-binding" updated
Cluster 'https://182.92.***.***:6443' added

验证

[root@prod-public-runner-k8s-node01 ~]# argocd cluster list
SERVER NAME VERSION STATUS MESSAGE PROJECT
https://182.92.***.***:6443 266043569625004212-c7ded9566e35f4533b4dd76e1abdae02a Unknown Cluster has no applications and is not being monitored.
https://kubernetes.default.svc in-cluster 1.20 Successful

argocd-addcluster

添加Repositories

[root@prod-public-runner-k8s-node01 ~]# argocd repo add http://git.******.com/op/devops-argocd-test.git --username sungaomeng --password ******
Repository 'http://git.******.com/op/devops-argocd-test.git' added

验证

[root@prod-public-runner-k8s-node01 ~]# argocd repo list
TYPE NAME REPO INSECURE OCI LFS CREDS STATUS MESSAGE PROJECT
git http://git.******.com/op/devops-argocd-test.git false false false true Successful

argocd-addrepo

添加Applications

[root@prod-public-runner-k8s-node01 ~]# cat test.yaml 
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: devops-argocd-test
namespace: argocd
spec:
project: default # 定义的项目名
source:
repoURL: http://git.******.com/op/devops-argocd-test.git # git地址
targetRevision: master # git分支
path: manifests # git路径对应到目录下的配置
destination:
server: https://182.92.***.***:6443 # k8s api
namespace: my-app # 名称空间

[root@prod-public-runner-k8s-node01 ~]# kubectl apply -f test.yaml
application.argoproj.io/devops-argocd-test created

验证

[root@prod-public-runner-k8s-node01 ~]# argocd app list
NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET
argocd/devops-argocd-test https://182.92.234.138:6443 my-app default OutOfSync Missing <none> <none> http://git.365jiating.com/op/devops-argocd-test.git manifests master

argocd-addapplications

devops-argocd-test 模板

❯ find manifests
manifests
manifests/deployment.yaml
❯ cat manifests/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
run: argocd-test-app
name: argocd-test-app
spec:
replicas: 1
selector:
matchLabels:
run: argocd-test-app
template:
metadata:
labels:
run: argocd-test-app
spec:
containers:
- image: registry.cn-hangzhou.aliyuncs.com/rookieops/argocd-test-app:v1
imagePullPolicy: IfNotPresent
name: argocd-test-app
restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
labels:
run: argocd-test-app
name: argocd-test-app
spec:
ports:
- name: tcp-8080
port: 8080
protocol: TCP
targetPort: 8080
selector:
run: argocd-test-app
type: NodePort

同步Applications

添加应用后,默认需要手动sync下

可以使用Web UI 点击SYNC同步,或者使用命令行同步

[root@prod-public-runner-k8s-node01 ~]# argocd app list
NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET
argocd/devops-argocd-test https://182.92.***.***:6443 my-app default OutOfSync Missing <none> <none> http://git.******.com/op/devops-argocd-test.git manifests master
[root@prod-public-runner-k8s-node01 ~]# argocd app sync devops-argocd-test
TIMESTAMP GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
2023-02-17T14:30:55+08:00 Service my-app argocd-test-app OutOfSync Missing
2023-02-17T14:30:55+08:00 apps Deployment my-app argocd-test-app OutOfSync Missing
2023-02-17T14:30:56+08:00 Service my-app argocd-test-app Synced Healthy
2023-02-17T14:30:56+08:00 Service my-app argocd-test-app Synced Healthy service/argocd-test-app created
2023-02-17T14:30:56+08:00 apps Deployment my-app argocd-test-app OutOfSync Missing deployment.apps/argocd-test-app created
2023-02-17T14:30:56+08:00 apps Deployment my-app argocd-test-app Synced Progressing deployment.apps/argocd-test-app created

Name: argocd/devops-argocd-test
Project: default
Server: https://182.92.***.***:6443
Namespace: my-app
URL: https://argocd.roobo.net/applications/devops-argocd-test
Repo: http://git.******.com/op/devops-argocd-test.git
Target: master
Path: manifests
SyncWindow: Sync Allowed
Sync Policy: <none>
Sync Status: Synced to master (18be256)
Health Status: Progressing

Operation: Sync
Sync Revision: 18be2568288fff0143a4c3a127ea0866cb39a583
Phase: Succeeded
Start: 2023-02-17 14:30:55 +0800 CST
Finished: 2023-02-17 14:30:56 +0800 CST
Duration: 1s
Message: successfully synced (all tasks run)

GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
Service my-app argocd-test-app Synced Healthy service/argocd-test-app created
apps Deployment my-app argocd-test-app Synced Progressing deployment.apps/argocd-test-app created

Web UI 此时显示绿色状态

argocd-applicationssync

argocd-applicationssyncstatus

在目标集群查看应用状态

[root@prod-server-k8s0155010node ~]# kubelet -n my-app get all
NAME READY STATUS RESTARTS AGE
pod/argocd-test-app-86d78cd54c-jpmjx 1/1 Running 0 3m21s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/argocd-test-app NodePort 172.31.0.27 <none> 8080:30573/TCP 3m21s

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/argocd-test-app 1/1 1 1 3m21s

NAME DESIRED CURRENT READY AGE
replicaset.apps/argocd-test-app-86d78cd54c 1 1 1 3m21s


[root@prod-server-k8s0155010node ~]# kubectl -n my-app get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
argocd-test-app NodePort 172.31.0.27 <none> 8080:30573/TCP 8m35s

[root@prod-server-k8s0155010node ~]# curl 172.31.0.27:8080
{"data":"这是第一版,Hello Joker","version":"v1"}

此时应用版本为 “version”:”v1”

验证CD功能

测试CD流程

Argo CD 默认情况下每 3 分钟会检测 Git 仓库一次,用于判断应用实际状态是否和 Git 中声明的期望状态一致,如果不一致,状态就转换为 OutOfSync。默认情况下并不会触发更新,除非通过 syncPolicy 配置了自动同步。

如果嫌周期性同步太慢了,也可以通过设置 Webhook 来使 Git 仓库更新时立即触发同步。具体的使用方式会放到后续的教程中,本文不再赘述。

手动触发(SYNC)

  1. devops-argocd-test 仓库将manifests/deployment.yaml中 image 由v1改为v2 并提交master分支
  2. 3分钟左右agrocd会检测到,并将App状态转换为OutOfSync
  3. 此时就需要手动点击SYNC 将更改同步到目标集群

argocd-app-outoofsync

argocd-app-outoofsyncstatus

[root@prod-public-runner-k8s-node01 kube-config]# argocd app list
NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET
argocd/devops-argocd-test https://182.92.***.***:6443 my-app default OutOfSync Healthy <none> <none> http://git.******.com/op/devops-argocd-test.git manifests master

[root@prod-public-runner-k8s-node01 kube-config]# argocd app sync devops-argocd-test
TIMESTAMP GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
2023-02-17T14:53:25+08:00 Service my-app argocd-test-app Synced Healthy
2023-02-17T14:53:25+08:00 apps Deployment my-app argocd-test-app OutOfSync Healthy

Name: argocd/devops-argocd-test
Project: default
Server: https://182.92.***.***:6443
Namespace: my-app
URL: https://argocd.roobo.net/applications/devops-argocd-test
Repo: http://git.******.com/op/devops-argocd-test.git
Target: master
Path: manifests
SyncWindow: Sync Allowed
Sync Policy: <none>
Sync Status: Synced to master (1c52f4d)
Health Status: Progressing

Operation: Sync
Sync Revision: 1c52f4d2c229361cbdfa30f36f723e2795855fd1
Phase: Succeeded
Start: 2023-02-17 14:53:25 +0800 CST
Finished: 2023-02-17 14:53:25 +0800 CST
Duration: 0s
Message: successfully synced (all tasks run)

GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE
Service my-app argocd-test-app Synced Healthy service/argocd-test-app unchanged
apps Deployment my-app argocd-test-app Synced Progressing deployment.apps/argocd-test-app configured


[root@prod-public-runner-k8s-node01 kube-config]# argocd app list
NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET
argocd/devops-argocd-test https://182.92.***.***:6443 my-app default Synced Healthy <none> <none> http://git.365jiating.com/op/devops-argocd-test.git manifests master

在目标集群查看应用状态

[root@prod-server-k8s0155010node ~]# kubectl -n my-app get all
NAME READY STATUS RESTARTS AGE
pod/argocd-test-app-68546f4f4b-tt6kq 1/1 Running 0 2m15s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/argocd-test-app NodePort 172.31.0.27 <none> 8080:30573/TCP 20m

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/argocd-test-app 1/1 1 1 20m

NAME DESIRED CURRENT READY AGE
replicaset.apps/argocd-test-app-68546f4f4b 1 1 1 2m15s
replicaset.apps/argocd-test-app-86d78cd54c 0 0 0 20m
[root@prod-server-k8s0155010node ~]# curl 172.31.0.27:8080
{"data":"这是第二版,Hello JackMa","version":"v2"}

此时应用版本为 “version”:”v2”

自动触发

为应用开启自动同步,开启后检测到状态不一致将会自动同步,无需手动sync

配置自动同步运行:

argocd app set devops-argocd-test --sync-policy automated

或者 kubectl -n argocd edit app devops-argocd-test

spec:
syncPolicy:
automated: {}

测试下

  1. devops-argocd-test 仓库将manifests/deployment.yaml中 image 由v2改回v1,并将replicas改为2 ,提交master分支
  2. 3分钟左右agrocd会检测到,并自动同步到目标集群

argocd-applicationsautosyncstatus

在目标集群查看应用状态

[root@prod-server-k8s0155010node ~]# kubectl -n my-app get all
NAME READY STATUS RESTARTS AGE
pod/argocd-test-app-86d78cd54c-84bf5 1/1 Running 0 13m
pod/argocd-test-app-86d78cd54c-ffpr8 1/1 Running 0 13m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/argocd-test-app NodePort 172.31.0.27 <none> 8080:30573/TCP 44m

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/argocd-test-app 2/2 2 2 44m

NAME DESIRED CURRENT READY AGE
replicaset.apps/argocd-test-app-68546f4f4b 0 0 0 26m
replicaset.apps/argocd-test-app-86d78cd54c 2 2 2 44m
[root@prod-server-k8s0155010node ~]#
[root@prod-server-k8s0155010node ~]# curl 172.31.0.27:8080
{"data":"这是第一版,Hello Joker","version":"v1"}

此时应用版本为 “version”:”v1” ,副本为2