ubuntu20.04 环境下Kubernetes与Kubeflow一站式搭建

0 / 623

前言
服务器系统:ubuntu20.04
服务器网络:192.168.1.109, 192.168.1.110
Kubernetes 版本:1.20.1
Docker 版本:18.09.8
Kubeflow 版本:1.3.1

配置要求

Kubernetes 的旧版本可能与最新的 Kubeflow 版本不兼容,前往链接自行查看:https://www.kubeflow.org/docs/started/k8s/overview/#minimum-system-requirements

安装 kubernetes

1.关闭防火墙、selinux 以及 swap(master 节点和 node 节点均执行)
a、关闭防火墙

sudo ufw disable

b、关闭 selinux(自行选择)

sudo apt install selinux-utils

临时关闭:
setenforce 0 # 临时

永久关闭:
sed -i 's/enforcing/disabled' /etc/selinux/config

c、关闭 swap

sudo sed -i 's/^.*swap/#&/g' /etc/fstab
sudo swapoff -a # 临时

2.将桥接的 IPV4 流量传递到 iptables(master 节点和 node 节点均执行)
指令如下:

sudo tee /etc/sysctl.d/k8s.conf <<-'EOF'
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward=1
EOF

使其生效:

sudo sysctl --system

3.时间同步(master 节点和 node 节点均执行)

sudo apt install ntpdate -y
sudo ntpdate time.windows.com

4.安装 Docker(master 节点和 node 节点均执行)
卸载旧版本:

sudo apt-get remove docker docker-engine docker-ce docker.io # 卸载旧版本

更新 sources.list(提示:更新前请自行备份 sources.list):

sudo tee /etc/apt/sources.list <<-'EOF'
deb http://mirrors.aliyun.com/ubuntu/ focal main
deb-src http://mirrors.aliyun.com/ubuntu/ focal main

deb http://mirrors.aliyun.com/ubuntu/ focal-updates main
deb-src http://mirrors.aliyun.com/ubuntu/ focal-updates main

deb http://mirrors.aliyun.com/ubuntu/ focal universe
deb-src http://mirrors.aliyun.com/ubuntu/ focal universe
deb http://mirrors.aliyun.com/ubuntu/ focal-updates universe
deb-src http://mirrors.aliyun.com/ubuntu/ focal-updates universe

deb http://mirrors.aliyun.com/ubuntu/ focal-security main
deb-src http://mirrors.aliyun.com/ubuntu/ focal-security main
deb http://mirrors.aliyun.com/ubuntu/ focal-security universe
deb-src http://mirrors.aliyun.com/ubuntu/ focal-security universe
EOF

安装 docker(依次执行):

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
apt-cache madison docker-ce # 查看docker版本列表,选择安装自己需要的版本(此处选择安装18.09.8)
sudo apt install docker-ce

更新用户组

sudo groupadd docker
sudo gpasswd -a $USER docker
newgrp docker

创建 HTTP/HTTPS 代理文件

sudo vim /etc/profile
# 加入
export http_proxy=http://192.168.1.110:1080
export https_proxy=http://192.168.1.110:1080
export no_proxy="127.0.0.1,192.168.1.0/24,localhost,10.96.0.0/12,10.244.0.0/16"

#需要取消的话
$ unset http_proxy
$ unset https_proxy

# 创建docker.service目录
$ sudo mkdir -p /etc/systemd/system/docker.service.d
$ sudo vim /etc/systemd/system/docker.service.d/http-proxy.conf

具体配置内容: # proxy-addr 为代理 IP 或域名;proxy-port 为代理端口;NO_PROXY 后面接不需要代理的私有仓库的域名或者 IP,以英文逗号结尾

[Service]
Environment="HTTP_PROXY=http://proxy-addr:proxy-port"       #代理服务器地址
Environment="HTTPS_PROXY=http://proxy-addr:proxy-port"     #proxy-addr也是http开头,而不是https,否则会报错
Environment="NO_PROXY=localhost,127.0.0.0/8,docker-registry.example.com,.corp"  #哪些不需要代理

查看配置结果

$ sudo systemctl daemon-reload
$ sudo systemctl show --property=Environment docker
Environment=HTTP_PROXY=http://proxy-addr:proxy-port/ HTTPS_PROXY=http://proxy-addr:proxy-port/ NO_PROXY=localhost,127.0.0.0/8,docker-registry.example.com,.corp

安装 nvidia-docker2:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
&& curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \
&& curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update
sudo apt-get install -y nvidia-docker2

可能出现两个问题:

gpg: 找不到有效的 OpenPGP (源于指令:curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -)

E: 无法定位软件包 nvidia-docker2 (源于指令:curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list)

需要把ip地址写入host文件中!!

sudo vim /etc/hosts
复制以下ip到host中

185.199.108.153 nvidia.github.io
185.199.109.153 nvidia.github.io
185.199.110.153 nvidia.github.io
185.199.111.153 nvidia.github.io

创建文件 /etc/docker/daemon.json ,添加内容,命令如下:

sudo mkdir -p /etc/docker

sudo tee /etc/docker/daemon.json <<-'EOF'
{
"runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
 },
"exec-opts": ["native.cgroupdriver=systemd"],
"registry-mirrors": ["http://hub-mirror.c.163.com"],
"default-runtime": "nvidia"
}
EOF

重启 docker 并设置开机自启动:

sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl enable docker

验证安装:

docker run --rm nvidia/cuda:11.0-base nvidia-smi

关闭 dns 服务,注释 dns,内容如下:

sudo vim /etc/NetworkManager/NetworkManager.conf # 打开配置文件
[main]
plugins=ifupdown,keyfile,ofono
#dns=dnsmasq  # 这行注释

保存并关闭,重启服务:

sudo systemctl restart network-manager

5.安装 kubeadm,kubelet,kubectl(master 节点和 node 节点均执行)

sudo apt-get update && sudo apt-get install -y apt-transport-https

curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -

创建文件 /etc/apt/sources.list.d/kubernetes.list 并添加内容, 指令如下:

sudo tee /etc/apt/sources.list.d/kubernetes.list <<-'EOF'
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF

sudo apt-get update
sudo apt-get install -y kubelet=1.20.1-00 kubeadm=1.20.1-00 kubectl=1.20.1-00

提示:以上步骤,master 节点和 node 节点均需执行,后面的步骤 master 节点和 node 节点执行略有不同

创建集群并初始化(仅 master 节点执行):

sudo kubeadm init --kubernetes-version v1.20.1 --apiserver-advertise-address=192.168.1.109 --pod-network-cidr=10.244.0.0/16 --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers --upload-certs |tee kubeadm-init.log

使用 ifstat 工具实时观测下网卡速率,等待安装完成后,根据提示初始化 kubectl,例:

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
sudo scp /run/systemd/resolve/resolv.conf openoker@192.168.1.110:/run/systemd/resolve/resolv.conf

以下内容用于添加 node 节点,是由主节点初始化后生成的,注意保存对应的自己机器生成的信息,例:

kubeadm join 192.168.1.109:6443 --token y07gsn.pd62oi5wbujgfnxs
--discovery-token-ca-cert-hash sha256:efc9349a6b0a5a2d2384ab5b568b9f35b3041f80d3474f5b9c631d4268feadb6

如果出错,重置并重新初始化,直至成功:

sudo kubeadm reset

sudo kubeadm init --kubernetes-version v1.20.1 --apiserver-advertise-address=192.168.1.109 --pod-network-cidr=10.244.0.0/16 --upload-certs |tee kubeadm-init.log --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers

设置开机启动:

sudo systemctl enable kubelet

启动 k8s 服务程序

sudo systemctl start kubelet

安装 flannel 网络插件:

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

kubectl apply -f kube-flannel.yml

#重启pod
kubectl get pod -n kube-system | grep kube-proxy |awk '{system("kubectl delete pod "$1" -n kube-system")}'

查看状态

kubectl get pod -n kube-system

有时会有 coreDNS 两个容器起不来

执行命令$ sudo vim /run/flannel/subnet.env,添加以下内容:
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true

让 master 也可以接受调度:

kubectl get nodes 

kubectl taint node xxx-nodename node-role.kubernetes.io/master-  #将 Master 也当作 Node 使用
kubectl taint node xxx-nodename node-role.kubernetes.io/master="":NoSchedule #将 Master 恢复成 Master Only 状态

此处选用 flannel 网络插件原因如下:之前环境测试过程中选择了 calico 网络插件,导致安装 kubeflow 后有几个 pod 一直起不来,猜测可能由网络问题导致,如有需要,calico 网络插件安装方式如下
安装 calico 网络插件:

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

查看状态:

kubectl get pod -n kube-system

如果插件 pod 起不来,重启机器,然后依次执行以下命令:

sudo ufw disable
setenforce 0
sudo swapoff -a
sudo systemctl restart docker
sudo systemctl restart kubelet.service

添加 Node 节点
Node 节点环境配置方式如下
将主节点中的/etc/kubernetes/admin.conf 文件拷贝到 node 节点相同的目录下,执行远程拷贝指令:

sudo scp /etc/kubernetes/admin.conf openoker@192.168.1.110:/etc/kubernetes

注意:该指令在主节点下执行,openoker@192.168.1.110 为 node 节点 IP,需换成你自己的 IP

然后依次执行:

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

kubeadm join 192.168.1.109:6443 --token y07gsn.pd62oi5wbujgfnxs 
--discovery-token-ca-cert-hash sha256:efc9349a6b0a5a2d2384ab5b568b9f35b3041f80d3474f5b9c631d4268feadb6

注意:记得关闭 swap

安装网络插件,Node 节点安装网络插件方式与主节点方式一致。

  • 重要额外配置:
在Master节点修改k8s配置文件,进入 /etc/kubernetes/manifests/

修改 kube-apiserver.yaml,加入如下内容
# v1.21.X以下版本需要执行,解决configmap "istio-ca-root-cert" not found报错
- --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
- --service-account-issuer=kubernetes.default.svc

# 升级到1.20.X以上版本需要执行,解决NFS动态存储一直Pending的问题
- --feature-gates=RemoveSelfLink=false 

修改 kube-controller-manager.yaml 、kube-scheduler.yaml,将 - --port=0 注释掉

重启kubelet:systemctl restart kubelet

至此 kubernetes 集群部署完毕

安装 kubeflow

设置 GPU

将以下内容保存成 nvidia-device-plugin.yml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      # This annotation is deprecated. Kept here for backward compatibility
      # See https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ""
      labels:
        name: nvidia-device-plugin-ds
    spec:
      tolerations:
      # This toleration is deprecated. Kept here for backward compatibility
      # See https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/
      - key: CriticalAddonsOnly
        operator: Exists
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      # Mark this pod as a critical add-on; when enabled, the critical add-on
      # scheduler reserves resources for critical add-on pods so that they can
      # be rescheduled after a failure.
      # See https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/
      priorityClassName: "system-node-critical"
      containers:
      - image: nvcr.io/nvidia/k8s-device-plugin:v0.9.0
        name: nvidia-device-plugin-ctr
        args: ["--fail-on-init-error=false"]
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        volumeMounts:
          - name: device-plugin
            mountPath: /var/lib/kubelet/device-plugins
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins

然后执行

$ kubectl apply -f nvidia-device-plugin.yml

可以通过一下命令追踪 pods 的创建状态:

$ kubectl get pods -n kube-system | grep nvidia
nvidia-device-plugin-daemonset-74kv8       1/1     Running            0          2d4h
nvidia-device-plugin-daemonset-75845       1/1     Running            0          2d4h
nvidia-device-plugin-daemonset-8nlsp       1/1     Running            0          2d4h
nvidia-device-plugin-daemonset-rnq8w       1/1     Running            0          2d4h

有几台机器,就会有几个 pods 被创建。

设置 NFS、创建 PV

在安装 kubeflow 之前,首先设置 nfs 并创建 pv,安装 kubeflow 时 MySQL、katib 等组件时需要使用 pvc,如果不提前创建,会导致这些节点状态为 pending

选择一台机器作为 nfs 服务器

sudo apt install nfs-kernel-server 

#创建共享文件夹
sudo mkdir /data/nfs-kubeflow
cd /data/nfs-kubeflow
sudo mkdir v1
sudo mkdir v2
sudo mkdir v3
sudo mkdir v4
sudo mkdir v5
sudo chmod -R 777 /data/nfs-kubeflow

#配置nfs
sudo vim /etc/exports
加入如下内容:
/data/nfs-kubeflow *(insecure,rw,no_root_squash,no_all_squash,sync)

#使配置生效
sudo exportfs  -r
#启动服务rpcbind、nfs服务
sudo service nfs-kernel-server restart

#查看效果
sudo showmount -e
# 在Node节点测试
sudo apt-get install nfs-common
sudo mount -t nfs -o nolock,nfsvers=4,rsize=2048,wsize=2048,timeo=15 192.168.1.109:/data/nfs-kubeflow /data/nfs-kubeflow

如果nfs服务卡死,甚至kill -9也不行,可以通过连续执行“sudo umount -f /data/nfs-kubeflow/”后退出。

配置 pv 文件 pv.yaml,加入如下内容,注意替换 path 与 server

apiVersion: v1
kind: PersistentVolume
metadata:
    name: pv001
    labels:
        name: pv001
spec:
    nfs:
        path: /data/nfs-kubeflow/v1
        server: 192.168.1.109
    accessModes: ["ReadWriteMany","ReadWriteOnce"]
    capacity:
        storage: 20Gi
---

apiVersion: v1
kind: PersistentVolume
metadata:
    name: pv002
    labels:
        name: pv002
spec:
    nfs:
        path: /data/nfs-kubeflow/v2
        server: 192.168.1.109
    accessModes: ["ReadWriteMany","ReadWriteOnce"]
    capacity:
        storage: 20Gi
---

apiVersion: v1
kind: PersistentVolume
metadata:
    name: pv003
    labels:
        name: pv003
spec:
    nfs:
        path: /data/nfs-kubeflow/v3
        server: 192.168.1.109
    accessModes: ["ReadWriteMany","ReadWriteOnce"]
    capacity:
        storage: 30Gi
---

apiVersion: v1
kind: PersistentVolume
metadata:
    name: pv004
    labels:
        name: pv004
spec:
    nfs:
        path: /data/nfs-kubeflow/v4
        server: 192.168.1.109
    accessModes: ["ReadWriteMany","ReadWriteOnce"]
    capacity:
        storage: 50Gi

执行 kubectl apply -f pv.yaml 创建 pv

安装 kubeflow

wget https://github.com/kubeflow/manifests/archive/refs/tags/v1.3.1.zip
unzip v1.3.1.zip
wget https://github.com/kubernetes-sigs/kustomize/releases/download/v3.2.0/kustomize_3.2.0_linux_amd64
mv kustomize_3.2.0_linux_amd64 kustomize
chmod +x kustomize
sudo mv kustomize /usr/bin/
cd manifests-1.3.1/

#安装cert-manager
kustomize build common/cert-manager/cert-manager/base | kubectl apply -f -
kustomize build common/cert-manager/kubeflow-issuer/base | kubectl apply -f -

#安装istio
kustomize build common/istio-1-9/istio-crds/base | kubectl apply -f -
kustomize build common/istio-1-9/istio-namespace/base | kubectl apply -f -
kustomize build common/istio-1-9/istio-install/base | kubectl apply -f -

#安装dex
kustomize build common/dex/overlays/istio | kubectl apply -f -

#安装oidc
kustomize build common/oidc-authservice/base | kubectl apply -f -

#安装knative-serving
kubectl apply -f https://github.com/knative/serving/releases/download/v0.17.1/serving-crds.yaml
kustomize build common/knative/knative-serving/base | kubectl apply -f -
kustomize build common/istio-1-9/cluster-local-gateway/base | kubectl apply -f -
kustomize build common/knative/knative-eventing/base | kubectl apply -f -

#创建kubeflow namespace
kustomize build common/kubeflow-namespace/base | kubectl apply -f -

#创建kubeflow-roles
kustomize build common/kubeflow-roles/base | kubectl apply -f -

#创建istio-resource
kustomize build common/istio-1-9/kubeflow-istio-resources/base | kubectl apply -f -

#安装pipeline
kustomize build apps/pipeline/upstream/env/platform-agnostic-multi-user | kubectl apply -f -

#安装kfserving
kustomize build apps/kfserving/upstream/overlays/kubeflow | kubectl apply -f -

#安装katib
kustomize build apps/katib/upstream/installs/katib-with-kubeflow | kubectl apply -f -

#安装kubeflow dashboard
kustomize build apps/centraldashboard/upstream/overlays/istio | kubectl apply -f -

#安装admission-webhook
kustomize build apps/admission-webhook/upstream/overlays/cert-manager | kubectl apply -f -

#安装notebook
kustomize build apps/jupyter/notebook-controller/upstream/overlays/kubeflow | kubectl apply -f -

#安装jupyter
kustomize build apps/jupyter/jupyter-web-app/upstream/overlays/istio | kubectl apply -f -

#安装kfam
kustomize build apps/profiles/upstream/overlays/kubeflow | kubectl apply -f -

#安装volume
kustomize build apps/volumes-web-app/upstream/overlays/istio | kubectl apply -f -

#安装tensorboard
kustomize build apps/tensorboard/tensorboards-web-app/upstream/overlays/istio | kubectl apply -f -
kustomize build apps/tensorboard/tensorboard-controller/upstream/overlays/kubeflow | kubectl apply -f -

#安装各种operator
kustomize build apps/tf-training/upstream/overlays/kubeflow | kubectl apply -f -
kustomize build apps/pytorch-job/upstream/overlays/kubeflow | kubectl apply -f -
kustomize build apps/mpi-job/upstream/overlays/kubeflow | kubectl apply -f -
kustomize build apps/mxnet-job/upstream/overlays/kubeflow | kubectl apply -f -
kustomize build apps/xgboost-job/upstream/overlays/kubeflow | kubectl apply -f -

#创建namespace
kustomize build common/user-namespace/base | kubectl apply -f -

等待所有 pod 均为 running 状态
直接访问(可选):kubectl port-forward svc/istio-ingressgateway --address 0.0.0.0 -n istio-system 8080:80

暴露指定端口 30000,编辑 kubeflow-ui-nodeport.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: istio-ingressgateway
    install.operator.istio.io/owning-resource: unknown
    istio: ingressgateway
    istio.io/rev: default
    operator.istio.io/component: IngressGateways
    release: istio
  name: istio-ingressgateway
  namespace: istio-system
spec:
  ports:
  - name: status-port
    port: 15021
    protocol: TCP
    targetPort: 15021
  - name: http2
    port: 80
    protocol: TCP
    targetPort: 8080
    nodePort: 30000
  - name: https
    port: 443
    protocol: TCP
    targetPort: 8443
  - name: tcp
    port: 31400
    protocol: TCP
    targetPort: 31400
  - name: tls
    port: 15443
    protocol: TCP
    targetPort: 15443
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  type: NodePort

执行 kubectl apply -f kubeflow-ui-nodeport.yaml

到这里 kubeflow1.3 的部署就完成了,可通过 http://ip:30000 来访问,默认账号为 user@example.com,密码为 12341234

设置 storageClass

默认情况下,创建 notebook server 会报错误,原因是没有配置默认的 StorageClass

编辑 storage-nfs.yaml,注意替换 NFS_SERVER、NFS_PATH 内容

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-client-provisioner
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccount: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: nfs-client
            - name: NFS_SERVER
              value: 192.168.1.109
            - name: NFS_PATH
              value: /data/nfs-kubeflow/v5
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.1.109
            path: /data/nfs-kubeflow/v5

编辑 storage-rbac.yaml

kind: ServiceAccount
apiVersion: v1
metadata:
  name: nfs-client-provisioner
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: default      #替换成你要部署NFS Provisioner的 Namespace
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    namespace: default      #替换成你要部署NFS Provisioner的 Namespace
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

编辑 storage-class.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-storage
  annotations:
    storageclass.kubernetes.io/is-default-class: "true" #---设置为默认的storageclass
provisioner: nfs-client    #---动态卷分配者名称,必须和上面创建的"provisioner"变量中设置的Name一致
parameters:
  archiveOnDelete: "true"  #---设置为"false"时删除PVC不会保留数据,"true"则保留数据
allowVolumeExpansion: True  #允许pvc创建后扩容

依次执行如下命令

kubectl apply -f storage-nfs.yaml
kubectl apply -f storage-rbac.yaml
kubectl apply -f storage-class.yaml
kubectl patch storageclass nfs-storage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

设置 Https 访问

如果要通过 NodePort / LoadBalancer / Ingress 暴露服务到非 localhost 网络,那么必须使用 https。否则能打开主页面但是 notebook、SSH 等均无法连接。创建生成 ssl 自签名证书脚本 create_self-signed-cert.sh

#!/bin/bash -e

help ()
{
    echo  ' ================================================================ '
    echo  ' --ssl-domain: 生成ssl证书需要的主域名,如不指定则默认为www.rancher.local,如果是ip访问服务,则可忽略;'
    echo  ' --ssl-trusted-ip: 一般ssl证书只信任域名的访问请求,有时候需要使用ip去访问server,那么需要给ssl证书添加扩展IP,多个IP用逗号隔开;'
    echo  ' --ssl-trusted-domain: 如果想多个域名访问,则添加扩展域名(SSL_TRUSTED_DOMAIN),多个扩展域名用逗号隔开;'
    echo  ' --ssl-size: ssl加密位数,默认2048;'
    echo  ' --ssl-cn: 国家代码(2个字母的代号),默认CN;'
    echo  ' 使用示例:'
    echo  ' ./create_self-signed-cert.sh --ssl-domain=www.test.com --ssl-trusted-domain=www.test2.com \ '
    echo  ' --ssl-trusted-ip=1.1.1.1,2.2.2.2,3.3.3.3 --ssl-size=2048 --ssl-date=3650'
    echo  ' ================================================================'
}

case "$1" in
    -h|--help) help; exit;;
esac

if [[ $1 == '' ]];then
    help;
    exit;
fi

CMDOPTS="$*"
for OPTS in $CMDOPTS;
do
    key=$(echo ${OPTS} | awk -F"=" '{print $1}' )
    value=$(echo ${OPTS} | awk -F"=" '{print $2}' )
    case "$key" in
        --ssl-domain) SSL_DOMAIN=$value ;;
        --ssl-trusted-ip) SSL_TRUSTED_IP=$value ;;
        --ssl-trusted-domain) SSL_TRUSTED_DOMAIN=$value ;;
        --ssl-size) SSL_SIZE=$value ;;
        --ssl-date) SSL_DATE=$value ;;
        --ca-date) CA_DATE=$value ;;
        --ssl-cn) CN=$value ;;
    esac
done

# CA相关配置
CA_DATE=${CA_DATE:-3650}
CA_KEY=${CA_KEY:-cakey.pem}
CA_CERT=${CA_CERT:-cacerts.pem}
CA_DOMAIN=cattle-ca

# ssl相关配置
SSL_CONFIG=${SSL_CONFIG:-$PWD/openssl.cnf}
SSL_DOMAIN=${SSL_DOMAIN:-'www.rancher.local'}
SSL_DATE=${SSL_DATE:-3650}
SSL_SIZE=${SSL_SIZE:-2048}

## 国家代码(2个字母的代号),默认CN;
CN=${CN:-CN}

SSL_KEY=$SSL_DOMAIN.key
SSL_CSR=$SSL_DOMAIN.csr
SSL_CERT=$SSL_DOMAIN.crt

echo -e "\033[32m ---------------------------- \033[0m"
echo -e "\033[32m       | 生成 SSL Cert |       \033[0m"
echo -e "\033[32m ---------------------------- \033[0m"

if [[ -e ./${CA_KEY} ]]; then
    echo -e "\033[32m ====> 1. 发现已存在CA私钥,备份"${CA_KEY}"为"${CA_KEY}"-bak,然后重新创建 \033[0m"
    mv ${CA_KEY} "${CA_KEY}"-bak
    openssl genrsa -out ${CA_KEY} ${SSL_SIZE}
else
    echo -e "\033[32m ====> 1. 生成新的CA私钥 ${CA_KEY} \033[0m"
    openssl genrsa -out ${CA_KEY} ${SSL_SIZE}
fi

if [[ -e ./${CA_CERT} ]]; then
    echo -e "\033[32m ====> 2. 发现已存在CA证书,先备份"${CA_CERT}"为"${CA_CERT}"-bak,然后重新创建 \033[0m"
    mv ${CA_CERT} "${CA_CERT}"-bak
    openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}"
else
    echo -e "\033[32m ====> 2. 生成新的CA证书 ${CA_CERT} \033[0m"
    openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}"
fi

echo -e "\033[32m ====> 3. 生成Openssl配置文件 ${SSL_CONFIG} \033[0m"
cat > ${SSL_CONFIG} <<EOM
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
EOM

if [[ -n ${SSL_TRUSTED_IP} || -n ${SSL_TRUSTED_DOMAIN} ]]; then
    cat >> ${SSL_CONFIG} <<EOM
subjectAltName = @alt_names
[alt_names]
EOM
    IFS=","
    dns=(${SSL_TRUSTED_DOMAIN})
    dns+=(${SSL_DOMAIN})
    for i in "${!dns[@]}"; do
      echo DNS.$((i+1)) = ${dns[$i]} >> ${SSL_CONFIG}
    done

    if [[ -n ${SSL_TRUSTED_IP} ]]; then
        ip=(${SSL_TRUSTED_IP})
        for i in "${!ip[@]}"; do
          echo IP.$((i+1)) = ${ip[$i]} >> ${SSL_CONFIG}
        done
    fi
fi

echo -e "\033[32m ====> 4. 生成服务SSL KEY ${SSL_KEY} \033[0m"
openssl genrsa -out ${SSL_KEY} ${SSL_SIZE}

echo -e "\033[32m ====> 5. 生成服务SSL CSR ${SSL_CSR} \033[0m"
openssl req -sha256 -new -key ${SSL_KEY} -out ${SSL_CSR} -subj "/C=${CN}/CN=${SSL_DOMAIN}" -config ${SSL_CONFIG}

echo -e "\033[32m ====> 6. 生成服务SSL CERT ${SSL_CERT} \033[0m"
openssl x509 -sha256 -req -in ${SSL_CSR} -CA ${CA_CERT} \
    -CAkey ${CA_KEY} -CAcreateserial -out ${SSL_CERT} \
    -days ${SSL_DATE} -extensions v3_req \
    -extfile ${SSL_CONFIG}

echo -e "\033[32m ====> 7. 证书制作完成 \033[0m"
echo
echo -e "\033[32m ====> 8. 以YAML格式输出结果 \033[0m"
echo "----------------------------------------------------------"
echo "ca_key: |"
cat $CA_KEY | sed 's/^/  /'
echo
echo "ca_cert: |"
cat $CA_CERT | sed 's/^/  /'
echo
echo "ssl_key: |"
cat $SSL_KEY | sed 's/^/  /'
echo
echo "ssl_csr: |"
cat $SSL_CSR | sed 's/^/  /'
echo
echo "ssl_cert: |"
cat $SSL_CERT | sed 's/^/  /'
echo

echo -e "\033[32m ====> 9. 附加CA证书到Cert文件 \033[0m"
cat ${CA_CERT} >> ${SSL_CERT}
echo "ssl_cert: |"
cat $SSL_CERT | sed 's/^/  /'
echo

echo -e "\033[32m ====> 10. 重命名服务证书 \033[0m"
echo "cp ${SSL_DOMAIN}.key tls.key"
cp ${SSL_DOMAIN}.key tls.key
echo "cp ${SSL_DOMAIN}.crt tls.crt"
cp ${SSL_DOMAIN}.crt tls.crt
chmod +x create_self-signed-cert.sh

./create_self-signed-cert.sh --ssl-domain=kubeflow.cn

kubectl create --namespace istio-system secret tls kf-tls-cert --key /root/ssl/kubeflow.cn.key --cert /root/ssl/kubeflow.cn.crt

kubectl edit cm config-domain --namespace knative-serving

#在 data 下面添加:kubeflow.cn: ""

编辑 kubeflow-https.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeflow-gateway
  namespace: kubeflow
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
  - hosts:
    - '*'
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: kf-tls-cert

执行 kubectl apply -f kubeflow-https.yaml

执行 kubectl -n istio-system get service istio-ingressgateway 获取 https 端口为 32143

可通过 https://ip:32143 来进行 https 访问

Multi Users 隔离设置

kubeflow 安装结束后,默认情况下只能通过 user@example.com 一个账号登录操作,下面进行多账户号设置,设置完成后,每个账户拥有独立的 namespace 与 profile

首先创建用户的 profile,下面的 YAML 文件创建了 2 个 profile,namespace 分别为 test1 与 test2

apiVersion: kubeflow.org/v1beta1
kind: Profile
metadata:
  name: test1
spec:
  owner:
    kind: User
    name: test1@example.com
---
apiVersion: kubeflow.org/v1beta1
kind: Profile
metadata:
  name: test2
spec:
  owner:
    kind: User
    name: test2@example.com

执行 kubectl get configmap dex -n auth -o jsonpath='{.data.config.yaml}' > dex-yaml.yaml 将 dex 的用户信息输出到 dex-yaml.yaml 文件,其中 user@example.com 部署是按照时创建的,编辑该文件,添加 test1 与 test2 部分

执行 kubectl create configmap dex --from-file=config.yaml=dex-yaml.yaml -n auth --dry-run=client -o YAML | kubectl apply -f - 使新用户生效

执行 kubectl rollout restart deployment dex -n auth 重启 dex,这样就完成了多用户的设置

minio 的使用(可选)

# 获取accesskey和secretkey
kubectl get secret mlpipeline-minio-artifact --namespace=kubeflow -o yaml

kubectl port-forward -n kubeflow svc/minio-service --address 0.0.0.0 9000:9000
# 浏览器访问9000端口

安装 Harbor(可选)

在 kubeflow 的使用过程中,我们需要一个私有的镜像仓库来存放执行过程中生成的镜像文件,这里我们选择部署一套 harbor 作为镜像仓库,也可以选择 docker registry,选择一台机器操作即可

  • 安装 docker-compose:
wget https://github.com/docker/compose/releases/download/1.29.2/docker-compose-Linux-x86_64
mv docker-compose-Linux-x86_64 docker-compose
chmod +x docker-compose
mv docker-compose /usr/bin/
  • 可执行 docker-compose version 验证是否安装成功
wget https://github.com/goharbor/harbor/releases/download/v1.9.2/harbor-offline-installer-v1.9.2.tgz
tar -zxvf harbor-offline-installer-v1.9.2.tgz
cd harbor
vim harbor.yaml
./install.sh
  • 修改 harbor.yaml 中的 hostname 和 port

访问 http://43.128.14.116:86/,即可打开 harbor 页面,43.128.14.116 为服务器外网地址、172.19.0.14 为其内网地址,默认账号 admin,密码为 Harbor12345。登录之后创建一个名为 kubeflow 的公开项目作为我们的私有仓库。

  • 修改 docker 配置文件,/etc/docker/daemon.json,加入如下内容,之后重启 docker

{ "insecure-registries": ["172.19.0.14:86"] }

docker login -u admin -p Harbor12345 172.19.0.14:86

harbor 搭建完成 ,docker-compose up -d 重启

  • 上传及下载镜像到 Harbor
#1.标记镜像
docker tag {镜像名}:{tag} {Harbor地址}:{端口}/{Harbor项目名}/{自定义镜像名}:{自定义tag}
#eg:docker tag vmware/harbor-adminserver:v1.1.0 172.19.0.14:86/test/harbor-adminserver:v1.1.0

#2.push 到Harbor
docker push {Harbor地址}:{端口}/{自定义镜像名}:{自定义tag}
#eg:docker push 172.19.0.14:86/test/harbor-adminserver:v1.1.0

#3.pull 到本地
docker pull 172.19.0.14:86/test/harbor-adminserver:v1.1.0

常用命令

  • 清除状态为 Evicted 的 pod
kubectl get pods -nkubeflow | grep Evicted | awk '{print $2}' | xargs kubectl delete pod -nkubeflow

常见问题

  • 通过 SSH 连接 ubuntu20.04 有问题
sudo vim /etc/ssh/sshd_config最后添加如下设置:

KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1

sudo systemctl restart sshd
  • no available network
安装完docker启动遇到如下错误:
Error starting daemon:Error initializing network controller: list bridge addresses failed: no available network

执行如下命令

sudo ip link add name docker0 type bridge
sudo ip addr add dev docker0 172.17.42.1/16

重启docker就解决了
  • 报错:unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
方法一:回退k8s到v1.22+以下
方法二:apiextensions.k8s.io/v1beta1  用 apiextensions.k8s.io/v1 替换
  • 报错:error: unable to recognize "STDIN": no matches for kind "CompositeController" in version "metacontroller.k8s.io/v1alpha1"
把apps/pipeline/upstream/env/platform-agnostic-multi-user/kustomization.yaml文件里的metacontroller.k8s.io/v1alpha1 替换成 kustomize.config.k8s.io/v1beta1
  • 报错:failed to set bridge addr: "cni0" already has an IP address different from 10.244.1.1/24
在所有节点执行:
ifconfig cni0 down    
ip link delete cni0
  • 报错:error: unable to upgrade connection: container not found
找不到容器所在worker节点的IP,需要调整master节点和worker节点的配置,执行:sudo vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
添加如下一行:
Environment="KUBELET_EXTRA_ARGS=--node-ip=192.168.1.110"
注意,将“192.168.1.110”替换为节点网络地址

重启kubelet,执行:
sudo systemctl daemon-reload && sudo systemctl restart kubelet.service
  • 报错:MountVolume.SetUp failed for volume "mlpipeline-minio-artifact" : secret "mlpipeline-minio-artifact" not found
kubectl get secret mlpipeline-minio-artifact --namespace=kubeflow -o yaml | sed 's/namespace: kubeflow/namespace: kubeflow-user-example-com/' | kubectl create -f -