Kubernetes网络相关问题及解决

0 / 900

本文主要解决以下几个问题:

  • pod无法访问ClusterIP
  • busybox做dns查询
  • pod间互访及访问外网

问题1: pod无法访问ClusterIP

step0 - kube-proxy日志

从kube-proxy的日志中看到Unknown proxy mode "", assuming iptables proxy

$ kubectl logs -n kube-system kube-proxy-5n29r | more
W0720 03:22:47.942827       1 server_others.go:559] Unknown proxy mode "", assuming iptables proxy
I0720 03:22:48.245820       1 node.go:136] Successfully retrieved node IP: 10.160.18.183
I0720 03:22:48.245876       1 server_others.go:186] Using iptables Proxier.
I0720 03:22:48.246253       1 server.go:583] Version: v1.18.3
I0720 03:22:48.395170       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0720 03:22:48.395210       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0720 03:22:48.395578       1 conntrack.go:83] Setting conntrack hashsize to 32768
I0720 03:22:48.414004       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0720 03:22:48.414067       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0720 03:22:48.533638       1 config.go:315] Starting service config controller
I0720 03:22:48.533673       1 shared_informer.go:223] Waiting for caches to sync for service config
I0720 03:22:48.533997       1 config.go:133] Starting endpoints config controller
I0720 03:22:48.534016       1 shared_informer.go:223] Waiting for caches to sync for endpoints config

step1 - 安装相关包

# 在所有节点上

$ apt install ipset ipvsadm

step2 - 加载module

$ modprobe -- ip_vs
$ modprobe -- ip_vs_rr
$ modprobe -- ip_vs_wrr
$ modprobe -- ip_vs_sh
$ modprobe -- nf_conntrack

# 注:很多帖子上都写的是nf_conntrack_ipv4,但在Ubuntu20.04上,这个已经变成了nf_conntrack

# 检查配置

$ lsmod | grep -e ipvs -e nf_conntrack
nf_conntrack_netlink    45056  0
nfnetlink              16384  3 nf_conntrack_netlink,ip_set
nf_conntrack          139264  5 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE,ip_vs
nf_defrag_ipv6         24576  2 nf_conntrack,ip_vs
nf_defrag_ipv4         16384  1 nf_conntrack
libcrc32c              16384  3 nf_conntrack,nf_nat,ip_vs

# 再注:Ubuntu 20.04上这些module可能默认已经加载了

注:如果ipvs默认没有加载的话,需要写一个脚本,系统重启时也需要加载

step3 - 修改kube-proxy配置文件

修改kube-proxy的configmap中的mode字段为ipvs

$ kubectl edit configmap kube-proxy -n kube-system
...
kind: KubeProxyConfiguration
metricsBindAddress: ""
mode: "ipvs"
nodePortAddresses: null
...

step4 - 重启kube-proxy

可以逐个删除kube-proxy的pod,由k8s自动重启,也可以批量删除

$ kubectl get pod -n kube-system | grep kube-proxy |awk '{system("kubectl delete pod "$1" -n kube-system")}'

查看kube-proxy的日志

$ kubectl logs -n kube-system kube-proxy-44zw5
I0720 05:37:30.026304       1 node.go:136] Successfully retrieved node IP: 10.160.18.181
I0720 05:37:30.026349       1 server_others.go:259] Using ipvs Proxier.
W0720 05:37:30.026600       1 proxier.go:429] IPVS scheduler not specified, use rr by default
I0720 05:37:30.026814       1 server.go:583] Version: v1.18.3
I0720 05:37:30.027200       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0720 05:37:30.027452       1 config.go:133] Starting endpoints config controller
I0720 05:37:30.027474       1 shared_informer.go:223] Waiting for caches to sync for endpoints config
I0720 05:37:30.027507       1 config.go:315] Starting service config controller
I0720 05:37:30.027529       1 shared_informer.go:223] Waiting for caches to sync for service config
I0720 05:37:30.127736       1 shared_informer.go:230] Caches are synced for endpoints config
I0720 05:37:30.127790       1 shared_informer.go:230] Caches are synced for service config

可以看到Using ipvs Proxier,说明IPVS已经启用了

现在,可以启动一个busybox的container来ping一下coredns的clusterIP了

问题2: busybox做dns查询失败

step0 - 问题现象

在解决了pod无法访问dns clusterIP的问题之后,发现busybox还是无法解析到某个service的IP

kubectl run -i --tty --image busybox dns-test --restart=Never --rm /bin/sh
If you don't see a command prompt, try pressing enter.
/ # nslookup web-0.nginx
;; connection timed out; no servers could be reached

上网查完后发现:busybox的版本高于1.28.4都存在这个问题

step1 - 解决方法

使用1.28.4的busybox镜像执行dns查询

$ kubectl run -i --tty --image busybox:1.28.4 dns-test --restart=Never --rm /bin/sh
If you don't see a command prompt, try pressing enter.
/ # nslookup web-0.nginx
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      web-0.nginx
Address 1: 172.1.2.175 web-0.nginx.default.svc.cluster.local
/ # ping web-0.nginx
PING web-0.nginx (172.1.2.175): 56 data bytes
64 bytes from 172.1.2.175: seq=0 ttl=62 time=1.050 ms
64 bytes from 172.1.2.175: seq=1 ttl=62 time=0.432 ms

问题3: pod互访及访问外网不通

问题原因:iptables

解决方法:

分别在每个节点上执行

$ iptables -P INPUT ACCEPT
$ iptables -P FORWARD ACCEPT
$ iptables -F
$ iptables -L -n
$ iptables -t nat -I POSTROUTING -s 172.1.2.0/24 -j MASQUERADE
# 注:172.1.2.0/24是每个节点的pod-network-cidr

测试

$ kubectl run -i --tty --image busybox:1.28.4 connect-test --restart=Never --rm -- /bin/sh
If you don't see a command prompt, try pressing enter.
/ # ping 10.96.0.10 -c 1
PING 10.96.0.10 (10.96.0.10): 56 data bytes
64 bytes from 10.96.0.10: seq=0 ttl=64 time=0.082 ms

--- 10.96.0.10 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.082/0.082/0.082 ms
/ # ping 10.96.0.1 -c 1
PING 10.96.0.1 (10.96.0.1): 56 data bytes
64 bytes from 10.96.0.1: seq=0 ttl=64 time=0.072 ms

--- 10.96.0.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.072/0.072/0.072 ms
/ # ping 172.1.1.193 -c 1
PING 172.1.1.193 (172.1.1.193): 56 data bytes
64 bytes from 172.1.1.193: seq=0 ttl=64 time=0.108 ms

--- 172.1.1.193 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.108/0.108/0.108 ms
/ # ping 223.5.5.5 -c 1
PING 223.5.5.5 (223.5.5.5): 56 data bytes
64 bytes from 223.5.5.5: seq=0 ttl=114 time=5.659 ms

--- 223.5.5.5 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 5.659/5.659/5.659 ms
/ #

# 注: 10.96.0.1为kube-apiserver的ClusterIP

# 10.96.0.10为coredns的ClusterIP

# 172.1.1.193为一个pod的IP
使iptables规则重启生效

分别在每个节点上执行:

$ iptables-save > /etc/iptables.up.rules
$ echo -e '#!/bin/bash\n/sbin/iptables-restore < /etc/iptables.up.rules' > /etc/network/if-pre-up.d/iptables
$ chmod +x /etc/network/if-pre-up.d/iptables