K8S使用calico设置成bgp模式默认使用mesh以后3台不同节点上的容器之间不能互通
v8: 192.168.0.108 (master)
v8-2: 192.168.0. 181 (node)
v8-3: 192.168.0.88 (node)
podIP: 10.244.0.0/16
操作环境、软件版本等信息
K8S版本1.21.2
calico版本v3.20.6 calico设置成bgp模式默认使用mesh ,calico.yaml已把ipip改为Never、CALICO_IPV4POOL_CIDR改成10.244.0.0/16,
v8: 192.168.0.108(master)
v8-2: 192.168.0.181(node)
v8-3: 192.168.0.88 (node)
尝试过的解决方法
1、 calico容器正常无日志报错
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-594649bd75-dnc9x 1/1 Running 0 4d18h
calico-node-4j5rv 1/1 Running 0 4d18h
calico-node-6xrch 1/1 Running 0 4d18h
calico-node-rl5ld 1/1 Running 0 4d18h
2、三台 caliao node状态正常
[root@v8 ~]# ./calicoctl node status
Calico process is running.
IPv4 BGP status
+---------------+-------------------+-------+------------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+---------------+-------------------+-------+------------+-------------+
| 192.168.0.181 | node-to-node mesh | up | 2023-11-22 | Established |
| 192.168.0.88 | node-to-node mesh | up | 2023-11-22 | Established |
+---------------+-------------------+-------+------------+-------------+
[root@v8-3 ~]# ./calicoctl node status
Calico process is running.
IPv4 BGP status
+---------------+-------------------+-------+------------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+---------------+-------------------+-------+------------+-------------+
| 192.168.0.108 | node-to-node mesh | up | 2023-11-22 | Established |
| 192.168.0.181 | node-to-node mesh | up | 2023-11-22 | Established |
+---------------+-------------------+-------+------------+-------------+
[root@v8-3 ~]# ./calicoctl get ippool
NAME CIDR SELECTOR
default-ipv4-ippool 10.244.0.0/16 all()
3、容器与不同节点的 宿主机能互ping通但容器与不同节点容器之间不通
容器ping 节点v8-3正常
root@v8 ~]# kubectl exec -it mix-6bffb65b57-6xg8s -- sh
/opt/www # ping 192.168.0.181
PING 192.168.0.181 (192.168.0.181): 56 data bytes
64 bytes from 192.168.0.88: seq=0 ttl=63 time=0.416 ms
64 bytes from 192.168.0.88: seq=1 ttl=63 time=0.331 ms
主机名v8-3,IP:192.168.0.88能ping 通主机名v8,IP :192.168.0.108里面容器
[root@v8-3 ~]# ping 10.244.157.2
PING 10.244.157.2 (10.244.157.2) 56(84) bytes of data.
64 bytes from 10.244.157.2: icmp_seq=1 ttl=63 time=115 ms
64 bytes from 10.244.157.2: icmp_seq=2 ttl=63 time=0.329 ms
主机:192.168.0.108 下的容器10.244.157.2 ping不通192.168.0.88下的容器10.244.100.142
/opt/www # ping 10.244.100.142
PING 10.244.100.142 (10.244.100.142): 56 data bytes
抓包过程
v8-192.168.0.108 容器10.244.157.2
/opt/www # tcpdump host 10.244.100.142 -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:50:56.505063 IP 10.244.157.2 > 10.244.100.142: ICMP echo request, id 56832, seq 0, length 64
10:50:57.505344 IP 10.244.157.2 > 10.244.100.142: ICMP echo request, id 56832, seq 1, length 64
10:50:58.505445 IP 10.244.157.2 > 10.244.100.142: ICMP echo request, id 56832, seq 2, length 64
10:50:59.505552 IP 10.244.157.2 > 10.244.100.142: ICMP echo request, id 56832, seq 3, length 64
192.168.0.108宿主机上抓包
[root@v8 ~]# tcpdump -i eth0 icmp -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:32:11.032797 IP 10.244.157.2 > 10.244.100.142: ICMP echo request, id 60160, seq 15, length 64
12:32:12.032887 IP 10.244.157.2 > 10.244.100.142: ICMP echo request, id 60160, seq 16, length 64
12:32:13.032961 IP 10.244.157.2 > 10.244.100.142: ICMP echo request, id 60160, seq 17, length 64
12:32:14.033033 IP 10.244.157.2 > 10.244.100.142: ICMP echo request, id 60160, seq 18, length 64
在192.168.0.88主机上抓包
无数据，怀疑在192.168.0.108主机上就丢弃数据包了，但路由正常
[root@v8-3 ~]# tcpdump -i cali1279804247d icmp -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on cali1279804247d, link-type EN10MB (Ethernet), capture size 262144 bytes
路由检查
v8----192.168.0.108
[root@v8 ~]# ip route
default via 192.168.0.1 dev eth0 proto dhcp metric 100
10.244.100.128/26 via 192.168.0.88 dev eth0 proto bird
10.244.125.64/26 via 192.168.0.181 dev eth0 proto bird
10.244.157.0 dev cali2143a67a07d scope link
blackhole 10.244.157.0/26 proto bird
10.244.157.1 dev cali24e13e64f3c scope link
**10.244.157.2 dev cali685dddd2f86 scope link
**10.244.157.3 dev cali0cf0a0c076c scope link
10.244.157.4 dev califf7284a32fd scope link
10.244.157.5 dev cali8d6f231a39f scope link
10.244.157.6 dev calif6a082606c1 scope link
10.244.157.7 dev cali30f721aa2f7 scope link
10.244.157.25 dev cali1e98019e276 scope link
169.254.169.254 via 192.168.0.254 dev eth0 proto dhcp metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.108 metric 100
v8--3: 192.168.0.181
[root@v8-3 ~]# ip route
default via 192.168.0.1 dev eth0 proto dhcp metric 100
10.244.100.128 dev calif60ce0e1724 scope link
blackhole 10.244.100.128/26 proto bird
10.244.100.140 dev cali5c1f6b32444 scope link
10.244.100.141 dev cali4d2ccbb62f8 scope link
**10.244.100.142 dev cali1279804247d scope link ******
10.244.125.64/26 via 192.168.0.181 dev eth0 proto bird
10.244.157.0/26 via 192.168.0.108 dev eth0 proto bird
169.254.169.254 via 192.168.0.254 dev eth0 proto dhcp metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev docker_gwbridge proto kernel scope link src 172.18.0.1
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.88 metric 100
iptables检查
Chain cali-from-wl-dispatch (2 references)
target prot opt source destination
cali-fw-cali0cf0a0c076c all -- anywhere anywhere [goto] /* cali:SesY7QVHDLNyf-l5 */
cali-fw-cali1e98019e276 all -- anywhere anywhere [goto] /* cali:YmZZJGKyRFmsXxF7 */
cali-from-wl-dispatch-2 all -- anywhere anywhere [goto] /* cali:wxqMZ2dPO8emNDKO */
cali-fw-cali30f721aa2f7 all -- anywhere anywhere [goto] /* cali:-T0wqOCykO5m6Yct */
cali-fw-cali685dddd2f86 all -- anywhere anywhere [goto] /* cali:d4cCz8Rpr6UlGuOu */
cali-fw-cali8d6f231a39f all -- anywhere anywhere [goto] /* cali:D4iaOuHHDK41VbhZ */
cali-from-wl-dispatch-f all -- anywhere anywhere [goto] /* cali:KpxN190c58iIPStk */
DROP all -- anywhere anywhere /* cali:poEU4sSjI9Z7kQ9g */ /* Unknown interface */
Chain cali-tw-cali685dddd2f86 (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere /* cali:mFwx-gC7cxToJiwu */ ctstate RELATED,ESTABLISHED
DROP all -- anywhere anywhere /* cali:5FBCakypEHLffZgb */ ctstate INVALID
MARK all -- anywhere anywhere /* cali:7x7QkABWmz8I_4dp */ MARK and 0xfffeffff
cali-pri-kns.default all -- anywhere anywhere /* cali:g6ul_iyrHpW9x4rN */
RETURN all -- anywhere anywhere /* cali:nBzTthOv0fR1zK4W */ /* Return if profile accepted */ mark match 0x10000/0x10000
cali-pri-ksa.default.default all -- anywhere anywhere /* cali:lwAZNpSySXnIIZz_ */
RETURN all -- anywhere anywhere /* cali:Xkko9Jc5cvP3OgLu */ /* Return if profile accepted */ mark match 0x10000/0x10000
DROP all -- anywhere anywhere /* cali:W58NKbFYKBE4-BRq */ /* Drop if no profiles matched */
我想要达到的结果
3个不同节点下的容器能通信
