大橙子网站建设,新征程启航
为企业提供网站建设、域名注册、服务器等服务
容器内频现无法访问外部服务,是用ping
测试有如下现象:
# ping baidu.com -c 4
PING baidu.com (110.242.68.66) 56(84) bytes of data.
64 bytes from 110.242.68.66 (110.242.68.66): icmp_seq=1 ttl=49 time=34.3 ms
64 bytes from 110.242.68.66 (110.242.68.66): icmp_seq=2 ttl=49 time=38.4 ms
64 bytes from 110.242.68.66 (110.242.68.66): icmp_seq=3 ttl=49 time=43.4 ms
64 bytes from 110.242.68.66 (110.242.68.66): icmp_seq=4 ttl=49 time=35.9 ms
--- baidu.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3013ms
rtt min/avg/max/mdev = 34.303/38.032/43.460/3.474 ms
#
# ping baidu.com
ping: www.baidu.com: Name or service not known
从容器所在宿主机dig
测试无异常,ping
测试无异常。
首先检查coredns pod运行是否正常:
kubectl -n kube-system get pod -o wide -l k8s-app=kube-dns
如果coredns pod不处于running状态,或pod频繁发生重启,则应检查pod运行异常原因:
kubectl -n kube-system describe pod
2、检查coredns日志如果pod状态正常,检查coredns日志:
kubectl -n kube-system logs
本次故障场景中,可在日志中看到大量如下报错:
[ERROR] plugin/errors: 2 localhost. AAAA: dial tcp 114.114.115.115:53: i/o timeout
[ERROR] plugin/errors: 2 localhost. AAAA: dial tcp 114.114.115.115:53: i/o timeout
[ERROR] plugin/errors: 2 localhost. AAAA: dial tcp 114.114.114.114:53: i/o timeout
[ERROR] plugin/errors: 2 ip. A: dial tcp 114.114.115.115:53: i/o timeout
[ERROR] plugin/errors: 2 localhost. AAAA: dial tcp 114.114.114.114:53: i/o timeout
[ERROR] plugin/errors: 2 www.baidu.com. A: dial tcp 114.114.114.114:53: i/o timeout
3、检查coredns负载情况看到上述报错,接着检查coredns pod负载情况:
kubectl -n kube-system top pod -l k8s-app=kube-dns
检查coredns资源限制情况:
# kubectl -n kube-system describe deploy coredns
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
与top命令查询的负载情况比较,如cpu、memory使用量接近Limits
值,考虑增加coredns副本数或适当增加Limits
值。
如上述检查均无异常,继续检查coredns配置文件,以确认步骤2报错原因,本次故障场景的配置文件如下:
# kubectl -n kube-system get cm coredns -o yaml
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . 211.136.17.107 114.114.114.114 114.114.115.115
cache 30
loop
reload
loadbalance
}
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
从上述配置可以看到,当访问外部域名时,域名查询请求转移到预定义DNS服务器211.136.17.107 114.114.114.114 114.114.115.115
,请求在三个DNS服务器间随机分发。
同时检查主机DNS配置:
# cat /etc/resolv.conf
nameserver 211.136.17.107
nameserver 211.136.20.203
发现主机与容器使用了不同的DNS服务器,而日志中大量出现的报错内容仅涉及容器是用的DNS服务器114.114.114.114 114.114.115.115
,且为tcp
查询。
是用如下命令测试dns服务器:
telnet 211.136.17.107 53
telnet 114.114.114.114 53
telnet 114.114.115.115 53
或如下命令进行更精准的测试:
测试tcp dns:
# dig +tcp @114.114.114.114 www.baidu.com
;; Connection to 114.114.114.114#53(114.114.114.114) for www.baidu.com failed: timed out.
;; Connection to 114.114.114.114#53(114.114.114.114) for www.baidu.com failed: timed out.
;<<>>DiG 9.11.5-P4-5.1+deb10u8-Debian<<>>+tcp @114.114.114.114 www.baidu.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
;; Connection to 114.114.114.114#53(114.114.114.114) for www.baidu.com failed: timed out.
测试udp dns:
# dig @114.114.114.114 www.baidu.com
;<<>>DiG 9.11.5-P4-5.1+deb10u8-Debian<<>>@114.114.114.114 www.baidu.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27280
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www.baidu.com. IN A
;; ANSWER SECTION:
www.baidu.com. 241 IN A 39.156.66.18
www.baidu.com. 241 IN A 39.156.66.14
;; Query time: 21 msec
;; SERVER: 114.114.114.114#53(114.114.114.114)
;; WHEN: Mon Dec 05 11:31:00 CST 2022
;; MSG SIZE rcvd: 74
多次测试可以发现,服务器访问DNS服务器114.114.114.114、114.114.115.115的tcp 53端口网络异常,且异常频率较高,可能因为移动云屏蔽了tcp 53端口的访问或屏蔽了友商的DNS服务器(测试了多个DNS服务器均有此现象)。
处理方法修改coredns配置文件,添加prefer_udp
参数,并删除友商DNS服务器:
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf {
prefer_udp
}
cache 30
loop
reload
loadbalance
}
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
prefer_udp
参数的表示即使查询请求通过tcp传入,也首先尝试使用udp。如果响应被截断(在响应中设置了 TC 标志),再通过tcp进行另一次尝试。可参看:https://coredns.io/plugins/forward/
有关Coredns排障的方法,也可参看:https://help.aliyun.com/document_detail/404754.html
你是否还在寻找稳定的海外服务器提供商?创新互联www.cdcxhl.cn海外机房具备T级流量清洗系统配攻击溯源,准确流量调度确保服务器高可用性,企业级服务器适合批量采购,新人活动首月15元起,快前往官网查看详情吧