티스토리 뷰
반응형
개요
Kubespray 을 통해 Kubernetes 를 구성 할 때 네트워크 관련에서 별다른 설정을 하지 않고 운영 할 경우 아래와 같은 메시지를 확인 할 수 있다.
$ kubectl get event
LAST SEEN TYPE REASON OBJECT MESSAGE
38s Normal CIDRNotAvailable node/node65.k8s Node node65.k8s status is now: CIDRNotAvailable
48s Normal CIDRNotAvailable node/node66.k8s Node node66.k8s status is now: CIDRNotAvailable
37s Normal CIDRNotAvailable node/node67.k8s Node node67.k8s status is now: CIDRNotAvailable원인
이 문제의 원인은 POD_SUBNET 에서 노드에 할당 가능한 CIDR 이 없기 때문에 발생한 문제다.
이를 일반적으로 64-Node Limit 문제라고 부르고 있으며 아래와 같이 살펴 볼 수 있다.
- Kubespray 기본 설정
- kube_service_addresses: 10.233.0.0/18
- kube_pods_subnet: 10.233.64.0/18
- kube_network_node_prefix: 24
여기서 문제가 되는 부분은 kube_pods_subnet 이라는 항목인데 10.233.64.0/18 은 총 16384 개의 IP를 사용 할 수 있지만 Kubernetes 에서는 노드 단위로 하나의 24 비트의 주소를 할당한다.
kube_pods_subnet가 10.233.64.0/18 이고 kube_network_node_prefix은 24 이니 아래와 같은 대역으로 할당이 가능
- 할당 가능한 대역 (총 64개)
- 10.233.64.0/24
- 10.233.65.0/24
... - 10.233.126.0/24
- 10.233.127.0/24
예) control-plane 3대와 node 70대가 있다고 가정
$ kubectl get nodes -oyaml | egrep "^ name:| podCIDR:"
name: control-plane01.k8s
podCIDR: 10.233.64.0/24
name: control-plane02.k8s
podCIDR: 10.233.65.0/24
...
name: node60.k8s
podCIDR: 10.233.126.0/24
name: node61.k8s
podCIDR: 10.233.127.0/24
node62.k8s (podCIDR 이 표시되지 않음)
...
node70.k8s (podCIDR 이 표시되지 않음)문제점
이와 같은 상황에서 여러가지 질문이 생길 수 있다. 질문을 기반으로 테스트를 진행
- node60.k8s 은 podCIDR 이
10.233.126.0/24으로 할당받았으니 node60.k8s 에서 생성되는 POD는 모두10.233.126.0/24내에서 할당이 되는가?
- 할당이 가능하며 하나의 CIDR 으로 할당이 되는 것을 볼 수 있다.
<네임스페이스 이름> <POD 이름> 1/1 Running 0 <날짜> 10.233.121.245 node60.k8s <none> <none>
<네임스페이스 이름> <POD 이름> 1/1 Running 0 <날짜> 10.233.121.22 node60.k8s <none> <none>
<네임스페이스 이름> <POD 이름> 1/1 Running 0 <날짜> 10.233.121.52 node60.k8s <none> <none>
<네임스페이스 이름> <POD 이름> 1/1 Running 0 <날짜> 10.233.121.137 node60.k8s <none> <none>
<네임스페이스 이름> <POD 이름> 1/1 Running 0 <날짜> 10.233.121.89 node60.k8s <none> <none>- node60.k8s 에는 podCIDR 가
10.233.126.0/24으로 설정이 되어있는데 왜10.233.121.0/24으로 할당되었는지?
kubectl get nodes -oyaml | egrep "^ name:| podCIDR:"명령어로 확인 했을 때 분명히 node60 의 CIDR 은10.233.126.0/24으로 출력되어있지만 해당 서버의 라우팅 테이블을 확인하면 다른 것을 확인 할 수 있다.
$ route -n | grep -v -E "tunl0|cali|eth"
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.233.121.0 0.0.0.0 255.255.255.0 U 0 0 0 *- 이 부분에 대해서는 추가적인 확인이 필요한 것으로 보임
- 그렇다면 CIDR 을 할당받지 못한 노드에서는 POD 구동이 되지 않는가?
- 그렇지 않으며 할당되지 않는 IPPool 에서 할당을 받는다. 아래는 podCIDR 가 없는 하나의 노드에서 확인 결과로 다양한 IP가 할당
<네임스페이스 이름> <POD 이름> 1/1 Running 0 <날짜> 10.233.96.76 node67.k8s <none> <none>
<네임스페이스 이름> <POD 이름> 1/1 Running 0 <날짜> 10.233.95.3 node67.k8s <none> <none>
<네임스페이스 이름> <POD 이름> 1/1 Running 0 <날짜> 10.233.95.97 node67.k8s <none> <none>
<네임스페이스 이름> <POD 이름> 1/1 Running 0 <날짜> 10.233.96.163 node67.k8s <none> <none>
<네임스페이스 이름> <POD 이름> 1/1 Running 0 <날짜> 10.233.95.85 node67.k8s <none> <none>
<네임스페이스 이름> <POD 이름> 1/1 Running 0 <날짜> 10.233.95.121 node67.k8s <none> <none>
<네임스페이스 이름> <POD 이름> 1/1 Running 0 <날짜> 10.233.95.47 node67.k8s <none> <none>
<네임스페이스 이름> <POD 이름> 1/1 Running 0 <날짜> 10.233.98.110 node67.k8s <none> <none>
<네임스페이스 이름> <POD 이름> 1/1 Running 0 <날짜> 10.233.97.181 node67.k8s <none> <none>
<네임스페이스 이름> <POD 이름> 1/1 Running 0 <날짜> 10.233.96.113 node67.k8s <none> <none>- 그렇다면 CIDRNotAvailable 이라는 이벤트 로그는 왜 발생하는거고 왜 관리자에게 출력이 되는건지?
- 제일 우려스러웠던 부분은 IP 의 중복성이였다. 얘를 들자면 node60.k8s 에서는
10.233.121.0/24으로 설정되어 해당 CIDR 을 기준으로 할당하는데 그 외 CIDR이 없는 노드에서10.233.121.0/24으로 할당받는다면 중복이 발생하지 않을까 싶었다. 이를 통해 AWS EC2 에서 아래와 같은 테스트를 진행했다.
CIDRNotAvailable 에 따른 POD IP 중복성 확인
- AWS 구성 환경
- 노드 : control-plane0[1-3].k8s, node0[1-2].k8s (총 5대)
- kube_pods_subnet : 192.168.0.0/24
- 사용 가능한 IP 수 : 256 (게이트웨이 및 브로드캐스트 제외 시 254)
- 192.168.0.0/24 1개의 대역만 사용 가능. 비용 문제 때문에 비싼 EC2을 사용 할 수 없기에 대역을 최소화
- kube_network_node_prefix : 24
- 최초 구성 후 IPPool 및 podCIDR 확인
$ calicoctl ipam show --show-blocks
+----------+----------------+-----------+------------+-----------+
| GROUPING | CIDR | IPS TOTAL | IPS IN USE | IPS FREE |
+----------+----------------+-----------+------------+-----------+
| IP Pool | 192.168.0.0/24 | 256 | 9 (4%) | 247 (96%) |
| Block | 192.168.0.0/24 | 256 | 9 (4%) | 247 (96%) |
+----------+----------------+-----------+------------+-----------+
$ kubectl get nodes -oyaml | egrep "^ name:| podCIDR:"
name: control-plane01.k8s
podCIDR: 192.168.0.0/24
name: control-plane02.k8s
name: control-plane03.k8s
name: node01.k8s
name: node02.k8s- NGINX Deployment 로 247개 POD 생성 및 중복 확인
$ kubectl get pod -o wide -A
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default nginx-86bc984445-242pj 1/1 Running 0 107s 192.168.0.197 node01.k8s <none> <none>
default nginx-86bc984445-27f2p 1/1 Running 0 111s 192.168.0.167 node01.k8s <none> <none>
default nginx-86bc984445-2v2xd 1/1 Running 0 114s 192.168.0.118 control-plane02.k8s <none> <none>
default nginx-86bc984445-2vc2c 1/1 Running 0 108s 192.168.0.214 control-plane02.k8s <none> <none>
default nginx-86bc984445-4vkh2 1/1 Running 0 113s 192.168.0.138 control-plane03.k8s <none> <none>
default nginx-86bc984445-55b8v 1/1 Running 0 116s 192.168.0.25 node02.k8s <none> <none>
default nginx-86bc984445-55tt5 1/1 Running 0 110s 192.168.0.217 node02.k8s <none> <none>
default nginx-86bc984445-59k9t 1/1 Running 0 116s 192.168.0.28 control-plane02.k8s <none> <none>
default nginx-86bc984445-5brcl 1/1 Running 0 114s 192.168.0.96 node01.k8s <none> <none>
default nginx-86bc984445-5fbj4 1/1 Running 0 111s 192.168.0.178 node01.k8s <none> <none>
default nginx-86bc984445-5hfkl 1/1 Running 0 112s 192.168.0.233 node02.k8s <none> <none>
default nginx-86bc984445-5hm7r 1/1 Running 0 116s 192.168.0.32 node01.k8s <none> <none>
default nginx-86bc984445-5lv62 1/1 Running 0 113s 192.168.0.61 node02.k8s <none> <none>
default nginx-86bc984445-5n2dr 1/1 Running 0 110s 192.168.0.199 node02.k8s <none> <none>
default nginx-86bc984445-5plqt 1/1 Running 0 111s 192.168.0.198 node02.k8s <none> <none>
default nginx-86bc984445-5qkw7 1/1 Running 0 116s 192.168.0.49 node02.k8s <none> <none>
default nginx-86bc984445-5qt5j 1/1 Running 0 106s 192.168.0.165 control-plane02.k8s <none> <none>
default nginx-86bc984445-5rnpm 1/1 Running 0 114s 192.168.0.40 node01.k8s <none> <none>
default nginx-86bc984445-5w2xl 1/1 Running 0 115s 192.168.0.91 control-plane03.k8s <none> <none>
default nginx-86bc984445-5xbkp 1/1 Running 0 110s 192.168.0.109 control-plane02.k8s <none> <none>
default nginx-86bc984445-6224x 1/1 Running 0 113s 192.168.0.65 node02.k8s <none> <none>
default nginx-86bc984445-644sb 1/1 Running 0 116s 192.168.0.12 node02.k8s <none> <none>
default nginx-86bc984445-64r9w 1/1 Running 0 108s 192.168.0.223 node01.k8s <none> <none>
default nginx-86bc984445-6h485 1/1 Running 0 116s 192.168.0.35 control-plane03.k8s <none> <none>
default nginx-86bc984445-6hj8v 1/1 Running 0 108s 192.168.0.151 control-plane03.k8s <none> <none>
default nginx-86bc984445-6jnxc 1/1 Running 0 109s 192.168.0.190 control-plane02.k8s <none> <none>
default nginx-86bc984445-6ktcw 1/1 Running 0 109s 192.168.0.168 control-plane02.k8s <none> <none>
default nginx-86bc984445-6mdd2 1/1 Running 0 116s 192.168.0.20 node02.k8s <none> <none>
default nginx-86bc984445-6qd9m 1/1 Running 0 116s 192.168.0.54 node02.k8s <none> <none>
default nginx-86bc984445-6vctb 1/1 Running 0 111s 192.168.0.152 node01.k8s <none> <none>
default nginx-86bc984445-6zqwq 1/1 Running 0 113s 192.168.0.70 node01.k8s <none> <none>
default nginx-86bc984445-75vwp 1/1 Running 0 113s 192.168.0.127 control-plane02.k8s <none> <none>
default nginx-86bc984445-7ft9b 1/1 Running 0 110s 192.168.0.141 node01.k8s <none> <none>
default nginx-86bc984445-7gc58 1/1 Running 0 106s 192.168.0.78 node01.k8s <none> <none>
default nginx-86bc984445-7m2qz 1/1 Running 0 106s 192.168.0.216 control-plane02.k8s <none> <none>
default nginx-86bc984445-7r8hd 1/1 Running 0 108s 192.168.0.194 node01.k8s <none> <none>
default nginx-86bc984445-7tgds 1/1 Running 0 112s 192.168.0.201 control-plane03.k8s <none> <none>
default nginx-86bc984445-7v4kv 1/1 Running 0 108s 192.168.0.122 control-plane03.k8s <none> <none>
default nginx-86bc984445-7wsld 1/1 Running 0 114s 192.168.0.60 node02.k8s <none> <none>
default nginx-86bc984445-7zq74 1/1 Running 0 112s 192.168.0.232 node02.k8s <none> <none>
default nginx-86bc984445-84gp5 1/1 Running 0 107s 192.168.0.239 node02.k8s <none> <none>
default nginx-86bc984445-85ndt 1/1 Running 0 116s 192.168.0.26 control-plane02.k8s <none> <none>
default nginx-86bc984445-896d6 1/1 Running 0 114s 192.168.0.215 node01.k8s <none> <none>
default nginx-86bc984445-89mfl 1/1 Running 0 111s 192.168.0.171 node01.k8s <none> <none>
default nginx-86bc984445-8cbkp 1/1 Running 0 114s 192.168.0.185 control-plane03.k8s <none> <none>
default nginx-86bc984445-8ggqw 1/1 Running 0 107s 192.168.0.202 control-plane02.k8s <none> <none>
default nginx-86bc984445-8gr7c 1/1 Running 0 110s 192.168.0.159 control-plane03.k8s <none> <none>
default nginx-86bc984445-8nz8q 1/1 Running 0 109s 192.168.0.157 control-plane02.k8s <none> <none>
default nginx-86bc984445-8vpsq 1/1 Running 0 116s 192.168.0.37 node02.k8s <none> <none>
default nginx-86bc984445-8wdcc 1/1 Running 0 115s 192.168.0.74 control-plane03.k8s <none> <none>
default nginx-86bc984445-8x2bw 1/1 Running 0 112s 192.168.0.230 control-plane03.k8s <none> <none>
default nginx-86bc984445-8xcqn 1/1 Running 0 114s 192.168.0.55 node02.k8s <none> <none>
default nginx-86bc984445-8zbk9 1/1 Running 0 107s 192.168.0.173 control-plane02.k8s <none> <none>
default nginx-86bc984445-92mqq 1/1 Running 0 114s 192.168.0.82 control-plane02.k8s <none> <none>
default nginx-86bc984445-94hjt 1/1 Running 0 106s 192.168.0.228 control-plane02.k8s <none> <none>
default nginx-86bc984445-97dkq 1/1 Running 0 116s 192.168.0.13 node01.k8s <none> <none>
default nginx-86bc984445-98bp8 1/1 Running 0 114s 192.168.0.231 node02.k8s <none> <none>
default nginx-86bc984445-9df49 1/1 Running 0 116s 192.168.0.42 control-plane03.k8s <none> <none>
default nginx-86bc984445-9jj4g 1/1 Running 0 114s 192.168.0.112 control-plane03.k8s <none> <none>
default nginx-86bc984445-9td2f 1/1 Running 0 107s 192.168.0.191 node02.k8s <none> <none>
default nginx-86bc984445-9wnrv 1/1 Running 0 115s 192.168.0.98 node01.k8s <none> <none>
default nginx-86bc984445-9x69b 1/1 Running 0 110s 192.168.0.136 control-plane03.k8s <none> <none>
default nginx-86bc984445-b2gfb 1/1 Running 0 112s 192.168.0.76 node01.k8s <none> <none>
default nginx-86bc984445-b6czk 1/1 Running 0 111s 192.168.0.176 node01.k8s <none> <none>
default nginx-86bc984445-b7z82 1/1 Running 0 113s 192.168.0.161 control-plane03.k8s <none> <none>
default nginx-86bc984445-bcs87 1/1 Running 0 111s 192.168.0.93 control-plane03.k8s <none> <none>
default nginx-86bc984445-bdbl8 1/1 Running 0 106s 192.168.0.128 node01.k8s <none> <none>
default nginx-86bc984445-blbdp 1/1 Running 0 110s 192.168.0.209 control-plane02.k8s <none> <none>
default nginx-86bc984445-bqx98 1/1 Running 0 109s 192.168.0.108 control-plane03.k8s <none> <none>
default nginx-86bc984445-c5kt5 1/1 Running 0 107s 192.168.0.123 node01.k8s <none> <none>
default nginx-86bc984445-c69hl 1/1 Running 0 113s 192.168.0.58 node01.k8s <none> <none>
default nginx-86bc984445-c6qjw 1/1 Running 0 110s 192.168.0.222 node01.k8s <none> <none>
default nginx-86bc984445-c985j 1/1 Running 0 116s 192.168.0.9 node02.k8s <none> <none>
default nginx-86bc984445-ccfxc 1/1 Running 0 115s 192.168.0.72 control-plane02.k8s <none> <none>
default nginx-86bc984445-cdh2z 1/1 Running 0 106s 192.168.0.75 node01.k8s <none> <none>
default nginx-86bc984445-cfxd7 1/1 Running 0 109s 192.168.0.86 node01.k8s <none> <none>
default nginx-86bc984445-cg4p9 1/1 Running 0 108s 192.168.0.119 node02.k8s <none> <none>
default nginx-86bc984445-cmx67 1/1 Running 0 106s 192.168.0.207 control-plane02.k8s <none> <none>
default nginx-86bc984445-cpvch 1/1 Running 0 113s 192.168.0.180 node01.k8s <none> <none>
default nginx-86bc984445-d5qbb 1/1 Running 0 116s 192.168.0.36 control-plane02.k8s <none> <none>
default nginx-86bc984445-d8c6q 1/1 Running 0 107s 192.168.0.69 node02.k8s <none> <none>
default nginx-86bc984445-dcl5j 1/1 Running 0 112s 192.168.0.213 control-plane02.k8s <none> <none>
default nginx-86bc984445-dgrpx 1/1 Running 0 110s 192.168.0.140 control-plane03.k8s <none> <none>
default nginx-86bc984445-dhnlt 1/1 Running 0 116s 192.168.0.45 control-plane02.k8s <none> <none>
default nginx-86bc984445-dk9dt 1/1 Running 0 107s 192.168.0.250 control-plane03.k8s <none> <none>
default nginx-86bc984445-dmmcr 1/1 Running 0 108s 192.168.0.115 node02.k8s <none> <none>
default nginx-86bc984445-dxhsw 1/1 Running 0 111s 192.168.0.241 control-plane02.k8s <none> <none>
default nginx-86bc984445-f2qrv 1/1 Running 0 105s 192.168.0.144 control-plane02.k8s <none> <none>
default nginx-86bc984445-f2w67 1/1 Running 0 116s 192.168.0.34 node02.k8s <none> <none>
default nginx-86bc984445-f49pd 1/1 Running 0 107s 192.168.0.184 node02.k8s <none> <none>
default nginx-86bc984445-f59gw 1/1 Running 0 116s 192.168.0.51 node01.k8s <none> <none>
default nginx-86bc984445-ft95r 1/1 Running 0 116s 192.168.0.50 node01.k8s <none> <none>
default nginx-86bc984445-fwmw8 1/1 Running 0 113s 192.168.0.160 control-plane03.k8s <none> <none>
default nginx-86bc984445-g2qp4 1/1 Running 0 107s 192.168.0.175 node01.k8s <none> <none>
default nginx-86bc984445-g46xc 1/1 Running 0 106s 192.168.0.67 node02.k8s <none> <none>
default nginx-86bc984445-g57r2 1/1 Running 0 115s 192.168.0.102 control-plane02.k8s <none> <none>
default nginx-86bc984445-g9gxv 1/1 Running 0 113s 192.168.0.59 node01.k8s <none> <none>
default nginx-86bc984445-gcq7w 1/1 Running 0 109s 192.168.0.225 node01.k8s <none> <none>
default nginx-86bc984445-gg4wl 1/1 Running 0 106s 192.168.0.120 control-plane03.k8s <none> <none>
default nginx-86bc984445-ggs98 1/1 Running 0 112s 192.168.0.255 control-plane03.k8s <none> <none>
default nginx-86bc984445-gpmrx 1/1 Running 0 106s 192.168.0.245 control-plane03.k8s <none> <none>
default nginx-86bc984445-gs7nx 1/1 Running 0 106s 192.168.0.224 control-plane02.k8s <none> <none>
default nginx-86bc984445-gsq92 1/1 Running 0 112s 192.168.0.129 control-plane02.k8s <none> <none>
default nginx-86bc984445-gvh55 1/1 Running 0 116s 192.168.0.48 node01.k8s <none> <none>
default nginx-86bc984445-gwc4b 1/1 Running 0 109s 192.168.0.153 node01.k8s <none> <none>
default nginx-86bc984445-gwlnb 1/1 Running 0 114s 192.168.0.177 control-plane02.k8s <none> <none>
default nginx-86bc984445-gwtvv 1/1 Running 0 108s 192.168.0.211 node01.k8s <none> <none>
default nginx-86bc984445-hd56c 1/1 Running 0 106s 192.168.0.179 node02.k8s <none> <none>
default nginx-86bc984445-hhsgv 1/1 Running 0 109s 192.168.0.235 control-plane03.k8s <none> <none>
default nginx-86bc984445-hlth5 1/1 Running 0 116s 192.168.0.19 control-plane03.k8s <none> <none>
default nginx-86bc984445-hmmgn 1/1 Running 0 112s 192.168.0.71 node01.k8s <none> <none>
default nginx-86bc984445-hr7s8 1/1 Running 0 109s 192.168.0.146 node02.k8s <none> <none>
default nginx-86bc984445-hrccq 1/1 Running 0 116s 192.168.0.15 control-plane03.k8s <none> <none>
default nginx-86bc984445-htv8t 1/1 Running 0 109s 192.168.0.210 node01.k8s <none> <none>
default nginx-86bc984445-hxc5z 1/1 Running 0 115s 192.168.0.47 control-plane02.k8s <none> <none>
default nginx-86bc984445-hxh4c 1/1 Running 0 107s 192.168.0.226 node01.k8s <none> <none>
default nginx-86bc984445-j7zv7 1/1 Running 0 109s 192.168.0.94 node02.k8s <none> <none>
default nginx-86bc984445-j877b 1/1 Running 0 115s 192.168.0.57 control-plane03.k8s <none> <none>
default nginx-86bc984445-jhsrs 1/1 Running 0 110s 192.168.0.248 control-plane03.k8s <none> <none>
default nginx-86bc984445-jjhrl 1/1 Running 0 111s 192.168.0.111 node02.k8s <none> <none>
default nginx-86bc984445-jl2fj 1/1 Running 0 109s 192.168.0.172 control-plane03.k8s <none> <none>
default nginx-86bc984445-jpbms 1/1 Running 0 116s 192.168.0.85 node01.k8s <none> <none>
default nginx-86bc984445-k58lv 1/1 Running 0 116s 192.168.0.90 node01.k8s <none> <none>
default nginx-86bc984445-kb6dl 1/1 Running 0 110s 192.168.0.195 node01.k8s <none> <none>
default nginx-86bc984445-kc2f8 1/1 Running 0 111s 192.168.0.154 control-plane02.k8s <none> <none>
default nginx-86bc984445-kdzk9 1/1 Running 0 116s 192.168.0.31 node02.k8s <none> <none>
default nginx-86bc984445-knl2s 1/1 Running 0 115s 192.168.0.83 node01.k8s <none> <none>
default nginx-86bc984445-kq4qq 1/1 Running 0 107s 192.168.0.206 control-plane02.k8s <none> <none>
default nginx-86bc984445-kt88h 1/1 Running 0 105s 192.168.0.254 control-plane03.k8s <none> <none>
default nginx-86bc984445-kx6gq 1/1 Running 0 116s 192.168.0.14 node02.k8s <none> <none>
default nginx-86bc984445-l5m7p 1/1 Running 0 111s 192.168.0.126 node02.k8s <none> <none>
default nginx-86bc984445-l5xvq 1/1 Running 0 108s 192.168.0.134 node01.k8s <none> <none>
default nginx-86bc984445-l676h 1/1 Running 0 114s 192.168.0.64 control-plane02.k8s <none> <none>
default nginx-86bc984445-lf4wb 1/1 Running 0 111s 192.168.0.238 control-plane02.k8s <none> <none>
default nginx-86bc984445-lgc4h 1/1 Running 0 107s 192.168.0.242 control-plane03.k8s <none> <none>
default nginx-86bc984445-lm8gt 1/1 Running 0 113s 192.168.0.212 control-plane03.k8s <none> <none>
default nginx-86bc984445-ltnbr 1/1 Running 0 116s 192.168.0.131 node01.k8s <none> <none>
default nginx-86bc984445-lxjtx 1/1 Running 0 113s 192.168.0.189 control-plane02.k8s <none> <none>
default nginx-86bc984445-lzj7p 1/1 Running 0 109s 192.168.0.169 control-plane03.k8s <none> <none>
default nginx-86bc984445-m4r9m 1/1 Running 0 116s 192.168.0.27 node02.k8s <none> <none>
default nginx-86bc984445-m6r5z 1/1 Running 0 110s 192.168.0.142 node02.k8s <none> <none>
default nginx-86bc984445-m84lg 1/1 Running 0 113s 192.168.0.95 node02.k8s <none> <none>
default nginx-86bc984445-m8f74 1/1 Running 0 116s 192.168.0.10 control-plane03.k8s <none> <none>
default nginx-86bc984445-mbhdz 1/1 Running 0 111s 192.168.0.116 node02.k8s <none> <none>
default nginx-86bc984445-mcrqv 1/1 Running 0 106s 192.168.0.188 control-plane03.k8s <none> <none>
default nginx-86bc984445-mfl9w 1/1 Running 0 116s 192.168.0.30 node01.k8s <none> <none>
default nginx-86bc984445-mn9w8 1/1 Running 0 115s 192.168.0.62 control-plane02.k8s <none> <none>
default nginx-86bc984445-mpf5t 1/1 Running 0 113s 192.168.0.113 control-plane02.k8s <none> <none>
default nginx-86bc984445-msn44 1/1 Running 0 108s 192.168.0.110 node02.k8s <none> <none>
default nginx-86bc984445-n26cp 1/1 Running 0 114s 192.168.0.124 node01.k8s <none> <none>
default nginx-86bc984445-n2wfj 1/1 Running 0 110s 192.168.0.182 node01.k8s <none> <none>
default nginx-86bc984445-n4r2s 1/1 Running 0 110s 192.168.0.164 control-plane02.k8s <none> <none>
default nginx-86bc984445-n5f79 1/1 Running 0 111s 192.168.0.205 node02.k8s <none> <none>
default nginx-86bc984445-n7dmp 1/1 Running 0 116s 192.168.0.38 control-plane02.k8s <none> <none>
default nginx-86bc984445-n85jh 1/1 Running 0 105s 192.168.0.192 control-plane02.k8s <none> <none>
default nginx-86bc984445-nbvdt 1/1 Running 0 109s 192.168.0.87 node02.k8s <none> <none>
default nginx-86bc984445-ncfnn 1/1 Running 0 116s 192.168.0.41 node01.k8s <none> <none>
default nginx-86bc984445-ndpxc 1/1 Running 0 115s 192.168.0.29 node02.k8s <none> <none>
default nginx-86bc984445-nfgpn 1/1 Running 0 111s 192.168.0.156 control-plane03.k8s <none> <none>
default nginx-86bc984445-nh9vx 1/1 Running 0 114s 192.168.0.174 control-plane03.k8s <none> <none>
default nginx-86bc984445-nn8pv 1/1 Running 0 116s 192.168.0.22 node02.k8s <none> <none>
default nginx-86bc984445-nt9vq 1/1 Running 0 106s 192.168.0.196 node02.k8s <none> <none>
default nginx-86bc984445-nwr4j 1/1 Running 0 112s 192.168.0.84 control-plane02.k8s <none> <none>
default nginx-86bc984445-p26wt 1/1 Running 0 108s 192.168.0.150 node02.k8s <none> <none>
default nginx-86bc984445-p4x44 1/1 Running 0 109s 192.168.0.163 control-plane03.k8s <none> <none>
default nginx-86bc984445-p9nx4 1/1 Running 0 111s 192.168.0.121 control-plane03.k8s <none> <none>
default nginx-86bc984445-pccs6 1/1 Running 0 112s 192.168.0.100 control-plane02.k8s <none> <none>
default nginx-86bc984445-phvh4 1/1 Running 0 115s 192.168.0.44 control-plane03.k8s <none> <none>
default nginx-86bc984445-phz7w 1/1 Running 0 115s 192.168.0.139 node01.k8s <none> <none>
default nginx-86bc984445-pj72f 1/1 Running 0 115s 192.168.0.79 node02.k8s <none> <none>
default nginx-86bc984445-pntwc 1/1 Running 0 115s 192.168.0.253 control-plane03.k8s <none> <none>
default nginx-86bc984445-ppgzq 1/1 Running 0 114s 192.168.0.203 node01.k8s <none> <none>
default nginx-86bc984445-pttzf 1/1 Running 0 110s 192.168.0.148 node02.k8s <none> <none>
default nginx-86bc984445-pv5zh 1/1 Running 0 107s 192.168.0.237 node02.k8s <none> <none>
default nginx-86bc984445-pw58c 1/1 Running 0 116s 192.168.0.21 control-plane03.k8s <none> <none>
default nginx-86bc984445-q5kt6 1/1 Running 0 106s 192.168.0.208 control-plane03.k8s <none> <none>
default nginx-86bc984445-qb986 1/1 Running 0 106s 192.168.0.221 node02.k8s <none> <none>
default nginx-86bc984445-qcbc2 1/1 Running 0 116s 192.168.0.53 node01.k8s <none> <none>
default nginx-86bc984445-qgbwz 1/1 Running 0 116s 192.168.0.17 control-plane02.k8s <none> <none>
default nginx-86bc984445-qn46p 1/1 Running 0 105s 192.168.0.81 node01.k8s <none> <none>
default nginx-86bc984445-qpkk6 1/1 Running 0 105s 192.168.0.219 control-plane03.k8s <none> <none>
default nginx-86bc984445-qq7cr 1/1 Running 0 110s 192.168.0.105 control-plane02.k8s <none> <none>
default nginx-86bc984445-r6856 1/1 Running 0 114s 192.168.0.187 control-plane02.k8s <none> <none>
default nginx-86bc984445-r8fn5 1/1 Running 0 111s 192.168.0.132 control-plane02.k8s <none> <none>
default nginx-86bc984445-rd7m4 1/1 Running 0 115s 192.168.0.46 control-plane03.k8s <none> <none>
default nginx-86bc984445-rk2zl 1/1 Running 0 108s 192.168.0.220 control-plane02.k8s <none> <none>
default nginx-86bc984445-rn57l 1/1 Running 0 116s 192.168.0.24 control-plane02.k8s <none> <none>
default nginx-86bc984445-rnq6s 1/1 Running 0 108s 192.168.0.137 control-plane02.k8s <none> <none>
default nginx-86bc984445-rpfwv 1/1 Running 0 108s 192.168.0.249 control-plane03.k8s <none> <none>
default nginx-86bc984445-rrqdf 1/1 Running 0 112s 192.168.0.227 control-plane03.k8s <none> <none>
default nginx-86bc984445-rxdxt 1/1 Running 0 105s 192.168.0.89 node02.k8s <none> <none>
default nginx-86bc984445-s5f24 1/1 Running 0 106s 192.168.0.147 control-plane03.k8s <none> <none>
default nginx-86bc984445-s7gxd 1/1 Running 0 112s 192.168.0.125 control-plane02.k8s <none> <none>
default nginx-86bc984445-sb2nh 1/1 Running 0 112s 192.168.0.236 control-plane02.k8s <none> <none>
default nginx-86bc984445-slvnn 1/1 Running 0 108s 192.168.0.162 control-plane02.k8s <none> <none>
default nginx-86bc984445-sm46j 1/1 Running 0 107s 192.168.0.240 control-plane03.k8s <none> <none>
default nginx-86bc984445-smj29 1/1 Running 0 113s 192.168.0.97 node02.k8s <none> <none>
default nginx-86bc984445-sngch 1/1 Running 0 113s 192.168.0.244 control-plane02.k8s <none> <none>
default nginx-86bc984445-sr7fh 1/1 Running 0 115s 192.168.0.52 control-plane02.k8s <none> <none>
default nginx-86bc984445-srkvk 1/1 Running 0 116s 192.168.0.33 control-plane03.k8s <none> <none>
default nginx-86bc984445-t2hlg 1/1 Running 0 110s 192.168.0.117 control-plane03.k8s <none> <none>
default nginx-86bc984445-t2vt2 1/1 Running 0 107s 192.168.0.114 control-plane03.k8s <none> <none>
default nginx-86bc984445-t4jgf 1/1 Running 0 106s 192.168.0.145 node01.k8s <none> <none>
default nginx-86bc984445-tbtjm 1/1 Running 0 107s 192.168.0.200 control-plane02.k8s <none> <none>
default nginx-86bc984445-td26n 1/1 Running 0 116s 192.168.0.11 node02.k8s <none> <none>
default nginx-86bc984445-tdhdt 1/1 Running 0 116s 192.168.0.23 control-plane03.k8s <none> <none>
default nginx-86bc984445-tfmjz 1/1 Running 0 109s 192.168.0.246 control-plane02.k8s <none> <none>
default nginx-86bc984445-tggfw 1/1 Running 0 106s 192.168.0.155 node02.k8s <none> <none>
default nginx-86bc984445-tgq5c 1/1 Running 0 107s 192.168.0.204 control-plane02.k8s <none> <none>
default nginx-86bc984445-tjbnk 1/1 Running 0 108s 192.168.0.193 control-plane03.k8s <none> <none>
default nginx-86bc984445-txd5g 1/1 Running 0 113s 192.168.0.104 node02.k8s <none> <none>
default nginx-86bc984445-v8gzl 1/1 Running 0 108s 192.168.0.181 control-plane03.k8s <none> <none>
default nginx-86bc984445-v94qg 1/1 Running 0 109s 192.168.0.99 node02.k8s <none> <none>
default nginx-86bc984445-v9npm 1/1 Running 0 114s 192.168.0.39 node02.k8s <none> <none>
default nginx-86bc984445-vb598 1/1 Running 0 114s 192.168.0.56 node02.k8s <none> <none>
default nginx-86bc984445-vjdz9 1/1 Running 0 114s 192.168.0.251 control-plane03.k8s <none> <none>
default nginx-86bc984445-vkkmx 1/1 Running 0 111s 192.168.0.106 control-plane03.k8s <none> <none>
default nginx-86bc984445-vmbxr 1/1 Running 0 115s 192.168.0.43 control-plane02.k8s <none> <none>
default nginx-86bc984445-vmzqs 1/1 Running 0 112s 192.168.0.92 node02.k8s <none> <none>
default nginx-86bc984445-vp9qc 1/1 Running 0 112s 192.168.0.183 control-plane03.k8s <none> <none>
default nginx-86bc984445-vvlw9 1/1 Running 0 116s 192.168.0.18 control-plane02.k8s <none> <none>
default nginx-86bc984445-vvtd9 1/1 Running 0 115s 192.168.0.77 control-plane02.k8s <none> <none>
default nginx-86bc984445-vz58m 1/1 Running 0 105s 192.168.0.130 node01.k8s <none> <none>
default nginx-86bc984445-w2ksl 1/1 Running 0 113s 192.168.0.234 control-plane03.k8s <none> <none>
default nginx-86bc984445-w4shb 1/1 Running 0 116s 192.168.0.63 node01.k8s <none> <none>
default nginx-86bc984445-wdvcr 1/1 Running 0 108s 192.168.0.135 node02.k8s <none> <none>
default nginx-86bc984445-wjjck 1/1 Running 0 111s 192.168.0.103 control-plane03.k8s <none> <none>
default nginx-86bc984445-wqwzl 1/1 Running 0 116s 192.168.0.16 node01.k8s <none> <none>
default nginx-86bc984445-wvgzt 1/1 Running 0 108s 192.168.0.166 node01.k8s <none> <none>
default nginx-86bc984445-wvnh7 1/1 Running 0 107s 192.168.0.143 node01.k8s <none> <none>
default nginx-86bc984445-ww98k 1/1 Running 0 112s 192.168.0.80 node02.k8s <none> <none>
default nginx-86bc984445-wzxx8 1/1 Running 0 112s 192.168.0.73 node01.k8s <none> <none>
default nginx-86bc984445-x7hxt 1/1 Running 0 110s 192.168.0.247 control-plane02.k8s <none> <none>
default nginx-86bc984445-x8mxf 1/1 Running 0 108s 192.168.0.170 control-plane02.k8s <none> <none>
default nginx-86bc984445-xc5wl 1/1 Running 0 113s 192.168.0.107 node02.k8s <none> <none>
default nginx-86bc984445-xn72t 1/1 Running 0 110s 192.168.0.186 node02.k8s <none> <none>
default nginx-86bc984445-xpt5q 1/1 Running 0 107s 192.168.0.252 control-plane03.k8s <none> <none>
default nginx-86bc984445-xqkwx 1/1 Running 0 113s 192.168.0.243 control-plane02.k8s <none> <none>
default nginx-86bc984445-z4t5n 1/1 Running 0 109s 192.168.0.158 node02.k8s <none> <none>
default nginx-86bc984445-zb62g 1/1 Running 0 112s 192.168.0.66 node01.k8s <none> <none>
default nginx-86bc984445-zc4db 1/1 Running 0 109s 192.168.0.149 control-plane02.k8s <none> <none>
default nginx-86bc984445-zcgzx 1/1 Running 0 112s 192.168.0.68 node01.k8s <none> <none>
default nginx-86bc984445-zgd8b 1/1 Running 0 115s 192.168.0.88 control-plane03.k8s <none> <none>
default nginx-86bc984445-zlblx 1/1 Running 0 106s 192.168.0.218 node01.k8s <none> <none>
default nginx-86bc984445-ztzp6 1/1 Running 0 109s 192.168.0.133 node01.k8s <none> <none>
default nginx-86bc984445-zvjpx 1/1 Running 0 115s 192.168.0.101 control-plane03.k8s <none> <none>
default nginx-86bc984445-zwq4j 1/1 Running 0 110s 192.168.0.229 node01.k8s <none> <none>
kube-system coredns-8474476ff8-qj8qk 1/1 Running 0 19m 192.168.0.5 control-plane02.k8s <none> <none>
kube-system coredns-8474476ff8-tsvtv 1/1 Running 0 19m 192.168.0.7 control-plane01.k8s <none> <none>
kube-system dns-autoscaler-5ffdc7f89d-szmdb 1/1 Running 0 19m 192.168.0.6 control-plane03.k8s <none> <none>
kube-system metrics-server-6978dd689f-kd5vx 1/1 Running 0 18m 192.168.0.8 control-plane03.k8s <none> <none>
$ kubectl get pods -o wide -A | grep -i running | grep -v 172.31 | awk '{if(NR>1)print $7}' | sort | uniq -d
중복 없음- IPPool 확인 후 새로운 POD 1개 생성
$ calicoctl ipam show --show-blocks
+----------+----------------+-----------+------------+----------+
| GROUPING | CIDR | IPS TOTAL | IPS IN USE | IPS FREE |
+----------+----------------+-----------+------------+----------+
| IP Pool | 192.168.0.0/24 | 256 | 256 (100%) | 0 (0%) |
| Block | 192.168.0.0/24 | 256 | 256 (100%) | 0 (0%) |
+----------+----------------+-----------+------------+----------+
$ cat pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: seob
spec:
containers:
- name: seob
image: nginx:1.14.2
ports:
- containerPort: 80
$ kubectl apply -f pod.yaml
pod/seob created
$ kubectl get pod | grep seob
seob 0/1 ContainerCreating 0 6s
$ kubectl describe pod/seob
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreatePodSandBox 11s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "85455fc3d6045d2f72c154ba5f0017539a5c554de375f044879bed80d271f092": failed to request IPv4 addresses: Assigned 0 out of 1 requested IPv4 addresses; No IPs available in pools: [192.168.0.0/24]
결론
- CIDRNotAvailable 이벤트 메시지가 발생해도 CIDR을 할당받지 못한 노드에서도 IPPool 에 사용 가능한 IP가 있다면 할당 받음
- 최초 구성 시 Kubernetes 규모 예측하여 네트워크 계산을 통해 적절한 값으로 설정
- kubespray 에서 기본적으로 kubelet_max_pods 값은 24 이며 kubelet_max_pods 은 110 으로 기본값을 가지는데 단일 노드에서 최대 POD 110개의 실행 할 수 있다는 의미. 따라서 kubelet_max_pods 을 25개 설정하여 효율적인 IP 관리 가능
- 운영 환경 및 테스트 환경에서는 IP 중복이 발생하지 않았지만 가능하면 문제를 조치
- Subnet 을 더 큰 대역으로 변경하는 방식으로 조치 가능
반응형
댓글
공지사항
최근에 올라온 글
