【作者】陳成，中國聯(lián)通軟件研究院容器云研發(fā)工程師，公共平臺(tái)與架構(gòu)研發(fā)事業(yè)部云計(jì)算研發(fā)組長，長期從事大規(guī)模基礎(chǔ)平臺(tái)建設(shè)相關(guān)工作，先后從事Mesos、KVM、K8S等研究，專注于容器云計(jì)算框架、集群調(diào)度、虛擬化等。

故事的開始，讓我們先從一件生產(chǎn)故障說起。5月29日，內(nèi)部某系統(tǒng)出現(xiàn)大規(guī)模訪問Service故障，發(fā)現(xiàn)Pod容器內(nèi)無法正常訪問ServiceIP:Port，整個(gè)故障持續(xù)時(shí)間超過12h，相關(guān)運(yùn)維支撐人員沒有找到根本原因和解決辦法。

經(jīng)過復(fù)盤，我們發(fā)現(xiàn)，大家對于K8S Service的原理不夠清晰，導(dǎo)致對問題的定位不能做得到快速準(zhǔn)確，如果當(dāng)時(shí)能夠按照如下的思路去思考問題，排查過程不至于花費(fèi)如此久的時(shí)間。

下面，我們就來細(xì)說一下Service在Kubernetes中的作用、使用方法及原理。

Service是一種暴露一組Pod網(wǎng)絡(luò)的抽象方式，K8S Service提供了針對于一組Pod的負(fù)載均衡的暴露。通過這樣的方式，可以避免不同的pod之間訪問時(shí)需要知曉對應(yīng)pod網(wǎng)絡(luò)信息的痛苦。例如：前端->后端，由于前端POD IP隨時(shí)變動(dòng)，后端亦如此，如何處理前端POD和后端POD的通信，就需要Service這一抽象，來保證簡單可靠。

Service的使用

1、典型服務(wù)配置方法

當(dāng)配置了selector之后，Service Controller會(huì)自動(dòng)查找匹配這個(gè)selector的pod，并且創(chuàng)建出一個(gè)同名的endpoint對象，負(fù)責(zé)具體service之后連接。

apiVersion: v1kind: Servicemetadata:  name: my-servicespec:  selector:    app: MyApp  ports:    - protocol: TCP      port: 80      targetPort: 9376

2、配置沒有selector的服務(wù)

沒有selector的service不會(huì)出現(xiàn)Endpoint的信息，需要手工創(chuàng)建Endpoint綁定，Endpoint可以是內(nèi)部的pod，也可以是外部的服務(wù)。

apiVersion: v1kind: Servicemetadata:  name: my-servicespec:  ports:    - protocol: TCP      port: 80      targetPort: 9376---apiVersion: v1kind: Endpointsmetadata:  name: my-servicesubsets:  - addresses:      - ip: 192.0.2.42    ports:      - port: 9376

Service的類型

1.CluserIP

kubectl expose pod nginx --type=CluserIP --port=80 --name=ng-svc apiVersion: v1kind: Servicemetadata:  name: ng-svc  namespace: defaultspec:  selector:      name: nginx  clusterIP: 11.254.0.2  ports:  - name: http    port: 80    protocol: TCP    targetPort: 1234  sessionAffinity: None  type: ClusterIP

2.LoadBalance

apiVersion: v1kind: Servicemetadata:  name: my-servicespec:  selector:    app: MyApp  ports:  - protocol: TCP    port: 80    targetPort: 9376  clusterIP: 10.0.171.239  type: LoadBalancerstatus:  loadBalancer:    ingress:    - ip: 192.0.2.127

apiVersion: v1kind: Servicemetadata:name: my-servicespec:type: NodePortselector:  app: MyAppports:  - port: 80    targetPort: 80    nodePort: 30007

apiVersion: v1kind: Servicemetadata:  labels:    run: curl  name: my-headless-service  namespace: defaultspec:  clusterIP: None  ports:  - port: 80    protocol: TCP    targetPort: 80  selector:    run: curl  type: ClusterIP

# ping my-headless-servicePING my-headless-service (172.200.6.207): 56 data bytes64 bytes from 172.200.6.207: seq=0 ttl=64 time=0.040 ms64bytesfrom172.200.6.207:seq=1ttl=64time=0.063ms

對沒有定義選擇算符的無頭服務(wù)，Endpoint 控制器不會(huì)創(chuàng)建 Endpoints 記錄。然而 DNS 系統(tǒng)會(huì)查找和配置，無論是：

• 對于 ExternalName 類型的服務(wù)，查找其 CNAME 記錄

• 對所有其他類型的服務(wù)，查找與 Service 名稱相同的任何 Endpoints 的記錄

Service的實(shí)現(xiàn)方式

1.用戶態(tài)代理訪問

即：當(dāng)對于每個(gè)Service，Kube-Proxy會(huì)在本地Node上打開一個(gè)隨機(jī)選擇的端口，連接到代理端口的請求，都會(huì)被代理轉(zhuǎn)發(fā)給Pod。那么通過Iptables規(guī)則，捕獲到達(dá)Service:Port的請求都會(huì)被轉(zhuǎn)發(fā)到代理端口，代理端口重新轉(zhuǎn)為對Pod的訪問

這種方式的缺點(diǎn)是存在內(nèi)核態(tài)轉(zhuǎn)為用戶態(tài)，再有用戶態(tài)轉(zhuǎn)發(fā)的兩次轉(zhuǎn)換，性能較差，一般不再使用

2.Iptables模式

3.Ipvs模式

Service Iptables實(shí)現(xiàn)原理

Iptables表和鏈及處理過程

Service的Traffic流量將會(huì)通過prerouting和output重定向到kube-service鏈

-APREROUTING-mcomment--comment"kubernetesserviceportals"-jKUBE-SERVICES-APOSTROUTING-mcomment--comment"kubernetespostroutingrules"-jKUBE-POSTROUTING-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES

• KUBE-SERVICES->KUBE-SVC-XXXXXXXXXXXXXXXX->KUBE-SEP-XXXXXXXXXXXXXXXX represents a ClusterIP service

• KUBE-NODEPORTS->KUBE-SVC-XXXXXXXXXXXXXXXX->KUBE-SEP-XXXXXXXXXXXXXXXX represents a NodePort service

幾種不同類型的Service在Kube-Proxy啟用Iptables模式下上的表現(xiàn)

• ClusterIP

-A KUBE-SERVICES ! -s 172.200.0.0/16 -d 10.100.160.92/32 -p tcp -m comment --comment "default/ccs-gateway-clusterip:http cluster IP" -m tcp --dport 30080 -j KUBE-MARK-MASQ-A KUBE-SERVICES -d 10.100.160.92/32 -p tcp -m comment --comment "default/ccs-gateway-clusterip:http cluster IP" -m tcp --dport 30080 -j KUBE-SVC-76GERFBRR2RGHNBJ  

-A KUBE-SVC-76GERFBRR2RGHNBJ -m comment --comment "default/ccs-gateway-clusterip:http" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-GBVECAZBIC3ZKMXB-A KUBE-SVC-76GERFBRR2RGHNBJ -m comment --comment "default/ccs-gateway-clusterip:http" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-PVCYYXEU44D3IMGK-A KUBE-SVC-76GERFBRR2RGHNBJ -m comment --comment "default/ccs-gateway-clusterip:http" -j KUBE-SEP-JECGZLHE32MEARRX-AKUBE-SVC-CEZPIJSAUFW5MYPQ-mcomment--comment"kubernetes-dashboard/kubernetes-dashboard"-jKUBE-SEP-QO6MV4HR5U56RP7M

-A KUBE-SEP-GBVECAZBIC3ZKMXB -s 172.200.6.224/32 -m comment --comment "default/ccs-gateway-clusterip:http" -j KUBE-MARK-MASQ-AKUBE-SEP-GBVECAZBIC3ZKMXB-ptcp-mcomment--comment"default/ccs-gateway-clusterip:http"-mtcp-jDNAT--to-destination172.200.6.224:80...

• NodePort

apiVersion: v1kind: Servicemetadata:labels:  app: ccs-gatewayspec:clusterIP: 10.101.156.39externalTrafficPolicy: Clusterports:- name: http  nodePort: 30081  port: 30080  protocol: TCP  targetPort: 80selector:  app: ccs-gatewaysessionAffinity: Nonetype: NodePort

-AKUBE-NODEPORTS-ptcp-mcomment--comment"default/ccs-gateway-service:http"-mtcp--dport30081-jKUBE-MARK-MASQ-AKUBE-NODEPORTS-ptcp-mcomment--comment"default/ccs-gateway-service:http"-mtcp--dport30081-jKUBE-SVC-QYHRFFHL5VINYT2K############################-AKUBE-SVC-QYHRFFHL5VINYT2K-mcomment--comment"default/ccs-gateway-service:http"-mstatistic--moderandom--probability0.50000000000-jKUBE-SEP-2NPKETIWKKVUXGCL-AKUBE-SVC-QYHRFFHL5VINYT2K-mcomment--comment"default/ccs-gateway-service:http"-jKUBE-SEP-6O5FHQRN5IVNPW4Q##########################-AKUBE-SEP-2NPKETIWKKVUXGCL-s172.200.6.224/32-mcomment--comment"default/ccs-gateway-service:http"-jKUBE-MARK-MASQ-AKUBE-SEP-2NPKETIWKKVUXGCL-ptcp-mcomment--comment"default/ccs-gateway-service:http"-mtcp-jDNAT--to-destination172.200.6.224:80#########################-AKUBE-SEP-6O5FHQRN5IVNPW4Q-s172.200.6.225/32-mcomment--comment"default/ccs-gateway-service:http"-jKUBE-MARK-MASQ-A KUBE-SEP-6O5FHQRN5IVNPW4Q -p tcp -m comment --comment "default/ccs-gateway-service:http" -m tcp -j DNAT --to-destination 172.200.6.225:80

同時(shí)，可以看到Service所申請的端口38081被Kube-proxy所代理和監(jiān)聽

# netstat -ntlp | grep 30081tcp       0      00.0.0.0:30081           0.0.0.0:*               LISTEN     3665705/kube-proxy

• LoadBalancer

不帶有Endpoint的Service

kubectl create svc clusterip fake-endpoint --tcp=80 -A KUBE-SERVICES -d 10.101.117.0/32 -p tcp -m comment --comment "default/fake-endpoint:80 has no endpoints" -m tcp --dport 80 -j REJECT --reject-with icmp-port-unreachable

帶有外部endpoint的Service

直接通過iptable規(guī)則轉(zhuǎn)發(fā)到對應(yīng)的外部ep地址

apiVersion: v1kind: Servicemetadata:  labels:    app: external  name: external  namespace: defaultspec:  ports:  - name: http    protocol: TCP    port: 80  sessionAffinity: None  type: ClusterIP---apiVersion: v1kind: Endpointsmetadata:  labels:    app: external  name: external  namespace: defaultsubsets:- addresses:  - ip: 10.124.142.43  ports:  - name: http    port: 80protocol: TCP

-A KUBE-SERVICES ! -s 172.200.0.0/16 -d 10.111.246.87/32 -p tcp -m comment --comment "default/external:http cluster IP" -m tcp --dport 80 -j KUBE-MARK-MASQ-A KUBE-SERVICES -d 10.111.246.87/32 -p tcp -m comment --comment "default/external:http cluster IP" -m tcp --dport 80 -j KUBE-SVC-LI2K5327B6J24KJ3 

-A KUBE-SEP-QTGIPNOYXN2CZGD5 -s 10.124.142.43/32 -m comment --comment "default/external:http" -j KUBE-MARK-MASQ-A KUBE-SEP-QTGIPNOYXN2CZGD5 -p tcp -m comment --comment "default/external:http" -m tcp -j DNAT --to-destination 10.124.142.43:80

總結(jié)

• ClusterIP類型，KubeProxy監(jiān)聽Service和Endpoint創(chuàng)建規(guī)則，采用DNAT將目標(biāo)地址轉(zhuǎn)換為Pod的ip和端口，當(dāng)有多個(gè)ep時(shí)，按照策略進(jìn)行轉(zhuǎn)發(fā)，默認(rèn)RR模式時(shí)，iptables采用：比如有4個(gè)實(shí)例，四條規(guī)則的概率分別為0.25, 0.33, 0.5和 1，按照順序，一次匹配完成整個(gè)流量的分配。

• NodePort類型，將會(huì)在上述ClusterIP模式之后，再加上Kube-Proxy的監(jiān)聽（為了確保其他服務(wù)不會(huì)占用該端口）和KUBE-NODEPORT的iptable規(guī)則

審核編輯：郭婷