从零开始学习docker（二十）RoutingMesh--Ingress负载均衡
生活随笔
收集整理的這篇文章主要介紹了
从零开始学习docker（二十）RoutingMesh--Ingress负载均衡
小編覺得挺不錯的,現在分享給大家,幫大家做個參考.
我們知道容器之間的通信，例如上圖中的10.0.9.3與10.0.9.5通信是通過overlay網絡，是通過一個VXLAN tannel來實現的。
但是service和service之間通信是通過VIP實現的。例如client的service與web 的service進行通信，而web有一個scale，因此client訪問web是通過訪問虛擬IP（VIP）來實現的。那么VIP是怎么映射到具體的10.0.9.5或者10.0.9.6呢？這是通過LVS實現的。
什么是LVS？
LVS，Linux Virtual Server。可以實現在系統級別的負載均衡。
我們可以在三個節點上任何一個節點訪問80端口都可以訪問到wordpress，這個實現就是IngressNetWork的作用。任何一臺swarm節點上去訪問端口服務時，會通過端口服務通過本節點IPVS（IP Virtual Service），并通過LVS給loadbanlance到真正具有service上面，例如上圖中我們通過訪問Docker Host3?
轉發到另外兩個節點中。
我們的實驗環境跟上一節一樣，我們將whoami 的scale 變為2
iie4bu@swarm-manager:~$ docker service ls ID NAME MODE REPLICAS IMAGE PORTS h4wlczp85sw5 client replicated 1/1 busybox:1.28.3 9i6wz6cg4koc whoami replicated 3/3 jwilder/whoami:latest *:8000->8000/tcp iie4bu@swarm-manager:~$ docker service scale whoami=2 whoami scaled to 2 overall progress: 2 out of 2 tasks 1/2: running [==================================================>] 2/2: running [==================================================>] verify: Service converged查看whoami的運行情況：
iie4bu@swarm-manager:~$ docker service ps whoami ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS 6hhuf528spdw whoami.1 jwilder/whoami:latest swarm-manager Running Running 17 hours ago 9idgk9jbrlcm whoami.3 jwilder/whoami:latest swarm-worker2 Running Running 16 hours ago分別運行在swarm-manager和swarm-worker2節點上。
在swarm-manager中訪問whoami：
iie4bu@swarm-manager:~$ curl 127.0.0.1:8000 I'm cc9f97cc5056 iie4bu@swarm-manager:~$ curl 127.0.0.1:8000 I'm f47e05019fd9在swarm-worker1中訪問whoami：
iie4bu@swarm-worker1:~$ curl 127.0.0.1:8000 I'm f47e05019fd9 iie4bu@swarm-worker1:~$ curl 127.0.0.1:8000 I'm cc9f97cc5056為什么對于swarm-worker1來講本地并沒有whoami的service，確能訪問8000端口？
通過iptables可以看到本地的轉發規則：
iie4bu@swarm-worker1:~$ sudo iptables -nL -t nat [sudo] password for iie4bu: Chain PREROUTING (policy ACCEPT) target prot opt source destination DOCKER-INGRESS all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL DOCKER all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCALChain INPUT (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination DOCKER-INGRESS all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL DOCKER all -- 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCALChain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match src-type LOCAL MASQUERADE all -- 172.17.0.0/16 0.0.0.0/0 MASQUERADE all -- 172.19.0.0/16 0.0.0.0/0 MASQUERADE all -- 172.18.0.0/16 0.0.0.0/0 MASQUERADE tcp -- 172.17.0.2 172.17.0.2 tcp dpt:443 MASQUERADE tcp -- 172.17.0.2 172.17.0.2 tcp dpt:80 MASQUERADE tcp -- 172.17.0.2 172.17.0.2 tcp dpt:22Chain DOCKER (2 references) target prot opt source destination RETURN all -- 0.0.0.0/0 0.0.0.0/0 RETURN all -- 0.0.0.0/0 0.0.0.0/0 RETURN all -- 0.0.0.0/0 0.0.0.0/0 DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 to:172.17.0.2:443 DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:81 to:172.17.0.2:80 DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:23 to:172.17.0.2:22Chain DOCKER-INGRESS (2 references) target prot opt source destination DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8000 to:172.19.0.2:8000 RETURN all -- 0.0.0.0/0 0.0.0.0/0我們看到DOCKER-INGRESS，它的轉發規則是如果我們訪問tcp8000duank端口，它會轉發到172.19.0.2:8000上，那么這個172.19.0.2:8000是什么呢？
我們先看一下本地的ip:
iie4bu@swarm-worker1:~$ ifconfig br-3f2fc691f5da Link encap:Ethernet HWaddr 02:42:c8:f4:03:ad inet addr:172.18.0.1 Bcast:172.18.255.255 Mask:255.255.0.0UP BROADCAST MULTICAST MTU:1500 Metric:1RX packets:0 errors:0 dropped:0 overruns:0 frame:0TX packets:0 errors:0 dropped:0 overruns:0 carrier:0collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)docker0 Link encap:Ethernet HWaddr 02:42:43:69:b7:60 inet addr:172.17.0.1 Bcast:172.17.255.255 Mask:255.255.0.0inet6 addr: fe80::42:43ff:fe69:b760/64 Scope:LinkUP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1RX packets:0 errors:0 dropped:0 overruns:0 frame:0TX packets:83 errors:0 dropped:0 overruns:0 carrier:0collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:9208 (9.2 KB)docker_gwbridge Link encap:Ethernet HWaddr 02:42:41:bf:4d:15 inet addr:172.19.0.1 Bcast:172.19.255.255 Mask:255.255.0.0inet6 addr: fe80::42:41ff:febf:4d15/64 Scope:LinkUP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1RX packets:42 errors:0 dropped:0 overruns:0 frame:0TX packets:142 errors:0 dropped:0 overruns:0 carrier:0collisions:0 txqueuelen:0 RX bytes:3556 (3.5 KB) TX bytes:13857 (13.8 KB) ...... ......我們可以看到本地有個網絡是docker_gwbridge，它的ip是172.19.0.1，這個地址與172.19.0.2在同一個網段。所以說我們可以猜測172.19.0.2肯定是與docker_gwbridge相連的一個網絡。可以通過brctl show 查看：
iie4bu@swarm-worker1:~$ brctl show bridge name bridge id STP enabled interfaces br-3f2fc691f5da 8000.0242c8f403ad no docker0 8000.02424369b760 no veth75a496d docker_gwbridge 8000.024241bf4d15 no veth400f4b4veth44af5f8可以看到docker_gwbridge有兩個interface，這兩個interface哪個是呢？
通過docker network ls：
iie4bu@swarm-worker1:~$ docker network ls NETWORK ID NAME DRIVER SCOPE bdf23298113d bridge bridge local 969e60257ba5 docker_gwbridge bridge local cdcffe1b31cb host host local uz1kgf9j6m48 ingress overlay swarm cojus8blvkdo my-demo overlay swarm 3f2fc691f5da network_default bridge local dba4587ee914 none null local查看docker_gwbridge的詳細信息：
iie4bu@swarm-worker1:~$ docker network inspect docker_gwbridge [{"Name": "docker_gwbridge","Id": "969e60257ba50b070374d31ea43a0550d6cd3ae3e68623746642fe8736dee5a4","Created": "2019-04-08T09:47:59.343371327+08:00","Scope": "local","Driver": "bridge","EnableIPv6": false,"IPAM": {"Driver": "default","Options": null,"Config": [{"Subnet": "172.19.0.0/16","Gateway": "172.19.0.1"}]},"Internal": false,"Attachable": false,"Ingress": false,"ConfigFrom": {"Network": ""},"ConfigOnly": false,"Containers": {"5559895ccaea972dad4f2fe52c0ec754d2d7c485dceb35083719768f611552e7": {"Name": "gateway_bf5031da0049","EndpointID": "6ad54f228134798de719549c0f93c804425beb85dd97f024408c4f7fc393fdf9","MacAddress": "02:42:ac:13:00:03","IPv4Address": "172.19.0.3/16","IPv6Address": ""},"ingress-sbox": {"Name": "gateway_ingress-sbox","EndpointID": "177757eca7a18630ae91c01b8ac67bada25ce1ea050dad4ac5cc318093062003","MacAddress": "02:42:ac:13:00:02","IPv4Address": "172.19.0.2/16","IPv6Address": ""}},"Options": {"com.docker.network.bridge.enable_icc": "false","com.docker.network.bridge.enable_ip_masquerade": "true","com.docker.network.bridge.name": "docker_gwbridge"},"Labels": {}} ]可以看到與docker_gwbridge相連的container有兩個，分別是gateway_bf5031da0049和gateway_ingress-sbox，而gateway_ingress-sbox的ip正是172.19.0.2。也就是說數據被轉發到這個gateway_ingress-sbox network namespace中去了。
進入gateway_ingress-sbox：
iie4bu@swarm-worker1:~$ sudo ls /var/run/docker/netns 1-cojus8blvk 1-uz1kgf9j6m 44e6e70b2177 b1ba5b4dd9f2 bf5031da0049 ingress_sbox iie4bu@swarm-worker1:~$ sudo nsenter --net=//var/run/docker/netns/ingress_sbox -bash: /home/iie4bu/anaconda3/etc/profiel.d/conda.sh: No such file or directoryCommandNotFoundError: Your shell has not been properly configured to use 'conda activate'. If your shell is Bash or a Bourne variant, enable conda for the current user with$ echo ". /home/iie4bu/anaconda3/etc/profile.d/conda.sh" >> ~/.bashrcor, for all users, enable conda with$ sudo ln -s /home/iie4bu/anaconda3/etc/profile.d/conda.sh /etc/profile.d/conda.shThe options above will permanently enable the 'conda' command, but they do NOT put conda's base (root) environment on PATH. To do so, run$ conda activatein your terminal, or to put the base environment on PATH permanently, run$ echo "conda activate" >> ~/.bashrcPrevious to conda 4.4, the recommended way to activate conda was to modify PATH in your ~/.bashrc file. You should manually remove the line that looks likeexport PATH="/home/iie4bu/anaconda3/bin:$PATH"^^^ The above line should NO LONGER be in your ~/.bashrc file! ^^^root@swarm-worker1:~#這樣就進入了ingress_sbox的namespace了。查看ip
?
發現ip是172.19.0.2。
為了掩飾LVS，我們退出ingress_sbox的namespace。在swarm-worker1中安裝ipvsadm，這是lvs的一個管理工具。
iie4bu@swarm-worker1:~$ sudo apt-get install ipvsadm安裝成功之后，在進入到ingress_sbox中，然后輸入命令iptables -nL -t mangle
root@swarm-worker1:~# iptables -nL -t mangle Chain PREROUTING (policy ACCEPT) target prot opt source destination MARK tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8000 MARK set 0x100Chain INPUT (policy ACCEPT) target prot opt source destination MARK all -- 0.0.0.0/0 10.255.0.168 MARK set 0x100Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destinationMARK一行就表示負載均衡。
輸入命令ipvsadm -l
root@swarm-worker1:~# ipvsadm -l IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port Forward Weight ActiveConn InActConn FWM 256 rr-> 10.255.0.5:0 Masq 1 0 0 -> 10.255.0.7:0 Masq 1 0 010.255.0.5?10.255.0.7就是whoami的service地址。
在swarm-manager上查看whoami：
iie4bu@swarm-manager:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES cc9f97cc5056 jwilder/whoami:latest "/app/http" 19 hours ago Up 19 hours 8000/tcp whoami.1.6hhuf528spdw9j9pla7l3tv3t iie4bu@swarm-manager:~$ docker exec cc9 ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet 10.255.0.168/32 brd 10.255.0.168 scope global lovalid_lft forever preferred_lft foreverinet 10.0.2.5/32 brd 10.0.2.5 scope global lovalid_lft forever preferred_lft forever 18: eth0@if19: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP link/ether 02:42:0a:ff:00:05 brd ff:ff:ff:ff:ff:ffinet 10.255.0.5/16 brd 10.255.255.255 scope global eth0valid_lft forever preferred_lft forever 20: eth1@if21: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP link/ether 02:42:ac:12:00:03 brd ff:ff:ff:ff:ff:ffinet 172.18.0.3/16 brd 172.18.255.255 scope global eth1valid_lft forever preferred_lft forever 23: eth2@if24: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP link/ether 02:42:0a:00:02:07 brd ff:ff:ff:ff:ff:ffinet 10.0.2.7/24 brd 10.0.2.255 scope global eth2valid_lft forever preferred_lft forever iie4bu@swarm-manager:~$whoami的IP正是10.255.0.5。
在swarm-worker2上查看：
iie4bu@swarm-worker2:~$ docker container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f47e05019fd9 jwilder/whoami:latest "/app/http" 20 hours ago Up 20 hours 8000/tcp whoami.3.9idgk9jbrlcm3ufvkmbmvv2t8 633ddfc082b9 busybox:1.28.3 "sh -c 'while true; …" 20 hours ago Up 20 hours client.1.3iv3gworpyr5vdo0h9eortlw0 iie4bu@swarm-worker2:~$ docker exec -it f47 ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet 10.255.0.168/32 brd 10.255.0.168 scope global lovalid_lft forever preferred_lft foreverinet 10.0.2.5/32 brd 10.0.2.5 scope global lovalid_lft forever preferred_lft forever 24: eth2@if25: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP link/ether 02:42:0a:00:02:0d brd ff:ff:ff:ff:ff:ffinet 10.0.2.13/24 brd 10.0.2.255 scope global eth2valid_lft forever preferred_lft forever 26: eth1@if27: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP link/ether 02:42:ac:12:00:04 brd ff:ff:ff:ff:ff:ffinet 172.18.0.4/16 brd 172.18.255.255 scope global eth1valid_lft forever preferred_lft forever 28: eth0@if29: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP link/ether 02:42:0a:ff:00:07 brd ff:ff:ff:ff:ff:ffinet 10.255.0.7/16 brd 10.255.255.255 scope global eth0valid_lft forever preferred_lft foreverwhoami的IP正是10.255.0.7。
因此當我們的數據包進入到ingress_sbox通過lvs做了一個負載均衡，也就是說我們訪問8000端口，它會把數據包轉發到10.255.0.5和10.255.0.7做一個負載。然后就會進入到swarm節點中了。
與50位技術專家面對面20年技術見證，附贈技術全景圖總結
以上是生活随笔為你收集整理的从零开始学习docker（二十）RoutingMesh--Ingress负载均衡的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： 从零开始学习docker（十九）Swar
- 下一篇： 从零开始学习docker（二十一）ser