Kubernetes Multusの導入

Contents

1 概要
2 手順

概要

https://github.com/k8snetworkplumbingwg/multus-cni

Multusを導入します。MultusはPodに複数のNIC（CNI）を接続します。Multusで追加するNICはNodeの物理NICに紐づき、Pod Cidrとは異なるネットワークとなり、macvlanを使用する為、紐づけたNodeの物理NICと同じネットワークとなります。もしかすると2つ目のPod Cidrを割り当てることも可能？かも知れませんが、後日調査とします（できても意味がないかもしれません）。

デフォルトネットワーク（Pod Cidr）のCNIはCalicoを使用しています。

手順

NodeにNICを追加

Multus CNIと紐づけるNodeのNICを追加します。追加済みの場合は不要な手順です。Ubuntuの例となりますが、追加NIC（ens37）を99_config.yamlに追加します。

root@k8s-01:~# vim /etc/netplan/99_config.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    ens33:
      addresses:
        - 192.168.68.11/24
      gateway4: 192.168.68.1
      nameservers:
          search: [si1230.com]
          addresses: [192.168.64.2]
    ens37:
      addresses:
        - 192.168.69.11/24

root@k8s-01:~# netplan apply

Multusのインストール

インストール

git cloneでMultusをダウンロードし、multus-daemonset-thick-plugin.ymlをapplyします。

root@k8s-01:~# git clone https://github.com/k8snetworkplumbingwg/multus-cni.git && cd multus-cni
root@k8s-01:~# cat ./deployments/multus-daemonset-thick-plugin.yml | kubectl apply -f -

clusterrole.rbac.authorization.k8s.io/multus created
clusterrolebinding.rbac.authorization.k8s.io/multus created
serviceaccount/multus created
daemonset.apps/kube-multus-ds created

インストールの確認

Runningを確認します。

root@k8s-01:~# kubectl get pods --all-namespaces | grep -i multus

kube-system        kube-multus-ds-7vw49                      1/1     Running   0             62s
kube-system        kube-multus-ds-d6hvd                      1/1     Running   0             62s

Multus CNIの追加

CNIの追加

CNIを追加します。config内は追加する物理NICのネットワークに合わせます。

root@k8s-01:~# cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "macvlan",
      "master": "ens37",
      "mode": "bridge",
      "ipam": {
        "type": "host-local",
        "subnet": "192.168.69.0/24",
        "rangeStart": "192.168.69.200",
        "rangeEnd": "192.168.69.216",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ],
        "gateway": "192.168.69.1"
      }
    }'
EOF

networkattachmentdefinition.k8s.cni.cncf.io/macvlan-conf created

CNIの確認

macvlan-confの存在を確認します。

root@k8s-01:~# kubectl get network-attachment-definitions

NAME           AGE
macvlan-conf   13s

macvlan-confの詳細を確認します。

root@k8s-01:~# kubectl describe network-attachment-definitions macvlan-conf

Name:         macvlan-conf
Namespace:    default
Labels:       
Annotations:  
API Version:  k8s.cni.cncf.io/v1
Kind:         NetworkAttachmentDefinition
Metadata:
  Creation Timestamp:  2021-11-16T01:58:29Z
  Generation:          1
  Managed Fields:
    API Version:  k8s.cni.cncf.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:config:
    Manager:         kubectl-create
    Operation:       Update
    Time:            2021-11-16T01:58:29Z
  Resource Version:  441062
  UID:               16dde95d-5134-4034-85f9-cf2bc6876e27
Spec:
  Config:  { "cniVersion": "0.3.0", "type": "macvlan", "master": "ens37", "mode": "bridge", "ipam": { "type": "host-local", "subnet": "192.168.69.0/24", "rangeStart": "192.168.69.200", "rangeEnd": "192.168.69.216", "routes": [ { "dst": "0.0.0.0/0" } ], "gateway": "192.168.69.1" } }
Events:

Multus CNIを使用するPodの作成

Podの作成

コンテナイメージにalpineを使用しています。追加のCNIとして、annotationsでmacvlan-confを指定しています。

root@k8s-01:~# cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: samplepod
  annotations:
    k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
  containers:
  - name: samplepod
    command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: alpine
EOF

pod/samplepod created

Podの確認

追加したNIC（net1@eth0）が含まれています。eth0@if21はPod Cidrです。

root@k8s-01:~# kubectl exec -it samplepod -- ip a

1: lo:  mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: eth0@if21:  mtu 1450 qdisc noqueue state UP
    link/ether c6:71:4b:13:d3:01 brd ff:ff:ff:ff:ff:ff
    inet 10.255.179.41/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::c471:4bff:fe13:d301/64 scope link
       valid_lft forever preferred_lft forever
4: net1@eth0:  mtu 1500 qdisc noqueue state UP
    link/ether ca:90:d4:58:69:58 brd ff:ff:ff:ff:ff:ff
    inet 192.168.69.200/24 brd 192.168.69.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::c890:d4ff:fe58:6958/64 scope link
       valid_lft forever preferred_lft forever

追加したNICの疎通性を確認しています。

root@k8s-01:~# kubectl exec -it samplepod -- ping -c 4 192.168.69.1

PING 192.168.69.1 (192.168.69.1): 56 data bytes
64 bytes from 192.168.69.1: seq=0 ttl=128 time=0.411 ms
64 bytes from 192.168.69.1: seq=1 ttl=128 time=0.455 ms
64 bytes from 192.168.69.1: seq=2 ttl=128 time=0.412 ms
64 bytes from 192.168.69.1: seq=3 ttl=128 time=0.526 ms

Podの詳細を確認しています。

root@k8s-01:~# kubectl describe pod samplepod

Name:         samplepod
Namespace:    default
Priority:     0
Node:         k8s-02/192.168.68.12
Start Time:   Tue, 16 Nov 2021 02:06:52 +0000
Labels:       
Annotations:  cni.projectcalico.org/containerID: c0cb16eb837c2ff18ee5fae7ad4fffacf06e8cf63bfc7ebed534b3bf83297bdc
              cni.projectcalico.org/podIP: 10.255.179.41/32
              cni.projectcalico.org/podIPs: 10.255.179.41/32
              k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "k8s-pod-network",
                    "ips": [
                        "10.255.179.41"
                    ],
                    "default": true,
                    "dns": {}
                },{
                    "name": "default/macvlan-conf",
                    "interface": "net1",
                    "ips": [
                        "192.168.69.200"
                    ],
                    "mac": "ca:90:d4:58:69:58",
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks: macvlan-conf
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "k8s-pod-network",
                    "ips": [
                        "10.255.179.41"
                    ],
                    "default": true,
                    "dns": {}
                },{
                    "name": "default/macvlan-conf",
                    "interface": "net1",
                    "ips": [
                        "192.168.69.200"
                    ],
                    "mac": "ca:90:d4:58:69:58",
                    "dns": {}
                }]
Status:       Running
IP:           10.255.179.41
IPs:
  IP:  10.255.179.41
Containers:
  samplepod:
    Container ID:  containerd://22e1348217d8d85b05603fbf9b2448de06c505ace91bf487c2373405db5fdf23
    Image:         alpine
    Image ID:      docker.io/library/alpine@sha256:635f0aa53d99017b38d1a0aa5b2082f7812b03e3cdb299103fe77b5c8a07f1d2
    Port:          
    Host Port:     
    Command:
      /bin/ash
      -c
      trap : TERM INT; sleep infinity & wait
    State:          Running
      Started:      Tue, 16 Nov 2021 02:06:59 +0000
    Ready:          True
    Restart Count:  0
    Environment:    
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nz5jd (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-nz5jd:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason          Age    From               Message
  ----    ------          ----   ----               -------
  Normal  Scheduled       5m28s  default-scheduler  Successfully assigned default/samplepod to k8s-02
  Normal  AddedInterface  5m28s  multus             Add eth0 [10.255.179.41/32] from k8s-pod-network
  Normal  AddedInterface  5m28s  multus             Add net1 [192.168.69.200/24] from default/macvlan-conf
  Normal  Pulling         5m28s  kubelet            Pulling image "alpine"
  Normal  Pulled          5m22s  kubelet            Successfully pulled image "alpine" in 5.781145841s
  Normal  Created         5m22s  kubelet            Created container samplepod
  Normal  Started         5m22s  kubelet            Started container samplepod

目次に戻る