基于 openstack 的 kubernetes(rancher RKE) 测试

近日测试 openstack 作为 kubernetes 的基础设施平台为 kubernetes 提供基础服务(storage, loadbalancer)

目标：

• 为 kubernetes 提供 cinder 作为持久卷
• 通过 openstack 的 LBAAS v2 为 kubernetes 提供 LoadBalancer

集群架构

遇到的问题

x509: cannot validate certificate for <ipaddress> because it doesn't contain any IP SANs

这个问题没有解决，最后通过创建一个新的 openstack region，让 cinder, neutron, compute 的接口都不走 https 来规避这个问题

cloud-config

由于是通过 rancher 进行管理，所以 cloud-config 直接写在 rancher 集群配置文件中即可

如果是原始的 kubernetes 集群，需要在 kubelet 启动时指定 cloud-config 文件

cloud_provider: 
  name: "openstack"
  openstackCloudProvider: 
    block_storage: 
      ignore-volume-az: true
      trust-device-path: false
    global: 
      auth-url: "http://10.10.23.100:45000/v3"
      domain-id: "default"
      password: "******"
      region: "RegionTest" # 笔者创建所有 endpoints 走 http 的 region
      tenant-id: "287da09439154ae7a00c5583fa28de4f"
      username: "anyisalin"
    load_balancer: 
      create-monitor: false
      manage-security-groups: false
      monitor-delay: 0
      monitor-max-retries: 0
      monitor-timeout: 0
      subnet-id: "092e753f-cead-4af0-9200-cf779d0cdf04" # 这里是 private network 的 uuid
      use-octavia: false
    metadata: 
      request-timeout: 0
    route: 
      router-id: "8c98f2a8-eec5-451d-9249-d9f2078f768d"

cinder storageclass

cinder 作为持久卷要指定 volume type 作为参数

apiVersion: storage.k8s.io/v1
kind: StorageClass
parameters:
  type: RBD
provisioner: kubernetes.io/cinder

$ openstack volume list # 持久卷在 openstack 中的展现
+--------------------------------------+-------------------------------------------------------------+-----------+------+---------------------------------------------------------------+
| ID                                   | Display Name                                                | Status    | Size | Attached to                                                   |
+--------------------------------------+-------------------------------------------------------------+-----------+------+---------------------------------------------------------------+
| e437f3b4-7f02-412a-b8ac-38d36d79c424 | kubernetes-dynamic-pvc-5bee62ff-aec6-11e8-868b-fa163efa03b6 | available |    1 |                                                               |
| c360e7f0-4bfa-4424-a564-fdb7330f5fa2 | kubernetes-dynamic-pvc-404fe3e5-aec6-11e8-868b-fa163efa03b6 | available |    1 |                                                               |
| 7e2347af-752c-42e3-a92a-c2565ab6b12c | kubernetes-dynamic-pvc-1193480a-aec6-11e8-b38a-fa163eccc6d3 | in-use    |    1 | Attached to worker3 on /dev/vdc                               |
| 77d26e4e-40d1-4227-b605-5150559a5ccf | tst222                                                      | available |    2 |                                                               |
| c40a48cd-c36e-4cf6-bc08-2c5f326794c4 | kubernetes-dynamic-pvc-d22e1c0e-aebf-11e8-868b-fa163efa03b6 | in-use    |   10 | Attached to worker3 on /dev/vdd                               |
| b331a11a-c533-41fe-89aa-e4bca48484c7 | kubernetes-dynamic-pvc-d234bd67-aebf-11e8-868b-fa163efa03b6 | in-use    |   10 | Attached to worker2 on /dev/vdc                               |
| 19bc473b-8215-45ef-b280-4e1d7c26dc56 | kubernetes-dynamic-pvc-d239d596-aebf-11e8-868b-fa163efa03b6 | in-use    |   10 | Attached to worker1 on /dev/vdc                               |
| f03591f7-a69f-402d-9ab5-d6350d8078e9 | kubernetes-dynamic-pvc-5c4a5d33-aebf-11e8-868b-fa163efa03b6 | in-use    |    8 | Attached to worker3 on /dev/vdb                               |
| a1383dde-bbc3-4148-a931-f86962d482a6 | kubernetes-dynamic-pvc-5c36d2c2-aebf-11e8-868b-fa163efa03b6 | in-use    |    8 | Attached to worker2 on /dev/vdb                               |
| 568f72bc-6b74-4cd2-b795-ab109f753501 | kubernetes-dynamic-pvc-5c406970-aebf-11e8-868b-fa163efa03b6 | in-use    |    2 | Attached to worker1 on /dev/vdb                               |
| 4e3d531e-3ea6-4f3f-a40b-3639548ed9bc | kubernetes-dynamic-pvc-0742b2bc-aebe-11e8-8fc8-fa163e64cc4c | available |    1 |                                                               |
| 9528824a-35f0-47be-88b9-2108ba10e4cb | kubernetes-dynamic-pvc-1dae4606-acce-11e8-ad52-fa163eae37fd | in-use    |    2 | Attached to 1a5e3cee-ed4e-4a31-963f-2c7a96470c63 on /dev/vdb  |
| f573db1e-2df1-48f9-b45f-237667b891fa | kubernetes-dynamic-pvc-1b7761b6-acc0-11e8-81f7-fa163ef239c4 | in-use    |    1 | Attached to 454551f2-5736-4b52-8c85-321f795e0cff on /dev/vdb  |
| 2aa122b4-887b-4eaa-977e-7bbd6a001449 | kubernetes-dynamic-pvc-6a297820-ac29-11e8-b902-fa163e785cc9 | available |    1 |                                                               |
| c6af8fc5-9c1e-4ae2-87c0-2798dbf80bc4 | kubernetes-dynamic-pvc-37419971-ac22-11e8-b864-fa163ecfcbd0 | available |    1 |                                                               |
| 4eef9daa-3311-4787-afb8-2e41fd39c9ca | kubernetes-dynamic-pvc-b9f73456-ac18-11e8-b932-fa163e0fecaa | available |    8 |                                                               |
| a0c0ad15-d674-4389-918f-d37126ffb59d | kubernetes-dynamic-pvc-b9fe461d-ac18-11e8-b932-fa163e0fecaa | available |   10 |                                                               |
+--------------------------------------+-------------------------------------------------------------+-----------+------+---------------------------------------------------------------+

LoadBalancer

$ neutron lbaas-member-list  2e75b034-d291-4e87-b9bf-2421bce1fc3a # LoadBalancer 服务在 openstack 中的展现
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+-----------------------------------------------------+------------+---------------+--------+--------------------------------------+----------------+
| id                                   | name                                                | address    | protocol_port | weight | subnet_id                            | admin_state_up |
+--------------------------------------+-----------------------------------------------------+------------+---------------+--------+--------------------------------------+----------------+
| 2b09be24-e6b8-4fa2-872a-7840691e3f74 | member_a11a75f2eaec611e8b38afa163eccc6d_0_worker3   | 13.20.0.16 |         31127 |      1 | 092e753f-cead-4af0-9200-cf779d0cdf04 | True           |
| 34b22906-d794-4c10-976f-74e00dfdbb91 | member_a11a75f2eaec611e8b38afa163eccc6d_0_control-2 | 13.20.0.14 |         31127 |      1 | 092e753f-cead-4af0-9200-cf779d0cdf04 | True           |
| 5a98c11c-544f-4414-9ec8-2991b7d123e8 | member_a11a75f2eaec611e8b38afa163eccc6d_0_worker2   | 13.20.0.9  |         31127 |      1 | 092e753f-cead-4af0-9200-cf779d0cdf04 | True           |
| 68a65652-7c4f-4b68-82e4-4a2731542664 | member_a11a75f2eaec611e8b38afa163eccc6d_0_control-3 | 13.20.0.24 |         31127 |      1 | 092e753f-cead-4af0-9200-cf779d0cdf04 | True           |
| 8fe6c1a0-c802-4eda-b3b7-ccef920500b6 | member_a11a75f2eaec611e8b38afa163eccc6d_0_worker1   | 13.20.0.22 |         31127 |      1 | 092e753f-cead-4af0-9200-cf779d0cdf04 | True           |
| b3a05a40-7428-4d77-a773-bfec5f93544e | member_a11a75f2eaec611e8b38afa163eccc6d_0_control-1 | 13.20.0.20 |         31127 |      1 | 092e753f-cead-4af0-9200-cf779d0cdf04 | True           |
+--------------------------------------+-----------------------------------------------------+------------+---------------+--------+--------------------------------------+----------------+

kubernetes LoadBalancer 的原理就是通过外部服务对 service 的 NodePort 进行反代，NodePort 是通过 Kube-Proxy 做转发的，无法获取真实客户端的 IP 地址，最好还是通过 ingress 服务访问集群中的服务