跳转至

浅识 k8s 集群环境

这一部分我们仔细看看k8s内部架构的运作方式, 基于现成的 minikube 即可, 不需要手动搭建

首先给一个 abstraction + implementation 的架构图:

alt text

在网上搜索各类资料学习的过程中，我们会经常看到各种构建k8s的方式。

我多次见到: minikube / kind / kubeadm ...

但很少有文章系统梳理k8s环境搭建的方式。

这里我们来分类整理一下, 按照从小到大的顺序枚举一下:

  1. 本地开发, 小型k8s集群, 用于测试
  2. 通用型集群部署工具, 用于 "企业级/校园级" 自己搭建的服务
  3. 现有的企业级服务

常用的集群构建方式

(1) 一体化本地集群工具

仅单节点集群

主要用于学习、开发和测试，可以在单台机器上快速启动一个功能完备的、轻量级的 k8s 集群

  1. minikube: 最经典的本地 k8s 环境
    1. 它在一个虚拟机（如 VirtualBox, Hyper-V, KVM）或容器内启动一个单节点的 Kubernetes 集群
  2. kind:
    1. 全称: K8s IN Docker
    2. 使用 Docker 容器作为集群的“节点”，可以快速创建单节点或多节点集群，非常适合进行 CI/CD 流水线中的集成测试
  3. k3s/K3d:
    1. K3s 是一个经过 CNCF 认证的轻量级 k8s 发行版，专为边缘计算、物联网等资源受限环境设计
    2. K3d 是一个在其基础上使用 Docker 运行多节点集群的工具
  4. Docker Desktop:
    1. 新版本的 Docker Desktop 内置了 Kubernetes 功能，只需在设置中勾选即可 一键启动 一个单节点集群

(2) 通用型集群部署工具

可用于构建多节点集群

更灵活、更强大的集群部署和管理能力，支持在多种环境（包括公有云和私有数据中心）中部署生产级别的集群

  1. kubeadm:
    1. 官方推荐的集群引导工具, 用于快速搭建符合最佳实践的/具备最小化可用性的 k8s 集群
    2. 负责初始化 Master 节点（控制平面）和将 Worker 节点加入集群, 但不负责底层基础设施的创建
  2. Kubespray:
    1. 自动化地在多种 Linux 发行版和云平台上部署生产级别的高可用 k8s 集群
  3. kops:
    1. 全称: Kubernetes Operations

PS: 给一个 用 kubeadm 搭建集群环境 的教程

(3) 托管式云服务

各大公有云厂商提供了“开箱即用”的 k8s 服务

用户开箱即用，可获得一个高可用的、可弹性伸缩的生产级集群

  1. GKE: Google Kubernetes Engine
  2. EKS: Amazon Elastic Kubernetes Service
  3. AKS: Azure Kubernetes Service

安装 k8s dashboard 辅助开发

Kubernetes Dashboard 是 k8s集群的一个 WEB UI 管理工具，安装地址: 传送门

我们提供两种搭建dashboard的方式:

  1. 在 minikube 中直接使用内置命令, 一键打开
  2. 配置 dashboard.yaml, 人工形成dashboard, 然后在浏览器里验证访问

(1) minikube 内置指令

在CLI中直接输入: minikube dashboard

然后浏览器就会自动打开:

Bash
1
2
3
4
5
❯ minikube dashboard
🤔  正在验证 dashboard 运行情况 ...
🚀  正在启动代理...
🤔  正在验证 proxy 运行状况 ...
🎉  正在使用默认浏览器打开 http://127.0.0.1:58350/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ ...

alt text

(2) 手动构建

1) 写好yaml文件

注意一定要在 spec 字段加上 type: NodePort (相较于直接wget官网的)

YAML
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# ------------------- Dashboard Secret ------------------- #

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-certs
  namespace: kube-system
type: Opaque

---
# ------------------- Dashboard Service Account ------------------- #

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system

---
# ------------------- Dashboard Role & Role Binding ------------------- #

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubernetes-dashboard-minimal
  namespace: kube-system
rules:
  # Allow Dashboard to create 'kubernetes-dashboard-key-holder' secret.
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["create"]
  # Allow Dashboard to create 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["create"]
  # Allow Dashboard to get, update and delete Dashboard exclusive secrets.
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs"]
  verbs: ["get", "update", "delete"]
  # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
  resources: ["configmaps"]
  resourceNames: ["kubernetes-dashboard-settings"]
  verbs: ["get", "update"]
  # Allow Dashboard to get metrics from heapster.
- apiGroups: [""]
  resources: ["services"]
  resourceNames: ["heapster"]
  verbs: ["proxy"]
- apiGroups: [""]
  resources: ["services/proxy"]
  resourceNames: ["heapster", "http:heapster:", "https:heapster:"]
  verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kubernetes-dashboard-minimal
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubernetes-dashboard-minimal
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system

---
# ------------------- Dashboard Deployment ------------------- #

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
    spec:
      containers:
      - name: kubernetes-dashboard
        image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1
        ports:
        - containerPort: 8443
          protocol: TCP
        args:
          - --auto-generate-certificates
          # Uncomment the following line to manually specify Kubernetes API server Host
          # If not specified, Dashboard will attempt to auto discover the API server and connect
          # to it. Uncomment only if the default does not work.
          # - --apiserver-host=http://my-address:port
        volumeMounts:
        - name: kubernetes-dashboard-certs
          mountPath: /certs
          # Create on-disk volume to store exec logs
        - mountPath: /tmp
          name: tmp-volume
        livenessProbe:
          httpGet:
            scheme: HTTPS
            path: /
            port: 8443
          initialDelaySeconds: 30
          timeoutSeconds: 30
      volumes:
      - name: kubernetes-dashboard-certs
        secret:
          secretName: kubernetes-dashboard-certs
      - name: tmp-volume
        emptyDir: {}
      serviceAccountName: kubernetes-dashboard
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule

---
# ------------------- Dashboard Service ------------------- #

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  ports:
    - port: 443
      targetPort: 8443
  type: NodePort # <- need specify
  selector:
    k8s-app: kubernetes-dashboard

2) 定义并启动服务

Bash
1
kubectl create -f ./kubernetes-dashboard.yaml

3) 检查服务, 准备访问

Bash
1
2
3
4
❯ kubectl get pods -A | grep kubernetes-dashboard
kube-system            kubernetes-dashboard-9499d7f7-x7clc          1/1     Running   0          30m
kubernetes-dashboard   dashboard-metrics-scraper-77bf4d6c4c-vgnz7   1/1     Running   0          9h
kubernetes-dashboard   kubernetes-dashboard-855c9754f9-p25fz        1/1     Running   0          9h

现在存在两种类型的dashboard:

  • namespace = kube-system: 来自 minikube dashboard 启动的服务
  • namespace = kubernetes-dashboard: 来自这里手动yaml创建的服务

我们只需要关心手动yaml创建的:

Bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
❯ kubectl describe svc kubernetes-dashboard -n kube-system
Name:                     kubernetes-dashboard
Namespace:                kube-system
Labels:                   k8s-app=kubernetes-dashboard
Annotations:              <none>
Selector:                 k8s-app=kubernetes-dashboard
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.101.234.255
IPs:                      10.101.234.255
Port:                     <unset>  443/TCP
TargetPort:               8443/TCP
NodePort:                 <unset>  32081/TCP
Endpoints:                10.244.0.10:8443
Session Affinity:         None
External Traffic Policy:  Cluster
Internal Traffic Policy:  Cluster
Events:                   <none>

说明一切正常!

详细解析kubectl describe svc kubernetes-dashboard -n kube-system

kubectl describe svc kubernetes-dashboard -n kube-system

  • svc: service, 它在k8s里表示: 为一组功能相同的 Pod 提供了一个统一稳定的访问入口（一个固定的 IP 地址和端口）
  • kubernetes-dashboard: service name
  • -n: --namespace 的缩写, 它告诉 kubectl 应该去哪个 命名空间 (Namespace) 里寻找这个名叫 kubernetes-dashboard 的服务
几个很容易混淆的port

TODO

  1. 这里 10.xxx 的 ip 都是 pod 内部 "subnet" 的, 因此外界浏览器不能直接访问
  2. NodePort: 32081 是暴露给外界访问的网络接口

4) 浏览器访问

  1. 得到 minikube "单集群" 的endpoint:
    Bash
    1
2
    minikube ip
# 随后我得到: 192.168.49.2
  2. 浏览器访问:
    Text Only
    1
2
    # [https://] + [cluster_exIP] + [:] + [NodePort]
https://192.168.49.2:32081

之后看到:

alt text

后面具体的认证、登陆等琐碎步骤直接参考这里