# kubeadm

## kubeadm pre actions

{% hint style="info" %}

> references:
>
> * [`kubeadm config`](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-config/)
>   * [`kubeadm config print [flags]`](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-config/#cmd-config-print)
>   * [`kubeadm config print init-defaults [flags]`](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-config/#cmd-config-print-init-defaults)
>   * [`kubeadm config print join-defaults [flags]`](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-config/#cmd-config-print-join-defaults)
>   * [`kubeadm config migrate [flags]`](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-config/#cmd-config-migrate)
>   * [`kubeadm config images list [flags]`](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-config/#cmd-config-images-list)
>   * [`kubeadm config images pull [flags]`](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-config/#cmd-config-images-pull)
>     {% endhint %}

### [list and pull images](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#custom-images)

```bash
# list
$ kubeadm config images list
# or
$ kubeadm config images list --config=kubeadm-config.yml

# pull
$ kubeadm config images pull
# or
$ kubeadm config images pull --config=kubeadm-config.yml
```

### [kubeadm config](https://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/#save-the-kubeadm-clusterconfiguration-in-a-configmap-for-later-reference)

> \[!TIP] kubeadm saves the configuration passed to `kubeadm init` in a ConfigMap named `kubeadm-config` under `kube-system` namespace.
>
> This will ensure that kubeadm actions executed in future (e.g \`kubeadm upgrade\`) will be able to determine the actual/current cluster state and make new decisions based on that data.
>
> ; Please note that: - Before saving the \`ClusterConfiguration\`, sensitive information like the token is stripped from the configuration - Upload of control plane node configuration can be invoked individually with the \[\`kubeadm init phase upload-config\`]\(<https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init-phase/#cmd-phase-upload-config>) command - \[\*\*get current \`kubeadm-cfg.yml\`\*\*]\(<https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#kubeconfig-additional-users>) \`\`\`bash $ kubectl get cm kubeadm-config -n kube-system -o yaml # or: $ kubectl get cm kubeadm-config -n kube-system -o=jsonpath="{.data.ClusterConfiguration}" \`\`\` - references: - \[kubeadm kubeconfig]\(<https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-kubeconfig/>)

#### show default kubeadm-conf

```bash
$ sudo kubeadm config print init-defaults
```

```bash
# v1.30.4
$ sudo kubeadm config print init-defaults
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef                            # placeholder
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 1.2.3.4                                 # current IP
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock     # cri socket
  imagePullPolicy: IfNotPresent
  name: node
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
                                                            # certSANs: []
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
                                                            # controlPlaneEndpoint: vip:port
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: 1.30.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
                                                            # add podSubnet
scheduler: {}
```

### [show default `KubeletConfiguration`](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/kubelet-integration/#configure-kubelets-using-kubeadm)

{% hint style="info" %}

> * for v1.12.3
>
>   ```bash
>   [apis]
>     Available values: [ InitConfiguration
>                         ClusterConfiguration
>                         JoinConfiguration
>                         KubeProxyConfiguration
>                         KubeletConfiguration
>                         MasterConfiguration
>     ]
>   ```

{% endhint %}

```bash
$ sudo kubeadm config print init-defaults --component-configs KubeletConfiguration
$ sudo kubeadm config print init-defaults --component-configs KubeProxyConfiguration

# v1.12.3
$ sudo kubeadm config print-default
$ sudo kubeadm config print-defaults
$ sudo kubeadm config print-defaults --api-objects [apis]
```

#### sample kubeadm-config

```bash
# verify
$ sudo kubeadm config validate --config /etc/kubernetes/kubeadm-config.yaml
W0909 00:02:51.443429 1809154 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [172.21.0.10]; the provided value is: [172.21.0.3]
ok
# or
$ kubeadm config validate --config kubeadm-conf.yaml
ok
```

* stacked etcd

  ```bash
  apiVersion: kubeadm.k8s.io/v1beta3
  kind: InitConfiguration
  localAPIEndpoint:
    bindPort: 6443
  nodeRegistration:
    criSocket: unix:///var/run/crio/crio.sock
    imagePullPolicy: IfNotPresent
    taints: null
  ---
  apiVersion: kubeadm.k8s.io/v1beta3
  kind: ClusterConfiguration
  apiServer:
    timeoutForControlPlane: 4m0s
    certSANs:
      - "k8s-api.sample.com"
      - "10.68.78.205"
      - "k8s-01.sample.com"
      - "10.68.78.221"
      - "k8s-02.sample.com"
      - "10.68.78.222"
      - "k8s-03.sample.com"
      - "10.68.78.223"
  certificatesDir: /etc/kubernetes/pki
  controlPlaneEndpoint: "k8s-api.sample.com:16443"
  imageRepository: registry.k8s.io
  kubernetesVersion: 1.30.0
  networking:
    dnsDomain: cluster.local
    podSubnet: 192.168.0.0/16
    serviceSubnet: 10.96.0.0/12
  clusterName: kubernetes
  ```
* extenal etcd

  ```bash
  apiVersion: kubeadm.k8s.io/v1beta2
  kind: ClusterConfiguration
  kubernetesVersion: v1.15.3
  controlPlaneEndpoint: "k8s-api:6443"
  etcd:
    external:
      endpoints:
        - https://10.69.78.40:2379
        - https://10.69.78.41:2379
        - https://10.69.78.42:2379
      caFile: /etc/etcd/ssl/ca.pem
      certFile: /etc/etcd/ssl/client.pem
      keyFile: /etc/etcd/ssl/client-key.pem
  networking:
    dnsDomain: cluster.local
    podSubnet: 10.244.0.0/16
    serviceSubnet: 10.96.0.0/12
  apiServer:
    certSANs:
      - 10.69.78.15
      - k8s-api
      - 10.69.78.40
      - k8s-01
      - 10.69.78.41
      - k8s-02
      - 10.69.78.42
      - k8s-03
    extraArgs:
      etcd-cafile: /etc/etcd/ssl/ca.pem
      etcd-certfile: /etc/etcd/ssl/client.pem
      etcd-keyfile: /etc/etcd/ssl/client-key.pem
    timeoutForControlPlane: 4m0s
  imageRepository: k8s.gcr.io
  clusterName: "kubernetes"
  ```
* full kubeadm from kubespray ( extenal etcd )

  ```bash
  apiVersion: kubeadm.k8s.io/v1beta3
  kind: InitConfiguration
  localAPIEndpoint:
    advertiseAddress: 10.68.78.221                      # current IP
    bindPort: 6443
  nodeRegistration:
    name: "k8s-01"                                      # node name
    taints:
    - effect: NoSchedule
      key: node-role.kubernetes.io/control-plane
    criSocket: unix:///var/run/crio/crio.sock
  ---
  apiVersion: kubeadm.k8s.io/v1beta3
  kind: ClusterConfiguration
  clusterName: cluster.local
  etcd:
    external:
        endpoints:
        - https://10.68.78.221:2379
        - https://10.68.78.222:2379
        - https://10.68.78.223:2379
        caFile: /etc/ssl/etcd/ssl/ca.pem
        certFile: /etc/ssl/etcd/ssl/node-k8s-01.pem
        keyFile: /etc/ssl/etcd/ssl/node-k8s-01-key.pem
  dns:
    imageRepository: registry.k8s.io/coredns
    imageTag: v1.11.1
  networking:
    dnsDomain: cluster.local
    serviceSubnet: "172.21.0.0/16"
    podSubnet: "10.244.0.0/16"
  kubernetesVersion: v1.30.4
  controlPlaneEndpoint: k8s-api.sample.com:16443
  certificatesDir: /etc/kubernetes/ssl
  imageRepository: registry.k8s.io
  apiServer:
    extraArgs:
      etcd-compaction-interval: "5m0s"
      default-not-ready-toleration-seconds: "300"
      default-unreachable-toleration-seconds: "300"
      anonymous-auth: "True"
      authorization-mode: Node,RBAC
      bind-address: 0.0.0.0
      apiserver-count: "3"
      endpoint-reconciler-type: lease
      service-node-port-range: 30000-32767
      service-cluster-ip-range: "172.21.0.0/16"
      kubelet-preferred-address-types: "InternalDNS,InternalIP,Hostname,ExternalDNS,ExternalIP"
      profiling: "False"
      request-timeout: "1m0s"
      enable-aggregator-routing: "False"
      service-account-lookup: "True"
      storage-backend: etcd3
      allow-privileged: "true"
      event-ttl: 1h0m0s
    extraVolumes:
    - name: etc-pki-tls
      hostPath: /etc/pki/tls
      mountPath: /etc/pki/tls
      readOnly: true
    - name: etc-pki-ca-trust
      hostPath: /etc/pki/ca-trust
      mountPath: /etc/pki/ca-trust
      readOnly: true
    certSANs:
    - "kubernetes"
    - "kubernetes.default"
    - "kubernetes.default.svc"
    - "kubernetes.default.svc.cluster.local"
    - "172.21.0.1"
    - "localhost"
    - "127.0.0.1"
    - "10.68.78.205"
    - "k8s-api.sample.com"
    - "10.68.78.221"
    - "k8s-01"
    - "k8s-01.sample.com"
    - "10.68.78.222"
    - "k8s-02"
    - "k8s-02.sample.com"
    - "10.68.78.223"
    - "k8s-03"
    - "k8s-03.sample.com"
    timeoutForControlPlane: 5m0s
  controllerManager:
    extraArgs:
      node-monitor-grace-period: 40s
      node-monitor-period: 5s
      cluster-cidr: "10.244.0.0/16"
      service-cluster-ip-range: "172.21.0.0/16"
      allocate-node-cidrs: "false"
      profiling: "False"
      terminated-pod-gc-threshold: "12500"
      bind-address: 0.0.0.0
      leader-elect-lease-duration: 15s
      leader-elect-renew-deadline: 10s
      configure-cloud-routes: "false"
  scheduler:
    extraArgs:
      bind-address: 0.0.0.0
      config: /etc/kubernetes/kubescheduler-config.yaml
      profiling: "False"
    extraVolumes:
    - name: kubescheduler-config
      hostPath: /etc/kubernetes/kubescheduler-config.yaml
      mountPath: /etc/kubernetes/kubescheduler-config.yaml
      readOnly: true
  ---
  apiVersion: kubeproxy.config.k8s.io/v1alpha1
  kind: KubeProxyConfiguration
  bindAddress: 0.0.0.0
  clientConnection:
    acceptContentTypes:
    burst: 10
    contentType: application/vnd.kubernetes.protobuf
    kubeconfig:
    qps: 5
  clusterCIDR: "10.244.0.0/16"
  configSyncPeriod: 15m0s
  conntrack:
    maxPerCore: 32768
    min: 131072
    tcpCloseWaitTimeout: 1h0m0s
    tcpEstablishedTimeout: 24h0m0s
  enableProfiling: False
  healthzBindAddress: 0.0.0.0:10256
  hostnameOverride: "k8s-01"                            # node name
  iptables:
    masqueradeAll: False
    masqueradeBit: 14
    minSyncPeriod: 0s
    syncPeriod: 30s
  ipvs:
    excludeCIDRs: []
    minSyncPeriod: 0s
    scheduler: rr
    syncPeriod: 30s
    strictARP: True
    tcpTimeout: 0s
    tcpFinTimeout: 0s
    udpTimeout: 0s
  metricsBindAddress: 127.0.0.1:10249
  mode: ipvs
  nodePortAddresses: []
  oomScoreAdj: -999
  portRange:
  ---
  apiVersion: kubelet.config.k8s.io/v1beta1
  kind: KubeletConfiguration
  clusterDNS:
  - 172.21.0.3
  ```

## kubeadm init

![how kubeadm init](https://1999659500-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MHqOSW6on5cHbtwcSH6%2Fuploads%2Fgit-blob-445c6ebcbd6c384138aebc80a07ec9135f4cbbae%2FKubeadm-init.png?alt=media)

### environment

> \[!TIP|label:see also]
>
> * [\* iMarslo: environment](https://imarslo.gitbook.io/handbook/virtualization/kubernetes/init/kubeadm/env)

#### basic environment

> \[!TIP|label:see also]
>
> * [\* iMarslo: environment setup](https://imarslo.gitbook.io/handbook/virtualization/kubernetes/init/env#environment-setup)

* pre-install

  ```bash
  # rhel/centos
  $ sudo dnf install -y conntrack-tools socat ipvsadm ipset yum-utils jq \
             bash-completion net-tools htop rsync rsync-daemon traceroute
  ```
* swap

  ```bash
  $ sudo swapoff -a
  $ grep -q -E '^[^#]*swap' /etc/fstab && sudo sed -re 's:^[^#]*swap.*:# &:' -i /etc/fstab
  ```
* selinux

  ```bash
  $ sudo setenforce 0
  $ sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
  ```
* sysctl

  ```bash
  $ sudo grep -q -E '^[^#]*ipv4.ip_forward' /etc/sysctl.conf && sudo sed -re 's:^[^#]*ipv4.ip_forward.*:# &:' -i /etc/sysctl.conf
  $ sudo bash -c "cat >> /etc/sysctl.d/k8s.conf" << EOF
  net.ipv4.ip_forward = 1
  net.ipv4.ip_nonlocal_bind = 1
  vm.swappiness = 0
  net.bridge.bridge-nf-call-ip6tables = 1
  net.bridge.bridge-nf-call-iptables = 1
  EOF

  $ sudo sysctl -p /etc/sysctl.conf
  $ sudo sysctl -p /etc/sysctl.d/k8s.conf
  $ sudo sysctl --system
  ```
* modules

  ```bash
  $ sudo modprobe br_netfilter
  $ sudo modprobe overlay
  $ sudo modprobe bridge                   # optional
  $ sudo bash -c "cat /etc/modules-load.d/k8s.conf" << EOF
  overlay
  br_netfilter
  bridge
  EOF

  # check
  $ sudo lsmod | grep br_netfilter

  # module for ipvs (optional)
  $ cat << EOF >> ./ipvs.module
  modprobe -- ip_vs
  modprobe -- ip_vs_sh
  modprobe -- ip_vs_rr
  modprobe -- ip_vs_wrr
  modprobe -- nf_conntrack
  EOF
  $ chmod 755 ./ipvs.module && ./ipvs.module

  # or
  $ sudo bash -c "cat /etc/modules-load.d/ipvs.conf" << EOF
  ip_vs
  ip_vs_sh
  ip_vs_rr
  ip_vs_wrr
  nf_conntrack
  EOF
  $ sudo modprobe -- ip_vs
  ```
* firewall

  ```bash
  $ sudo systemctl stop firewalld
  $ sudo systemctl disable firewalld
  $ sudo systemctl mask firewalld
  ```

#### container runtime

* [docker-ce](https://imarslo.gitbook.io/handbook/virtualization/docker)
* [cri-o](https://imarslo.gitbook.io/handbook/virtualization/crio)

  ```bash
  $ CRIO_VERSION='v1.30'
  $ cat <<EOF | sudo tee /etc/yum.repos.d/cri-o.repo
  [cri-o]
  name=CRI-O
  baseurl=https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/rpm/
  enabled=1
  gpgcheck=1
  gpgkey=https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/rpm/repodata/repomd.xml.key
  exclude=cri-o
  EOF

  $ sudo bash -c "cat >>/etc/modules-load.d/crio.conf" << EOF
  overlay
  br_netfilter
  EOF

  $ sudo dnf install -y container-selinux
  $ sudo dnf install -y cri-o-1.30.4-150500.1.1 --disableexcludes=cri-o

  $ sudo systemctl enable --now crio.service
  ```

#### kubernetes packages

> \[!NOTE|label:more versions:]
>
> * rhel/centos
>   * [ol8 - v1.30.4](https://imarslo.gitbook.io/handbook/virtualization/kubernetes/init/env#ol8-v1304)
>   * [el7 - v1.15.3](https://imarslo.gitbook.io/handbook/virtualization/kubernetes/init/env#el7-v1153)
>   * [el7 - v1.12.3](https://imarslo.gitbook.io/handbook/virtualization/kubernetes/init/env#el7-v1123)

```bash
# ol8 - v1.30.4
$ cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.30/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.30/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

# get version
$ sudo dnf search --showduplicates kubeadm | grep 1.30.4

# install
$ VERSION='1.30.4-150500.1.1'
$ sudo dnf install -y kubelet-${VERSION} kubeadm-${VERSION} kubectl-${VERSION} --disableexcludes=kubernetes
$ sudo systemctl enable --now kubelet.service

# lock kube* for auto upgrade
$ sudo tail -1 /etc/yum.repos.d/kubernetes.repo
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
```

### init first control plane

```bash
$ sudo kubeadm init --config kubeadm-conf.yaml --ignore-preflight-errors=all --upload-certs --v=5

$ [[ -d $HOME/.kube ]] && rm -rf $HOME/.kube
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

# re-upload certs for HA
$ sudo kubeadm init phase upload-certs --upload-certs [ --config kubeadm-conf.yaml ]
```

```bash
$ sudo kubeadm init --config kubeadm-conf.yaml --upload-certs --v=5
I0906 15:55:01.147439  451702 initconfiguration.go:260] loading configuration from "kubeadm-conf.yaml"
I0906 15:55:01.148518  451702 interface.go:432] Looking for default routes with IPv4 addresses
I0906 15:55:01.148530  451702 interface.go:437] Default route transits interface "ens7"
I0906 15:55:01.148815  451702 interface.go:209] Interface ens7 is up
I0906 15:55:01.148860  451702 interface.go:257] Interface "ens7" has 3 addresses :[10.68.78.221/24 10.68.78.205/32 fe80::546f:c3ff:fe29:40e/64].
I0906 15:55:01.148873  451702 interface.go:224] Checking addr  10.68.78.221/24.
I0906 15:55:01.148887  451702 interface.go:231] IP found 10.68.78.221
I0906 15:55:01.148902  451702 interface.go:263] Found valid IPv4 address 10.68.78.221 for interface "ens7".
I0906 15:55:01.148916  451702 interface.go:443] Found active IP 10.68.78.221
I0906 15:55:01.148936  451702 kubelet.go:196] the value of KubeletConfiguration.cgroupDriver is empty; setting it to "systemd"
[init] Using Kubernetes version: v1.30.0
[preflight] Running pre-flight checks
I0906 15:55:01.157167  451702 checks.go:561] validating Kubernetes and kubeadm version
I0906 15:55:01.157191  451702 checks.go:166] validating if the firewall is enabled and active
I0906 15:55:01.167842  451702 checks.go:201] validating availability of port 6443
I0906 15:55:01.168088  451702 checks.go:201] validating availability of port 10259
I0906 15:55:01.168117  451702 checks.go:201] validating availability of port 10257
I0906 15:55:01.168150  451702 checks.go:278] validating the existence of file /etc/kubernetes/manifests/kube-apiserver.yaml
I0906 15:55:01.168170  451702 checks.go:278] validating the existence of file /etc/kubernetes/manifests/kube-controller-manager.yaml
I0906 15:55:01.168180  451702 checks.go:278] validating the existence of file /etc/kubernetes/manifests/kube-scheduler.yaml
I0906 15:55:01.168188  451702 checks.go:278] validating the existence of file /etc/kubernetes/manifests/etcd.yaml
I0906 15:55:01.168203  451702 checks.go:428] validating if the connectivity type is via proxy or direct
I0906 15:55:01.168230  451702 checks.go:467] validating http connectivity to first IP address in the CIDR
I0906 15:55:01.168252  451702 checks.go:467] validating http connectivity to first IP address in the CIDR
I0906 15:55:01.168271  451702 checks.go:102] validating the container runtime
I0906 15:55:01.187238  451702 checks.go:637] validating whether swap is enabled or not
I0906 15:55:01.187313  451702 checks.go:368] validating the presence of executable crictl
I0906 15:55:01.187350  451702 checks.go:368] validating the presence of executable conntrack
I0906 15:55:01.187379  451702 checks.go:368] validating the presence of executable ip
I0906 15:55:01.187398  451702 checks.go:368] validating the presence of executable iptables
I0906 15:55:01.187414  451702 checks.go:368] validating the presence of executable mount
I0906 15:55:01.187427  451702 checks.go:368] validating the presence of executable nsenter
I0906 15:55:01.187445  451702 checks.go:368] validating the presence of executable ebtables
I0906 15:55:01.187461  451702 checks.go:368] validating the presence of executable ethtool
I0906 15:55:01.187475  451702 checks.go:368] validating the presence of executable socat
I0906 15:55:01.187497  451702 checks.go:368] validating the presence of executable tc
I0906 15:55:01.187522  451702 checks.go:368] validating the presence of executable touch
I0906 15:55:01.187536  451702 checks.go:514] running all checks
I0906 15:55:01.196133  451702 checks.go:399] checking whether the given node name is valid and reachable using net.LookupHost
I0906 15:55:01.196278  451702 checks.go:603] validating kubelet version
I0906 15:55:01.234949  451702 checks.go:128] validating if the "kubelet" service is enabled and active
I0906 15:55:01.253904  451702 checks.go:201] validating availability of port 10250
I0906 15:55:01.253971  451702 checks.go:327] validating the contents of file /proc/sys/net/ipv4/ip_forward
I0906 15:55:01.254007  451702 checks.go:201] validating availability of port 2379
I0906 15:55:01.254029  451702 checks.go:201] validating availability of port 2380
I0906 15:55:01.254056  451702 checks.go:241] validating the existence and emptiness of directory /var/lib/etcd
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0906 15:55:01.254181  451702 checks.go:830] using image pull policy: IfNotPresent
I0906 15:55:01.289887  451702 checks.go:862] image exists: registry.k8s.io/kube-apiserver:v1.30.0
I0906 15:55:01.307358  451702 checks.go:862] image exists: registry.k8s.io/kube-controller-manager:v1.30.0
I0906 15:55:01.325819  451702 checks.go:862] image exists: registry.k8s.io/kube-scheduler:v1.30.0
I0906 15:55:01.343275  451702 checks.go:862] image exists: registry.k8s.io/kube-proxy:v1.30.0
I0906 15:55:01.359979  451702 checks.go:862] image exists: registry.k8s.io/coredns/coredns:v1.11.1
I0906 15:55:01.377002  451702 checks.go:862] image exists: registry.k8s.io/pause:3.9
I0906 15:55:01.394134  451702 checks.go:862] image exists: registry.k8s.io/etcd:3.5.12-0
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I0906 15:55:01.394225  451702 certs.go:112] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I0906 15:55:01.523905  451702 certs.go:483] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-01 k8s-01.sample.com k8s-02.sample.com k8s-03.sample.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local k8s-api.sample.com] and IPs [10.96.0.1 10.68.78.221 10.68.78.205 10.68.78.222 10.68.78.223]
[certs] Generating "apiserver-kubelet-client" certificate and key
I0906 15:55:02.031600  451702 certs.go:112] creating a new certificate authority for front-proxy-ca
[certs] Generating "front-proxy-ca" certificate and key
I0906 15:55:02.161869  451702 certs.go:483] validating certificate period for front-proxy-ca certificate
[certs] Generating "front-proxy-client" certificate and key
I0906 15:55:02.252315  451702 certs.go:112] creating a new certificate authority for etcd-ca
[certs] Generating "etcd/ca" certificate and key
I0906 15:55:02.503018  451702 certs.go:483] validating certificate period for etcd/ca certificate
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-01 localhost] and IPs [10.68.78.221 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-01 localhost] and IPs [10.68.78.221 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
I0906 15:55:03.093628  451702 certs.go:78] creating new public/private key files for signing service account users
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0906 15:55:03.488774  451702 kubeconfig.go:112] creating kubeconfig file for admin.conf
W0906 15:55:03.488970  451702 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
I0906 15:55:03.762480  451702 kubeconfig.go:112] creating kubeconfig file for super-admin.conf
W0906 15:55:03.762653  451702 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "super-admin.conf" kubeconfig file
I0906 15:55:03.904203  451702 kubeconfig.go:112] creating kubeconfig file for kubelet.conf
W0906 15:55:03.904377  451702 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
I0906 15:55:04.503418  451702 kubeconfig.go:112] creating kubeconfig file for controller-manager.conf
W0906 15:55:04.503611  451702 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0906 15:55:04.802570  451702 kubeconfig.go:112] creating kubeconfig file for scheduler.conf
W0906 15:55:04.802724  451702 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0906 15:55:04.997083  451702 local.go:65] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
I0906 15:55:04.997123  451702 manifests.go:103] [control-plane] getting StaticPodSpecs
I0906 15:55:04.997262  451702 certs.go:483] validating certificate period for CA certificate
I0906 15:55:04.997309  451702 manifests.go:129] [control-plane] adding volume "ca-certs" for component "kube-apiserver"
I0906 15:55:04.997316  451702 manifests.go:129] [control-plane] adding volume "etc-pki" for component "kube-apiserver"
I0906 15:55:04.997320  451702 manifests.go:129] [control-plane] adding volume "k8s-certs" for component "kube-apiserver"
I0906 15:55:04.997876  451702 manifests.go:158] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
I0906 15:55:04.997895  451702 manifests.go:103] [control-plane] getting StaticPodSpecs
I0906 15:55:04.998022  451702 manifests.go:129] [control-plane] adding volume "ca-certs" for component "kube-controller-manager"
I0906 15:55:04.998030  451702 manifests.go:129] [control-plane] adding volume "etc-pki" for component "kube-controller-manager"
I0906 15:55:04.998033  451702 manifests.go:129] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager"
I0906 15:55:04.998036  451702 manifests.go:129] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager"
I0906 15:55:04.998044  451702 manifests.go:129] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager"
I0906 15:55:04.998582  451702 manifests.go:158] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[control-plane] Creating static Pod manifest for "kube-scheduler"
I0906 15:55:04.998598  451702 manifests.go:103] [control-plane] getting StaticPodSpecs
I0906 15:55:04.998722  451702 manifests.go:129] [control-plane] adding volume "kubeconfig" for component "kube-scheduler"
I0906 15:55:04.999070  451702 manifests.go:158] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
I0906 15:55:04.999092  451702 kubelet.go:68] Stopping the kubelet
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 1.001504321s
[api-check] Waiting for a healthy API server. This can take up to 4m0s
I0906 15:55:07.167743  451702 with_retry.go:234] Got a Retry-After 1s response for attempt 1 to https://k8s-api.sample.com:16443/healthz?timeout=10s
I0906 15:55:08.169181  451702 with_retry.go:234] Got a Retry-After 1s response for attempt 2 to https://k8s-api.sample.com:16443/healthz?timeout=10s
I0906 15:55:09.169991  451702 with_retry.go:234] Got a Retry-After 1s response for attempt 3 to https://k8s-api.sample.com:16443/healthz?timeout=10s
I0906 15:55:10.170866  451702 with_retry.go:234] Got a Retry-After 1s response for attempt 4 to https://k8s-api.sample.com:16443/healthz?timeout=10s
[api-check] The API server is healthy after 4.009771869s
I0906 15:55:10.178049  451702 kubeconfig.go:608] ensuring that the ClusterRoleBinding for the kubeadm:cluster-admins Group exists
I0906 15:55:10.179174  451702 kubeconfig.go:681] creating the ClusterRoleBinding for the kubeadm:cluster-admins Group by using super-admin.conf
I0906 15:55:10.186100  451702 uploadconfig.go:112] [upload-config] Uploading the kubeadm ClusterConfiguration to a ConfigMap
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
I0906 15:55:10.195759  451702 uploadconfig.go:126] [upload-config] Uploading the kubelet component config to a ConfigMap
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
I0906 15:55:10.202391  451702 uploadconfig.go:131] [upload-config] Preserving the CRISocket information for the control plane
I0906 15:55:10.202417  451702 patchnode.go:31] [patchnode] Uploading the CRI Socket information "unix:///var/run/crio/crio.sock" to the Node API object "k8s-01" as an annotation
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
35c07b0896a3075ffa0df5f6dfbeb1a23119e3ffb9a6aa943135eb924601aea8
[mark-control-plane] Marking the node k8s-01 as control-plane by adding the labels: [node-role.kubernetes.io/control plane.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-01 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: umo0bu.h7c4rda7p2sovlk2
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
I0906 15:55:12.251287  451702 clusterinfo.go:47] [bootstrap-token] loading admin kubeconfig
I0906 15:55:12.251670  451702 clusterinfo.go:58] [bootstrap-token] copying the cluster from admin.conf to the bootstrap kubeconfig
I0906 15:55:12.251843  451702 clusterinfo.go:70] [bootstrap-token] creating/updating ConfigMap in kube-public namespace
I0906 15:55:12.403197  451702 request.go:629] Waited for 151.280911ms due to client-side throttling, not priority and fairness, request: POST:https://k8s-api.sample.com:16443/api/v1/namespaces/kube-public/configmaps?timeout=10s
I0906 15:55:12.406139  451702 clusterinfo.go:84] creating the RBAC rules for exposing the cluster-info ConfigMap in the kube-public namespace
I0906 15:55:12.410852  451702 kubeletfinalize.go:91] [kubelet-finalize] Assuming that kubelet client certificate rotation is enabled: found "/var/lib/kubelet/pki/kubelet-client-current.pem"
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
I0906 15:55:12.411566  451702 kubeletfinalize.go:145] [kubelet-finalize] Restarting the kubelet to enable client certificate rotation
I0906 15:55:12.803876  451702 request.go:629] Waited for 198.318189ms due to client-side throttling, not priority and fairness, request: POST:https://k8s-api.sample.com:16443/api/v1/namespaces/kube-system/configmaps?timeout=10s
I0906 15:55:13.003747  451702 request.go:629] Waited for 192.323644ms due to client-side throttling, not priority and fairness, request: POST:https://k8s-api.sample.com:16443/api/v1/namespaces/kube-system/serviceaccounts?timeout=10s
I0906 15:55:13.203497  451702 request.go:629] Waited for 189.318029ms due to client-side throttling, not priority and fairness, request: POST:https://k8s-api.sample.com:16443/api/v1/namespaces/kube-system/services?timeout=10s
[addons] Applied essential addon: CoreDNS
W0906 15:55:13.208744  451702 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
I0906 15:55:13.403414  451702 request.go:629] Waited for 193.327555ms due to client-side throttling, not priority and fairness, request: POST:https://k8s-api.sample.com:16443/api/v1/namespaces/kube-system/configmaps?timeout=10s
I0906 15:55:13.603875  451702 request.go:629] Waited for 192.131734ms due to client-side throttling, not priority and fairness, request: POST:https://k8s-api.sample.com:16443/api/v1/namespaces/kube-system/serviceaccounts?timeout=10s
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control plane running the following command on each as root:

  kubeadm join k8s-api.sample.com:16443 --token umo0bu.h7c4rda7p2sovlk2 \
        --discovery-token-ca-cert-hash sha256:16ec52ecec38a27cd42006451bf5403655f8f3c1fb74f54f1c019b5937ce8c87 \
        --control-plane --certificate-key 35c07b0896a3075ffa0df5f6dfbeb1a23119e3ffb9a6aa943135eb924601aea8

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join k8s-api.sample.com:16443 --token umo0bu.h7c4rda7p2sovlk2 \
        --discovery-token-ca-cert-hash sha256:16ec52ecec38a27cd42006451bf5403655f8f3c1fb74f54f1c019b5937ce8c87
```

## kubeadm join

> \[!TIP|label:reference:]
>
> * [kubeadm join](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-join/)
>   * `openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'`
>   * [Using init phases with kubeadm](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#init-phases)
>     * `sudo kubeadm init phase control-plane controller-manager --help`
>     * `sudo kubeadm init phase control-plane --help`
> * [Uploading control-plane certificates to the cluster](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#uploading-control-plane-certificates-to-the-cluster)
> * [kubeadm join](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-join/#token-based-discovery-with-ca-pinning)
> * [How do I find the join command for kubeadm on the master?](https://stackoverflow.com/a/55078533/2940319)

### join as control plane

> \[!NOTE|label:join command:] sync certs if init without `--upload-certs`
>
> ```bash
> # in k8s-01
> $ echo {02..03} | fmt -1 | while read -r _node; do
>     sudo rsync -avzrlpgoDP /etc/kubernetes/pki root@k8s-${_node}:/etc/kubernetes/pki
> done
>
> # or: https://blog.csdn.net/Weixiaohuai/article/details/135478349
> $ sudo scp /etc/kubernetes/admin.conf root@k8s-02:/etc/kubernetes/
> $ sudo scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} root@k8s-02:/etc/kubernetes/pki/
> $ sudo scp /etc/kubernetes/pki/etcd/ca.* root@k8s-02:/etc/kubernetes/pki/etc/
> ```

```bash
# in k8s-02 and k8s-03
$ sudo kubeadm join k8s-api.sample.com:16443 --token umo0bu.h7c4rda7p2sovlk2 \
        --discovery-token-ca-cert-hash sha256:16ec52ecec38a27cd42006451bf5403655f8f3c1fb74f54f1c019b5937ce8c87 \
        --control-plane --certificate-key 35c07b0896a3075ffa0df5f6dfbeb1a23119e3ffb9a6aa943135eb924601aea8 \
        --v=5

$ [[ -d $HOME/.kube ]] && rm -rf $HOME/.kube
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
```

### join as worker node

```bash
$ sudo swapoff -a
$ grep -q -E '^[^#]*swap' /etc/fstab && sudo sed -re 's:^[^#]*swap.*:# &:' -i /etc/fstab

$ sudo kubeadm join k8s-api.sample.com:16443 --token umo0bu.h7c4rda7p2sovlk2 \
               --discovery-token-ca-cert-hash sha256:16ec52ecec38a27cd42006451bf5403655f8f3c1fb74f54f1c019b5937ce8c87 \
               --v=5
```

### get join command

```bash
$ sudo kubeadm token create --print-join-command

# or
$ sudo kubeadm token create --print-join-command --ttl=0

# list
$ sudo kubeadm token list
```

* [token-based discovery with CA pinning](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-join/#token-based-discovery-with-ca-pinning)

  ```bash
  $ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt |
            openssl rsa -pubin -outform der 2>/dev/null |
            openssl dgst -sha256 -hex |
            sed 's/^.* //'
  ```

  * for worker nodes

    ```bash
    $ kubeadm join --discovery-token abcdef.1234567890abcdef --discovery-token-ca-cert-hash sha256:1234..cdef 1.2.3.4:6443
    ```
  * for control planes

    ```bash
    $ kubeadm join --discovery-token abcdef.1234567890abcdef --discovery-token-ca-cert-hash sha256:1234..cdef --control-plane 1.2.3.4:6443
    ```

### [retrive join command](https://stackoverflow.com/q/51126164/2940319)

* token ca hash

  ```bash
  $ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt \
      | openssl rsa -pubin -outform der 2>/dev/null \
      | openssl dgst -sha256 -hex \
      | sed 's/^.* //'
  ```
* bootstrap token

  ```bash
  $ kubeadm token list
  ```
* final command

  ```bash
  $ kubeadm join <ip-address>:6443\
      --token=<bootstrap-token> \
      --discovery-token-ca-cert-hash sha256:<ca-hash>
  ```
* function

  ```bash
  # get the join command from the kube master
  CERT_HASH=$(openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt \
                      | openssl rsa -pubin -outform der 2>/dev/null \
                      | openssl dgst -sha256 -hex \
                      | sed 's/^.* //')
  TOKEN=$(kubeadm token list -o json | jq -r '.token' | head -1)
  IP=$(kubectl get nodes -lnode-role.kubernetes.io/master -o json \
               | jq -r '.items[0].status.addresses[] | select(.type=="InternalIP") | .address')
  PORT=6443
  echo "sudo kubeadm join $IP:$PORT --token=$TOKEN --discovery-token-ca-cert-hash sha256:$CERT_HASH"
  ```

## [kubeadm upgrade](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/)

### upgrade kubeadm

```bash
$ sudo killall -s SIGTERM kube-apiserver            # trigger a graceful kube-apiserver shutdown
$ sleep 20

# upgrade kube*
$ sudo yum list --showduplicates kubeadm --disableexcludes=kubernetes
$ VERSION=1.31.x
$ sudo dnf install -y kubeadm-"${VERSION}-*" --disableexcludes=kubernetes
```

### upgrade cluster

```bash
# check upgrade plan
$ sudo kubeadm upgrade plan

# first control plane
$ sudo kubeadm upgrade apply v1.31.x

# peers control planes
$ sudo kubeadm upgrade node

# drain the node
$ kubectl drain <node-to-drain> --ignore-daemonsets
```

### upgrade kubelet and kubectl

```bash
$ sudo dnf install -y kubelet-"${VERSION}-*" kubectl-'${VERSION}-*' --disableexcludes=kubernetes
$ sudo systemctl daemon-reload
$ sudo systemctl restart kubelet

# uncordon the node
$ kubectl uncordon <node-to-uncordon>
```

## [reconfigure cluster](https://v1-28.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure/)

> \[!NOTE|label:references:]
>
> * [Reconfiguring a kubeadm cluster](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure/)
> * recommanded commands: `KUBECONFIG=/etc/kubernetes/admin.conf KUBE_EDITOR=nano kubectl edit <parameters>`
> * components-name:
>   * `apiServer`
>   * `controllerManager`
>   * `scheduler`
>   * `etcd`

```bash
# config map
$ kubectl edit cm -n kube-system kubeadm-config

# certs
$ kubeadm init phase certs <component-name> --config /path/to/kubeadm-config.yaml

# get new /etc/kubernetes/manifests
# for kubernetes control plane components
$ kubeadm init phase control-plane <component-name> --config <config-file>
# for local etcd
$ kubeadm init phase etcd local --config <config-file>

# kubelet
$ kubectl edit cm -n kube-system kubelet-config

# kubeproxy
$ kubectl edit cm -n kube-system kube-proxy
# restart kube-proxy pods
$ kubectl delete po -n kube-system kube-proxy.+

# coredns
$ kubectl edit deployment -n kube-system coredns
$ kubectl edit service -n kube-system kube-dns
# restart coredns
$ kubectl delete po -n kube-system coredns.+
```

## [kubeadm reset and teardown](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-reset/)

> \[!NOTE|label:see also:]
>
> * [\* iMarslo: tear down](https://imarslo.gitbook.io/handbook/virtualization/kubernetes/..#tear-down)

```bash
$ kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
$ kubectl delete node <node name>

$ sudo kubeadm reset --v=5 -f
# or
$ sudo kubeadm reset --cri-socket /var/run/crio/crio.sock --v=5 -f

$ sudo systemctl stop kubelet
$ sudo systemctl stop docker
$ sudo systemctl stop crio
$ sudo systemctl disable --now kubelet
$ sudo systemctl disable --now docker
$ sudo systemctl disable --now crio

$ docker system prune -a -f

# interfaces
# cni0
$ sudo ifconfig cni0 down
$ sudo ip link delete cni0
# calico
$ sudo ifconfig vxlan.calico down
$ sudo ip link delete vxlan.calico
# flannel
$ sudo ifconfig flannel.1 down
$ sudo ip link delete flannel.1

$ sudo rm -rf /etc/kubernetes /var/lib/cni /var/lib/kubelet/* /etc/cni/net.d /etc/cni/ ~/.kube/
$ sudo rm -rf /var/log/pods

$ sudo apt-get purge kubeadm kubectl kubelet kubernetes-cni kube*
$ sudo apt-get autoremove
# or
$ sudo dnf remove kubeadm kubectl kubelet kubernetes-cni kube* cri-o cri-tools --disableexcludes=kubernetes
$ sudo dnf clean all

# iptables
$ sudo iptables -P INPUT ACCEPT && sudo iptables -P FORWARD ACCEPT && sudo iptables -P OUTPUT ACCEPT
$ sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X

# via ipvs
$ sudo ipvsadm -C
```

* yum with versionlock

  ```bash
  $ sudo yum versionlock list
  Loaded plugins: fastestmirror, versionlock
  0:kubeadm-1.15.3-0.*
  0:kubectl-1.15.3-0.*
  0:kubelet-1.15.3-0.*
  0:kubernetes-cni-0.7.5-0.*
  3:docker-ce-18.09.9-3.el7.*
  1:docker-ce-cli-18.09.9-3.el7.*
  versionlock list done

  $ sudo yum versionlock clear
  Loaded plugins: fastestmirror, versionlock
  versionlock cleared

  $ sudo yum versionlock list
  Loaded plugins: fastestmirror, versionlock
  versionlock list done
  ```

## troubleshooting

* `error: open /var/lib/kubelet/config.yaml: no such file or directory`

  ```bash
  Nov 17 19:25:19 kube-node-01 systemd[1]: Started kubelet: The Kubernetes Node Agent.
  Nov 17 19:25:19 kube-node-01 kubelet[28335]: F1117 19:25:19.391266   28335 server.go:190] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
  Nov 17 19:25:19 kube-node-01 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
  Nov 17 19:25:19 kube-node-01 systemd[1]: kubelet.service: Failed with result 'exit-code'.
  ```

  * solution:
    * controller: [kubeadm init phase kubelet-start](https://stackoverflow.com/a/60301879/2940319)

      ```bash
      $ kubeadm init phase kubelet-start
      $ swapoff -a
      $ sudo bash -c "sed -e 's:^\\(.*swap.*\\)$:# \\1:' -i /etc/fstab"
      ```
    * node:

      ```bash
      $ kubeadm join api.kubernetes.com:6443 \
                --token 8*****.***************a \
                --discovery-token-ca-cert-hash sha256:e**************************************************************5 \
                --ignore-preflight-errors=all
      ```
* [`ERROR CRI`](https://forum.linuxfoundation.org/discussion/862825/kubeadm-init-error-cri-v1-runtime-api-is-not-implemented)

  > \[!NOTE|label:current system:]
  >
  > * os:
  >
  >   ```bash
  >   $ printf "$(uname -srm)\n$(cat /etc/os-release)\n"
  >   Linux 5.15.0-107-generic x86_64
  >   NAME="Ubuntu"
  >   VERSION="20.04.6 LTS (Focal Fossa)"
  >   ID=ubuntu
  >   ID_LIKE=debian
  >   PRETTY_NAME="Ubuntu 20.04.6 LTS"
  >   VERSION_ID="20.04"
  >   HOME_URL="https://www.ubuntu.com/"
  >   SUPPORT_URL="https://help.ubuntu.com/"
  >   BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  >   PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  >   VERSION_CODENAME=focal
  >   UBUNTU_CODENAME=focal
  >   ```
  > * packages:
  >
  >   ```bash
  >   $ apt list --installed | grep -E 'kube|container|cri-tools|socat|cni'
  >   containerd.io/focal,now 1.6.33-1 amd64 [installed]
  >   cri-tools/unknown,now 1.28.0-1.1 amd64 [installed,automatic]
  >   kubeadm/unknown,now 1.28.8-1.1 amd64 [installed,upgradable to: 1.28.10-1.1]
  >   kubectl/unknown,now 1.28.8-1.1 amd64 [installed,upgradable to: 1.28.10-1.1]
  >   kubelet/unknown,now 1.28.8-1.1 amd64 [installed,upgradable to: 1.28.10-1.1]
  >   kubernetes-cni/unknown,now 1.2.0-2.1 amd64 [installed,automatic]
  >   socat/focal,now 1.7.3.3-2 amd64 [installed,automatic]
  >   ```

  * error message

    ```bash
    $ sudo kubeadm join 192.168.1.100:6443 --token yspydx.6bhxbnvzojb9vxbr --discovery-token-ca-cert-hash sha256:ef2971478d52cfee63a543fe7ad6f97423d2a943456ff93641c705fc3e82b6c4 --v=5
    [preflight] Running pre-flight checks
      [WARNING Swap]: swap is enabled; production deployments should disable swap unless testing the NodeSwap feature gate of the kubelet
    error execution phase preflight: [preflight] Some fatal errors occurred:
      [ERROR CRI]: container runtime is not running: output: time="2024-06-07T19:42:13-07:00" level=fatal msg="validate service connection: validate CRI v1 runtime API for endpoint \"unix:///var/run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
    , error: exit status 1
    ```
  * solution

    ```bash
    # uninstall `containerd` if necessary
    $ sudo apt remove -y containerd

    # install containerd.io instead of
    $ sudo apt install -y containerd.io

    # remove default config file
    $ sudo mv /etc/containerd/config.toml{,.bak}

    # restart containerd services
    $ sudo systemctl restart containerd
    ```

## verify

* cluster

  ```bash
  # show endpoint
  $ kubectl get endpoints kubernetes
  NAME         ENDPOINTS                                               AGE
  kubernetes   10.68.78.221:6443,10.68.78.222:6443,10.68.78.223:6443   19h

  $ kubectl get cs
  Warning: v1 ComponentStatus is deprecated in v1.19+
  NAME                 STATUS    MESSAGE   ERROR
  controller-manager   Healthy   ok
  scheduler            Healthy   ok
  etcd-0               Healthy   ok

  $ kubectl get no -o wide
  NAME     STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                   KERNEL-VERSION                   CONTAINER-RUNTIME
  k8s-01   Ready    control-plane   19h   v1.30.4   10.68.78.221   <none>        Oracle Linux Server 8.10   5.15.0-202.135.2.el8uek.x86_64   cri-o://1.30.3
  k8s-02   Ready    control-plane   19h   v1.30.4   10.68.78.222   <none>        Oracle Linux Server 8.10   5.15.0-202.135.2.el8uek.x86_64   cri-o://1.30.3
  k8s-03   Ready    control-plane   19h   v1.30.4   10.68.78.223   <none>        Oracle Linux Server 8.10   5.15.0-202.135.2.el8uek.x86_64   cri-o://1.30.3
  ```
* nginx deployment

  ```bash
  $ cat << EOF | kubectl apply -f -
  apiVersion: v1
  kind: Namespace
  metadata:
    name: nginx
  ---
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: nginx
    namespace: nginx
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: nginx
    template:
      metadata:
        labels:
          app: nginx
      spec:
        containers:
        - name: nginx
          image: nginx:alpine
          ports:
          - containerPort: 80
  ---
  apiVersion: v1
  kind: Service
  metadata:
    name: nginx
    namespace: nginx
  spec:
    type: NodePort
    selector:
      app: nginx
    ports:
    - name: http
      port: 80
      targetPort: 80
  EOF

  $ curl -I <running-node.ip>:32520
  ```

## references

{% hint style="info" %}

> * scripts:
>   * [mritd/shell\_scripts](https://github.com/mritd/shell_scripts/tree/master)
>     * [init\_ubuntu.sh](https://github.com/mritd/shell_scripts/blob/master/init_ubuntu.sh)
> * installation:
>   * [\* install tools](https://kubernetes.io/docs/tasks/tools/)
>   * [\* Bootstrapping clusters with kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/_print/)
>   * [\* 使用 kubeadm 创建集群](https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/)
>   * [\* kubeadm 搭建 HA kubernetes 集群](https://mritd.com/2020/01/21/set-up-kubernetes-ha-cluster-by-kubeadm/)
>     * [Implementation details](https://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/)
>   * [\* kubeadm Configuration (v1beta3)](https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta3/)
>   * [\* Implementation details](https://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/)
>   * [\* How To Setup A Three Node Kubernetes Cluster Step By Step](https://k21academy.com/docker-kubernetes/three-node-kubernetes-cluster/)
>   * [cURLing the Kubernetes API server](https://nieldw.medium.com/curling-the-kubernetes-api-server-d7675cfc398c)
>   * [Troubleshooting kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/)
>   * [\* codefarm](https://blog.codefarm.me/category/#kubernetes)
>     * [Kubernetes Recovery from Master Failure with Kubeadm](https://blog.codefarm.me/2019/05/22/kubernetes-recovery-master-failure/)
>     * [1 - Kubernetes Objects](https://blog.codefarm.me/2019/02/22/kubernetes-crash-course-1/)
>     * [2 - Kubernetes Pods](https://blog.codefarm.me/2019/03/04/kubernetes-crash-course-2/)
>     * [3 - Kubernetes Services and Ingress](https://blog.codefarm.me/2019/03/04/kubernetes-crash-course-3/)
>     * [4 - Kubernetes Storage](https://blog.codefarm.me/2019/03/25/kubernetes-crash-course-4/)
>     * [5 - Kubernetes StatefulSet](https://blog.codefarm.me/2020/02/10/kubernetes-crash-course-5/)
>     * [6 - Kubernetes Monitoring](https://blog.codefarm.me/2020/02/10/kubernetes-crash-course-6/)
>   * [authenticating with bootstrap token](https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/)
>   * [kubernetes/design-proposals-archive](https://github.com/kubernetes/design-proposals-archive/blob/main/cluster-lifecycle/bootstrap-discovery.md)
>   * [design-proposals-archive/cluster-lifecycle/cluster-deployment.md](https://github.com/kubernetes/design-proposals-archive/blob/main/cluster-lifecycle/cluster-deployment.md)
>   * [How to Upgrade Kubernetes Cluster Using Kubeadm?](https://devopscube.com/upgrade-kubernetes-cluster-kubeadm/)
>   * [在 Kubernetes 上最小化安装 KubeSphere](https://kubesphere.io/zh/docs/v3.4/quick-start/minimal-kubesphere-on-k8s/)
> * management:
>   * [Administering Kubernetes](https://www.ibm.com/docs/en/fci/1.1.0?topic=administering-kubernetes)
> * upgrade:
>   * [Upgrading kubeadm clusters](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/)
>   * [kubeadm 集群升级](https://mritd.com/2020/01/21/how-to-upgrade-kubeadm-cluster/)
>
>     ```bash
>     $ kubeadm upgrade plan --config /etc/kubernetes/kubeadm.yaml
>     ```
> * network
>   * [Migrating a cluster from flannel to Calico](https://docs.projectcalico.org/v3.9/getting-started/kubernetes/installation/migration-from-flannel)
>   * [v3.9 Migrating a cluster from flannel to Calico](https://docs.tigera.io/archive/v3.9/getting-started/kubernetes/installation/migration-from-flannel)
>   * [#1817: kube-flannel dial tcp 10.96.0.1:443: i/o timeout](https://github.com/kubernetes/kubeadm/issues/1817)
>   * [Quickstart for Calico on Kubernetes](https://docs.tigera.io/archive/v3.9/getting-started/kubernetes/)
>     {% endhint %}

> \[!NOTE|label:references:]
>
> * [\* Generate Certificates Manually](https://kubernetes.io/docs/tasks/administer-cluster/certificates/)
>   * [easyrsa](https://kubernetes.io/docs/tasks/administer-cluster/certificates/#easyrsa)
>   * [openssl](https://kubernetes.io/docs/tasks/administer-cluster/certificates/#openssl)
>   * [cfssl](https://kubernetes.io/docs/tasks/administer-cluster/certificates/#cfssl)
> * [Manage TLS Certificates in a Cluster](https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/)
> * [\* Manual Rotation of CA Certificates](https://kubernetes.io/docs/tasks/tls/manual-rotation-of-ca-certificates/)